Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.9 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2432685 2431915 2026-07-02T08:39:14Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2432685 wikitext text/x-wiki gclq8iqtt6huukab1wabyebkqtma5mm 2432687 2432685 2026-07-02T08:41:44Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2432687 wikitext text/x-wiki 7aclj0f8ym975js3czn4hej3ebn4c93 2432688 2432687 2026-07-02T08:42:48Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2432688 wikitext text/x-wiki 4olt11ympg2354o1uf8vvrjr7i3a836 2432695 2432688 2026-07-02T08:55:23Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2432695 wikitext text/x-wiki sucafvw7k6s4lzazygj1e9r9lolg4x7 2432744 2432695 2026-07-02T10:18:26Z Stashbot 7414 taavi: deploy latest striker with T430120 T409341 fixes 2432744 wikitext text/x-wiki 8wdderukuyg5xnjs2d5wqempcwosiws 0 4108 2432349 2432256 2026-07-01T12:52:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1306456]] to Wednesday, July 01 UTC afternoon backport window 2432349 wikitext text/x-wiki l23xmogjvt9nj5qhx5otbt30hc1lbz2 2432374 2432349 2026-07-01T13:21:00Z ScheduleDeploymentBot 37566 Add [[gerrit:1306916]] to Wednesday, July 01 UTC afternoon backport window 2432374 wikitext text/x-wiki k7h2bhh5uehpsnzioxhsxbktwjb0acy 2432375 2432374 2026-07-01T13:21:05Z ScheduleDeploymentBot 37566 Add [[gerrit:1306925]] to Wednesday, July 01 UTC afternoon backport window 2432375 wikitext text/x-wiki qerd0hjg1knnv95w1fy0mvbcm5lb5ms 2432484 2432375 2026-07-01T15:59:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1306950]] to Wednesday, July 01 UTC late backport window 2432484 wikitext text/x-wiki nlkoxkcqi7ado4sco057lfufeqkpz5h 2432485 2432484 2026-07-01T15:59:29Z ScheduleDeploymentBot 37566 Add [[gerrit:1306910]] to Wednesday, July 01 UTC late backport window 2432485 wikitext text/x-wiki 4nmb0jullzeugjzsctlc5tpg0nycoqt 2432493 2432485 2026-07-01T16:13:42Z ScheduleDeploymentBot 37566 Add [[gerrit:1305773]] to Wednesday, July 01 UTC late backport window 2432493 wikitext text/x-wiki 28ias12dn5lomy5eg8ezi2qk8g380jv 2432506 2432493 2026-07-01T16:35:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1305773]] to Thursday, July 02 UTC afternoon backport window 2432506 wikitext text/x-wiki n7hsgh55p8yvfzrrlbudd9y2bkmmf4p 2432507 2432506 2026-07-01T16:36:22Z Aude 39 /* {{Deployment_day|date=2026-07-01}} */ moved to tomorrow 2432507 wikitext text/x-wiki ox5nft058fupwlvhsqpc4t878vizvnl 2432524 2432507 2026-07-01T18:02:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1306970]] to Thursday, July 02 UTC morning backport window 2432524 wikitext text/x-wiki 7k58071n58x6oy930n0n5nuqsl4nsna 2432525 2432524 2026-07-01T18:02:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1306971]] to Thursday, July 02 UTC morning backport window 2432525 wikitext text/x-wiki 09sg07vndqpc5qwbwdzyoyjv244yvis 2432578 2432525 2026-07-01T23:45:16Z ScheduleDeploymentBot 37566 Add [[gerrit:1306985]] to Thursday, July 02 UTC morning backport window 2432578 wikitext text/x-wiki 2m345601i6wdgunj5m3lw1c48xnjgnu 2432579 2432578 2026-07-01T23:48:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1306996]] to Thursday, July 02 UTC morning backport window 2432579 wikitext text/x-wiki 59vxkvv1tspkdizx720u0n7shr6akc9 2432642 2432579 2026-07-02T07:26:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1307058]] to Thursday, July 02 UTC morning backport window 2432642 wikitext text/x-wiki rpgmu2gvhafxkglsqyjy5qr8z3bkv7p 2432645 2432642 2026-07-02T07:41:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1307061]] to Thursday, July 02 UTC morning backport window 2432645 wikitext text/x-wiki 9yoe62juw988pfcpu6v6yrktizs7sr0 2432646 2432645 2026-07-02T07:41:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1307059]] to Thursday, July 02 UTC morning backport window 2432646 wikitext text/x-wiki dbhoblgkdlq4dgl85k0yht7k12m0z7g 2432753 2432646 2026-07-02T10:49:51Z ScheduleDeploymentBot 37566 Add [[gerrit:1307099]] to Thursday, July 02 UTC afternoon backport window 2432753 wikitext text/x-wiki qux7c2gsdrggy7h149774lctwkfm7u3 2432754 2432753 2026-07-02T10:53:54Z Atsuko-WMF 54082 /* {{Deployment_day|date=2026-07-02}} */ revert, doing emergency deploy 2432754 wikitext text/x-wiki dbhoblgkdlq4dgl85k0yht7k12m0z7g 0 4118 2432528 2431067 2026-07-01T18:12:38Z APaskulin (WMF) 14880 Add a cleanup task that came up when deleting the API Portal 2432528 wikitext text/x-wiki czqvwelronvlgg0gptue9so927uetfq 498 6726 2432663 2431906 2026-07-02T07:58:56Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2432663 wikitext text/x-wiki lw5ytlfj5m14j9fi8oah6fj77405oij 2432681 2432663 2026-07-02T08:19:14Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2432681 wikitext text/x-wiki 1vm96kojl05ecdfgmekgblji2vtemwg 2432684 2432681 2026-07-02T08:37:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2432684 wikitext text/x-wiki kn8j8bxrm1hjwe6296yszaksy7gm7k4 2432715 2432684 2026-07-02T09:35:21Z Stashbot 7414 godog: carry out nfs tests on toolsbeta-test-k8s-worker-nfs-8 - T411248 2432715 wikitext text/x-wiki im1yli6hh3a2ozwzbl7vvxhlzxkb5co 0 7919 2432335 2432334 2026-07-01T12:00:11Z Stashbot 7414 topranks: drain traffic on cr1-eqiad to allow for line card install and JunOS upgrade T426343 2432335 wikitext text/x-wiki kdo1he7w6h8k7i1e7c3dd09mamoncdw 2432336 2432335 2026-07-01T12:09:01Z Stashbot 7414 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage 2432336 wikitext text/x-wiki jisz173652jbsc5ptsw0v4p7o4d0aq5 2432337 2432336 2026-07-01T12:09:32Z Stashbot 7414 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage 2432337 wikitext text/x-wiki 83a8v5tjek8k9hudtmoj8bk1hgntvcf 2432338 2432337 2026-07-01T12:14:43Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage 2432338 wikitext text/x-wiki flkvf2a60892kp769j5pgjcoxtj7ldv 2432339 2432338 2026-07-01T12:17:12Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2240: Migration of db2240.codfw.wmnet completed 2432339 wikitext text/x-wiki j5nw0vrsm8g56l8svygrtwfc9mqrlj0 2432340 2432339 2026-07-01T12:17:13Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2432340 wikitext text/x-wiki agwwaifhi0zxlu8fcc1bcnr0s46tmwp 2432342 2432340 2026-07-01T12:19:22Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage 2432342 wikitext text/x-wiki 4v5s0gba47jexbiwl06j27cu1lb1by7 2432343 2432342 2026-07-01T12:38:43Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2083.codfw.wmnet with OS trixie 2432343 wikitext text/x-wiki 1mtbz4opv0mr8pn8oxks8j9ssbjzzzj 2432344 2432343 2026-07-01T12:42:36Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2100.codfw.wmnet with OS trixie 2432344 wikitext text/x-wiki 0j0wzmlsy2qqy6l2l6c1y5xrsuyvxgk 2432345 2432344 2026-07-01T12:50:10Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet 2432345 wikitext text/x-wiki 9kslp811n2mm40wyc8fduhl5za5d346 2432346 2432345 2026-07-01T12:50:11Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet 2432346 wikitext text/x-wiki m1yh1xs8ow779seav4cmzc9552it312 2432347 2432346 2026-07-01T12:51:44Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Primary switchover s6 T430814 2432347 wikitext text/x-wiki a4dvxue5fh9t4b07r5knc2qrdrwphqp 2432348 2432347 2026-07-01T12:51:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2229 with weight 0 T430814', diff saved to https://phabricator.wikimedia.org/P94658 and previous config saved to /var/cache/conftool/dbconfig/20260701-125149-fceratto.json 2432348 wikitext text/x-wiki ac9nzz4j48433tn2tcdwljochcqttet 2432352 2432348 2026-07-01T12:57:04Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 13 hosts with reason: router upgrade and line card install 2432352 wikitext text/x-wiki 2urp3mjb9slb16349g44wazgjcijkcc 2432354 2432352 2026-07-01T12:59:26Z Stashbot 7414 federico3: Starting s6 codfw failover from db2214 to db2229 - T430814 2432354 wikitext text/x-wiki ssvez8wfzat8dnl2q9y81fmvd41cnlp 2432355 2432354 2026-07-01T13:00:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2229 to s6 primary T430814', diff saved to https://phabricator.wikimedia.org/P94659 and previous config saved to /var/cache/conftool/dbconfig/20260701-125959-fceratto.json 2432355 wikitext text/x-wiki 5ka4gp9tuzay5t8z0wfjtbw02wy5eob 2432356 2432355 2026-07-01T13:01:59Z Stashbot 7414 moritzm: installing python3.13 security updates 2432356 wikitext text/x-wiki b95h3zl76xwmpf9tbu4v7kjblxj3zgd 2432357 2432356 2026-07-01T13:04:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2214 T430814', diff saved to https://phabricator.wikimedia.org/P94660 and previous config saved to /var/cache/conftool/dbconfig/20260701-130413-fceratto.json 2432357 wikitext text/x-wiki o9gyucssdgfp9x5x4frw2lb46autqn2 2432358 2432357 2026-07-01T13:04:22Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling after switchover 2432358 wikitext text/x-wiki noszd41a17fjkj5ce73e3evt625d4po 2432359 2432358 2026-07-01T13:05:16Z Stashbot 7414 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306873|Parsoid read views: Bump enwiki traffic to 75%]] 2432359 wikitext text/x-wiki dod4lpw6uru7ky7pdy1e0z1s2cggxrp 2432360 2432359 2026-07-01T13:05:53Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2214: Repooling after switchover 2432360 wikitext text/x-wiki ddicoqaggtdptjqjp0qrdn5ajpgavvk 2432361 2432360 2026-07-01T13:06:48Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: Maintenance 2432361 wikitext text/x-wiki 3835egl68qisc99zb8i0dyrql5dk9he 2432362 2432361 2026-07-01T13:06:59Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2432362 wikitext text/x-wiki o7118dexkux6nr9lk67xy75y3h39ryz 2432363 2432362 2026-07-01T13:07:30Z Stashbot 7414 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1306873|Parsoid read views: Bump enwiki traffic to 75%]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432363 wikitext text/x-wiki acqilz3w2avtfvce2dohbnngzqh55u5 2432364 2432363 2026-07-01T13:08:20Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2432364 wikitext text/x-wiki ip7neeg4wo8hpundbp7nx4woyd0qobn 2432365 2432364 2026-07-01T13:09:22Z Stashbot 7414 jgiannelos@deploy1003: jgiannelos: Continuing with deployment 2432365 wikitext text/x-wiki mv6xasetzdgm93gzb6jodkd0sj3c42u 2432366 2432365 2026-07-01T13:11:23Z Stashbot 7414 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org 2432366 wikitext text/x-wiki sqkufl2ko83xv3wz2cnzt0rj2ksa1gq 2432367 2432366 2026-07-01T13:11:28Z Stashbot 7414 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org 2432367 wikitext text/x-wiki 56ujp6c7l3td7o0vnhlihnh0soggnzr 2432368 2432367 2026-07-01T13:13:04Z Stashbot 7414 moritzm: installing qemu security updates 2432368 wikitext text/x-wiki hnriv37mzx44jfybsn2std8x8metor4 2432369 2432368 2026-07-01T13:13:45Z Stashbot 7414 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306873|Parsoid read views: Bump enwiki traffic to 75%]] (duration: 08m 29s) 2432369 wikitext text/x-wiki 3zjaitpb4jq5md4lvvowhi1yea8g96c 2432370 2432369 2026-07-01T13:15:19Z Stashbot 7414 topranks: rebooting routing-engine 1 on cr1-eqiad T417873 2432370 wikitext text/x-wiki b8e5ak2zfewzslq8o3pzazp89m1c22l 2432384 2432370 2026-07-01T13:26:47Z Stashbot 7414 caro@deploy1003: Started scap sync-world: Backport for [[gerrit:1306842|EditCheck: fix pre-save focusedAction error (T430741)]] 2432384 wikitext text/x-wiki bib1ch1bdm7gv4i4abz0wlyuo9pgk97 2432385 2432384 2026-07-01T13:27:03Z Stashbot 7414 topranks: route-engine failover cr1-eqiad 2432385 wikitext text/x-wiki cl7cdad8ezk0sd4wujovvm71nrsybz6 2432386 2432385 2026-07-01T13:28:52Z Stashbot 7414 caro@deploy1003: caro: Backport for [[gerrit:1306842|EditCheck: fix pre-save focusedAction error (T430741)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432386 wikitext text/x-wiki ihljm27sobxvs6l0qvazpcxeaj63q3n 2432387 2432386 2026-07-01T13:30:24Z Stashbot 7414 caro@deploy1003: caro: Continuing with deployment 2432387 wikitext text/x-wiki 7b3ftlula3mbg4my749x4gw1ordwmyp 2432388 2432387 2026-07-01T13:34:46Z Stashbot 7414 caro@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306842|EditCheck: fix pre-save focusedAction error (T430741)]] (duration: 07m 59s) 2432388 wikitext text/x-wiki 2p62aqywwn5rajxvm1yimff6dhsre3p 2432389 2432388 2026-07-01T13:35:25Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs[1017-1020].eqiad.wmnet with reason: router upgrades eqiad 2432389 wikitext text/x-wiki f53h9umv42s0pzs4eowm1xegh2lu5hw 2432390 2432389 2026-07-01T13:37:01Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on pfw1-eqiad with reason: router upgrades eqiad 2432390 wikitext text/x-wiki ktfmkjuz7knekhm0kqq3qqrd473szri 2432391 2432390 2026-07-01T13:37:18Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1005.wikimedia.org 2432391 wikitext text/x-wiki 0kpo13j941golz7uyl7hqfk9lk2virh 2432393 2432391 2026-07-01T13:41:46Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1005.wikimedia.org 2432393 wikitext text/x-wiki 0ckst4o8qh66ngjcdyjmqke43hfinb2 2432394 2432393 2026-07-01T13:41:50Z Stashbot 7414 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2092.codfw.wmnet with OS trixie 2432394 wikitext text/x-wiki 6euu0cprsqanf327lgwcs426tm1vkia 2432395 2432394 2026-07-01T13:43:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1006.wikimedia.org 2432395 wikitext text/x-wiki ig80mjwfv9jquc3phqhup0j8mfln86n 2432396 2432395 2026-07-01T13:44:32Z Stashbot 7414 atsuko@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cirrussearch2092.codfw.wmnet with OS trixie 2432396 wikitext text/x-wiki 8xoo8dunai3hzy6ng9fc98uoeif4wey 2432397 2432396 2026-07-01T13:48:06Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1006.wikimedia.org 2432397 wikitext text/x-wiki q8ty8pdoy3di567seimnhdmmth2v06t 2432398 2432397 2026-07-01T13:56:55Z Stashbot 7414 topranks: reboot routing-enginer RE0 on cr1-eqiad T417873 2432398 wikitext text/x-wiki 0y6a0rx25omvdlb47xch0aebexmyr3n 2432399 2432398 2026-07-01T13:57:25Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T430826 2432399 wikitext text/x-wiki bb6yvdc9iny0o2oidqxprautk6kf2xv 2432400 2432399 2026-07-01T13:58:46Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306456|eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916|Move non temporary accounts settings out TA section]], [[gerrit:1306925|Remove TA patrol rights from users on fishbowl + private (T425048)]] 2432400 wikitext text/x-wiki o8c8e216z35e25ox79owsz4fj856yf2 2432403 2432400 2026-07-01T13:59:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2159 with weight 0 T430826', diff saved to https://phabricator.wikimedia.org/P94662 and previous config saved to /var/cache/conftool/dbconfig/20260701-135906-fceratto.json 2432403 wikitext text/x-wiki taaeavs9qs0dz2776inyju6x42xjju2 2432405 2432403 2026-07-01T14:00:57Z Stashbot 7414 dreamyjazz@deploy1003: anzx, dreamyjazz: Backport for [[gerrit:1306456|eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916|Move non temporary accounts settings out TA section]], [[gerrit:1306925|Remove TA patrol rights from users on fishbowl + private (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432405 wikitext text/x-wiki 963mmw97prq25k9q1za09mgww7ddisl 2432407 2432405 2026-07-01T14:01:43Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2432407 wikitext text/x-wiki c7i4qvw52nxeigjy43658206d2rs260 2432408 2432407 2026-07-01T14:03:10Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudsw1-c8-eqiad,cloudsw1-d5-eqiad with reason: router upgrades eqiad 2432408 wikitext text/x-wiki eta3kfu132y6k7h1dzlec58m8fjc3yh 2432409 2432408 2026-07-01T14:03:29Z Stashbot 7414 topranks: flipping cr1-eqiad active routing-enginer back to RE0 T417873 2432409 wikitext text/x-wiki jerj3dy0s8kjmc3l2dh1tjxtw6vks99 2432410 2432409 2026-07-01T14:03:40Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2432410 wikitext text/x-wiki hh68cgsft7fq6911w338hx8eh9we2df 2432411 2432410 2026-07-01T14:03:48Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2432411 wikitext text/x-wiki t8hb9hf49y20n73li4a8ch9kjonokht 2432412 2432411 2026-07-01T14:04:14Z Stashbot 7414 federico3: Starting s7 codfw failover from db2220 to db2159 - T430826 2432412 wikitext text/x-wiki jlxqdovpl6sy0cg3w60sexjx3ktzd4g 2432413 2432412 2026-07-01T14:04:23Z Stashbot 7414 dreamyjazz@deploy1003: anzx, dreamyjazz: Continuing with deployment 2432413 wikitext text/x-wiki hadx0glg9hkey5captb240j3kyw8w93 2432415 2432413 2026-07-01T14:04:34Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2432415 wikitext text/x-wiki h9hs2peh5j3krsa6r9mhvie4jlpoqhq 2432416 2432415 2026-07-01T14:04:37Z Stashbot 7414 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org 2432416 wikitext text/x-wiki 5g4ovz98uhze1chho8ydtfm617bp6pa 2432417 2432416 2026-07-01T14:04:44Z Stashbot 7414 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org 2432417 wikitext text/x-wiki n0p6cmn1g9uttqjip6503x9ckm9nqor 2432418 2432417 2026-07-01T14:04:52Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2432418 wikitext text/x-wiki t8janahxm8czoieyo5rcs1djb68dez4 2432419 2432418 2026-07-01T14:05:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2159 to s7 primary T430826', diff saved to https://phabricator.wikimedia.org/P94663 and previous config saved to /var/cache/conftool/dbconfig/20260701-140503-fceratto.json 2432419 wikitext text/x-wiki kqpxdn4lev9dce086flm26yhg57yk21 2432420 2432419 2026-07-01T14:06:17Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2432420 wikitext text/x-wiki sfv1ml8iq5xze181euyz052r4qm9zzk 2432421 2432420 2026-07-01T14:06:23Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2432421 wikitext text/x-wiki 7cd3d7b34nsehxepag8bjb008mptb5y 2432422 2432421 2026-07-01T14:06:59Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2432422 wikitext text/x-wiki mvqrpnkplgb60ljuzhty1v2u9meja3v 2432423 2432422 2026-07-01T14:07:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2220 T430826', diff saved to https://phabricator.wikimedia.org/P94664 and previous config saved to /var/cache/conftool/dbconfig/20260701-140729-fceratto.json 2432423 wikitext text/x-wiki id1f4gkztmkio0d3ckh5mol81watcb0 2432424 2432423 2026-07-01T14:07:37Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover 2432424 wikitext text/x-wiki 896moaed0jdahl6qrjh63lrz1mp561e 2432425 2432424 2026-07-01T14:08:47Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306456|eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916|Move non temporary accounts settings out TA section]], [[gerrit:1306925|Remove TA patrol rights from users on fishbowl + private (T425048)]] (duration: 10m 01s) 2432425 wikitext text/x-wiki avs7vfiurjce4wot30ceur98uu8j4t6 2432426 2432425 2026-07-01T14:10:23Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2432426 wikitext text/x-wiki ce7e1bvsuriyvn1fvbu018apwla01bq 2432427 2432426 2026-07-01T14:10:48Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2432427 wikitext text/x-wiki q2ixleyr26zrqx31ss46t02eoody93u 2432428 2432427 2026-07-01T14:11:19Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2432428 wikitext text/x-wiki dlvamxeyju9xe6hqiw6mwbw0qc96gyn 2432429 2432428 2026-07-01T14:12:04Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2432429 wikitext text/x-wiki odm09br6o9k45uitdo71ioypzf2s4vi 2432430 2432429 2026-07-01T14:12:13Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2432430 wikitext text/x-wiki b3pc311nzfdlu6c3yi47mbbo1akpsp9 2432431 2432430 2026-07-01T14:12:40Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2432431 wikitext text/x-wiki khd1xut1k6gw5m8fs5dql7vjyrhurd7 2432432 2432431 2026-07-01T14:13:08Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2220: Repooling after switchover 2432432 wikitext text/x-wiki hbza4fi5m32tsykbii4rb3y7h5kg3dj 2432433 2432432 2026-07-01T14:13:48Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover 2432433 wikitext text/x-wiki n15zl0v4uehhu1tukz5yw6a14bisbdk 2432434 2432433 2026-07-01T14:14:08Z Stashbot 7414 topranks: re-enable routing-engine graceful-failover on cr1-eqiad T417873 2432434 wikitext text/x-wiki btisgwsi1zdm0zr9tj3mmz249vt1iwh 2432435 2432434 2026-07-01T14:15:21Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover 2432435 wikitext text/x-wiki o3iw15w4bzept9wd2mld2jmgx9obyt7 2432437 2432435 2026-07-01T14:16:49Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover 2432437 wikitext text/x-wiki 7f6c87poyf44xioezotn9lrh3eopeki 2432438 2432437 2026-07-01T14:22:02Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover 2432438 wikitext text/x-wiki dw25e1o71i7hzixnhw0gvebs6mjh63v 2432439 2432438 2026-07-01T14:22:11Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306938|Remove group permissions definitions later in the request (T425048)]] 2432439 wikitext text/x-wiki oc7jjl1yqibgi89x783ahsxokhpklqp 2432440 2432439 2026-07-01T14:24:21Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1306938|Remove group permissions definitions later in the request (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432440 wikitext text/x-wiki 1yaahwzymqcwqs34xxo0hckrbcomhac 2432441 2432440 2026-07-01T14:26:44Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2432441 wikitext text/x-wiki ik96ckt214fhc4dyxoyz1x5b044up4k 2432442 2432441 2026-07-01T14:29:38Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover 2432442 wikitext text/x-wiki rer67705me704yiszli7ov68enhqyxh 2432443 2432442 2026-07-01T14:31:08Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306938|Remove group permissions definitions later in the request (T425048)]] (duration: 08m 57s) 2432443 wikitext text/x-wiki qyemcpbiugfxsvb5pomorl4g6u4ljf0 2432444 2432443 2026-07-01T14:31:11Z Stashbot 7414 topranks: POWERING DOWN CR1-EQIAD for line card installation T426343 2432444 wikitext text/x-wiki 9ysxf5pm2cv25zb2qrwao4x5g9ebv3f 2432447 2432444 2026-07-01T15:02:49Z Stashbot 7414 papaul: ongoing maintenance on lsw1-a8-codfw 2432447 wikitext text/x-wiki jf3zpggipnn4rvrfmta9tzew7dcque7 2432448 2432447 2026-07-01T15:06:34Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042,2046].codfw.wmnet 2432448 wikitext text/x-wiki desjsff7zv1fv0xepg95v4f6m0cm9k1 2432449 2432448 2026-07-01T15:07:43Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042,2046].codfw.wmnet 2432449 wikitext text/x-wiki l5qu8hhoe0a0po2imu3tcnga76b8unq 2432450 2432449 2026-07-01T15:09:36Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover 2432450 wikitext text/x-wiki 4yenl851nvh5q9zmarr1x58st76hr72 2432451 2432450 2026-07-01T15:12:07Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt with reason: Junos upograde 2432451 wikitext text/x-wiki 5nftycw0nnfsmgfvdjtab9qxay5xdoe 2432452 2432451 2026-07-01T15:12:12Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply 2432452 wikitext text/x-wiki bygzrsrz7ut3lveqdkz6upp2ax7573p 2432453 2432452 2026-07-01T15:12:16Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply 2432453 wikitext text/x-wiki 7z2egnwfllqlkpxeah5lllforzmf47d 2432455 2432453 2026-07-01T15:12:33Z Stashbot 7414 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage 2432455 wikitext text/x-wiki bsoq1y33eojeynr5ayspym9xtcjm63y 2432456 2432455 2026-07-01T15:12:34Z Stashbot 7414 _joe_: restarted manually alertmanager-irc-relay 2432456 wikitext text/x-wiki jrbg2voifh6lfrheq422gkvdka7rq9o 2432457 2432456 2026-07-01T15:19:16Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2432457 wikitext text/x-wiki jng340bi5ygavgbjgemgw7pti4wiyaa 2432458 2432457 2026-07-01T15:20:00Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2432458 wikitext text/x-wiki mdfmdnnbgvmfg1a3glb8uwloi3fdvj3 2432459 2432458 2026-07-01T15:20:03Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage 2432459 wikitext text/x-wiki 5fsydv5pl43txt219uilu4q3xur0usr 2432460 2432459 2026-07-01T15:21:08Z Stashbot 7414 brett: Stopping pybal on lvs2012 in preparation for codfw rack b2 maintenance - T429861 2432460 wikitext text/x-wiki ichldmtdj72stwwvarztrahfegzmi2t 2432461 2432460 2026-07-01T15:22:35Z Stashbot 7414 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2012.codfw.wmnet with reason: Rack B2 maintenance - T429861 2432461 wikitext text/x-wiki 0o2vrfj7lccd3qxkiqufygszgsxf4vk 2432462 2432461 2026-07-01T15:23:36Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply 2432462 wikitext text/x-wiki bdjrhjoj3xi50japqnochvv7wlle7do 2432463 2432462 2026-07-01T15:23:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply 2432463 wikitext text/x-wiki 17lvh00xm2x7t0iylbrkl9dl7wwtclr 2432464 2432463 2026-07-01T15:25:56Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply 2432464 wikitext text/x-wiki f0os6k49xp802i0vifes8a02ksm53ep 2432465 2432464 2026-07-01T15:26:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply 2432465 wikitext text/x-wiki gpwu1g93v5jtrlgvxr7zdlgk7rnfekk 2432466 2432465 2026-07-01T15:29:23Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2432466 wikitext text/x-wiki 9rj5txjt09dsqo8mzp0gifm8ph89v53 2432467 2432466 2026-07-01T15:30:21Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2432467 wikitext text/x-wiki 319uiur7gi90zc0o9117bt3w0iv50le 2432468 2432467 2026-07-01T15:32:16Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2432468 wikitext text/x-wiki aw4hj0tvi0rjl9t4u9dzjf72bgvs8gl 2432469 2432468 2026-07-01T15:32:51Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2432469 wikitext text/x-wiki awnenwedupdy1yqsueo83azx0kef167 2432470 2432469 2026-07-01T15:35:00Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply 2432470 wikitext text/x-wiki k3j7pjbxm7xxvdmjr7e8p92nn9wl182 2432471 2432470 2026-07-01T15:35:32Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply 2432471 wikitext text/x-wiki mpfphnmf3fdd559dhgup420j2ir7ava 2432472 2432471 2026-07-01T15:37:23Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2432472 wikitext text/x-wiki 6z6z9ly4oi6ghyqf5bsl2gfyaf8z3v6 2432473 2432472 2026-07-01T15:38:25Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2432473 wikitext text/x-wiki 7ayh8gtuuvpqmophhttu0kh92xlqa7k 2432474 2432473 2026-07-01T15:40:06Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2432474 wikitext text/x-wiki l0472e8rdedgxakjmki2cyued667b37 2432475 2432474 2026-07-01T15:41:06Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2432475 wikitext text/x-wiki 09vjjk0awg5tmulvt6z3za4qiqy4f2x 2432476 2432475 2026-07-01T15:48:24Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2092.codfw.wmnet with OS trixie 2432476 wikitext text/x-wiki 84v103x38h3lyvcwranbzgdycpm8bwt 2432477 2432476 2026-07-01T15:50:58Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover 2432477 wikitext text/x-wiki kxf3d5scy2gc2cf7ho2llmke5a5np02 2432478 2432477 2026-07-01T15:50:59Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2432478 wikitext text/x-wiki 7evt8ae4p557id9wey2tc1m83rlkt7y 2432479 2432478 2026-07-01T15:51:09Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2220: Repooling after switchover 2432479 wikitext text/x-wiki prmhhezl74qxm3mc2o0s1i6m6u9kcia 2432480 2432479 2026-07-01T15:51:12Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2432480 wikitext text/x-wiki pi1g3ei5rpi508vpa3rnm8agq0fkh78 2432481 2432480 2026-07-01T15:55:44Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2042,2046].codfw.wmnet 2432481 wikitext text/x-wiki qe5p076xa3tfxzo0dhexfa3su45abhd 2432482 2432481 2026-07-01T15:55:46Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2042,2046].codfw.wmnet 2432482 wikitext text/x-wiki 4t38jcm4jwiiyoyug0k58v37t9gc7k0 2432483 2432482 2026-07-01T15:57:43Z Stashbot 7414 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2061.codfw.wmnet with OS trixie 2432483 wikitext text/x-wiki 570y0zpl4ejxca2g6um1ng1asfd7tgp 2432486 2432483 2026-07-01T15:59:41Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt 2432486 wikitext text/x-wiki 7m6vo07lpjf34fcn6ukgce65nxvo1gb 2432487 2432486 2026-07-01T15:59:43Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt 2432487 wikitext text/x-wiki irolwmgpy3wi61ofp25y52z5x5rlob1 2432488 2432487 2026-07-01T16:00:10Z Stashbot 7414 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2076.codfw.wmnet with OS trixie 2432488 wikitext text/x-wiki mg93d41p1ma6yorbn47rw69jpat8h6g 2432489 2432488 2026-07-01T16:00:29Z Stashbot 7414 papaul: ongoing maintenance on lsw1-b2-codfw 2432489 wikitext text/x-wiki l8xhu3z0sqrebejz2r5fp3o4gofciiw 2432490 2432489 2026-07-01T16:06:44Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt with reason: Junos upograde 2432490 wikitext text/x-wiki im8jptwh24erqq07kk82l7e3n8b7xlp 2432491 2432490 2026-07-01T16:07:29Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: maintenance 2432491 wikitext text/x-wiki agz93r34qxp22vmqcnr31k4ztjapfg5 2432492 2432491 2026-07-01T16:12:45Z Stashbot 7414 jasmine@dns1004: START - running authdns-update 2432492 wikitext text/x-wiki 7tfwyqyow8f10wpc3y809rhnqkn4bpf 2432494 2432492 2026-07-01T16:14:51Z Stashbot 7414 jasmine@dns1004: END - running authdns-update 2432494 wikitext text/x-wiki stwxldlxby8it12gu6h07cugytopz3m 2432495 2432494 2026-07-01T16:15:55Z Stashbot 7414 jasmine@dns1004: START - running authdns-update 2432495 wikitext text/x-wiki 5as879nesvs15u3si2ketku20qof76i 2432496 2432495 2026-07-01T16:16:01Z Stashbot 7414 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage 2432496 wikitext text/x-wiki 0fo5qgae2f3g49cqbw4g72fhh3udh8t 2432497 2432496 2026-07-01T16:16:18Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2432497 wikitext text/x-wiki qeb5daegr2gkftt4c1cu7lv4s25o4vy 2432498 2432497 2026-07-01T16:16:31Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2432498 wikitext text/x-wiki ojiuhc6h1iqz6czxt9kz5o5j4ct8w6c 2432499 2432498 2026-07-01T16:18:06Z Stashbot 7414 jasmine@dns1004: END - running authdns-update 2432499 wikitext text/x-wiki 5m5odp7z9pp8uxnkf0hkn74qw77fv0y 2432500 2432499 2026-07-01T16:18:15Z Stashbot 7414 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage 2432500 wikitext text/x-wiki 8xfmk4hwnv63thr97ee6ua79859mpjt 2432501 2432500 2026-07-01T16:19:58Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage 2432501 wikitext text/x-wiki 8tqjxj117svx38z1etccbcvm1qse2vo 2432503 2432501 2026-07-01T16:23:52Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage 2432503 wikitext text/x-wiki aa7hepywtdypcki4zcvjs1g2fs8kmbm 2432504 2432503 2026-07-01T16:28:34Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.271.0" for 2 host(s) 2432504 wikitext text/x-wiki m2e3c0herqwoj9o5sy4byrd3v1xz95k 2432505 2432504 2026-07-01T16:30:28Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.271.0" completed for 2 hosts 2432505 wikitext text/x-wiki pfxzwuwkyfskjip4gih6sq8gzp1minh 2432508 2432505 2026-07-01T16:42:23Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2061.codfw.wmnet with OS trixie 2432508 wikitext text/x-wiki rmpe5lze8hisecaqdo64ellfqx449e5 2432509 2432508 2026-07-01T16:48:34Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 59 hosts 2432509 wikitext text/x-wiki b4zt8d34ur1790fum60hbj50nkisme5 2432510 2432509 2026-07-01T16:49:05Z Stashbot 7414 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.remove-downtime (exit_code=97) for 59 hosts 2432510 wikitext text/x-wiki cqb14hzumldunhijv3pzrxle0jcg86w 2432511 2432510 2026-07-01T16:49:08Z Stashbot 7414 brett: Start pybal on lvs2012 - T429861 2432511 wikitext text/x-wiki 0s1i53ed63c8f9fo8ldx2ctk00gnj19 2432512 2432511 2026-07-01T16:49:19Z Stashbot 7414 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2076.codfw.wmnet with OS trixie 2432512 wikitext text/x-wiki 2o696sgt7iatr3f6zgs47svq678ekdn 2432513 2432512 2026-07-01T16:51:37Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs2012.codfw.wmnet 2432513 wikitext text/x-wiki dzbz064nx5cjvo8sdv1gfv024khg67h 2432514 2432513 2026-07-01T16:51:38Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2012.codfw.wmnet 2432514 wikitext text/x-wiki lb00ptpy9ln9dcynfg3z480nxl77lik 2432515 2432514 2026-07-01T16:51:46Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt 2432515 wikitext text/x-wiki 2o2krz0oro64y3xc10lsmj388y9yxjy 2432516 2432515 2026-07-01T16:51:48Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt 2432516 wikitext text/x-wiki q5qpklw9mt9rqc8oi3djbdc759xlifw 2432517 2432516 2026-07-01T16:52:40Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2202.codfw.wmnet 2432517 wikitext text/x-wiki 7qots62nn5oy3i5rkyg6e6c0n6bneaq 2432518 2432517 2026-07-01T16:52:40Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2202.codfw.wmnet 2432518 wikitext text/x-wiki rkbacgwz1an363x6c9nj95kioy4kfqj 2432519 2432518 2026-07-01T16:57:54Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 30 hosts 2432519 wikitext text/x-wiki bduoynsuh8t71rm184r8knrua2c6txf 2432520 2432519 2026-07-01T16:58:11Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 30 hosts 2432520 wikitext text/x-wiki 6f5eh00ll8ea0d8h515l9xp4ufs21tk 2432522 2432520 2026-07-01T17:40:25Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie 2432522 wikitext text/x-wiki cmd2w9iimuxjcsdhotnhk9y28ax30h1 2432526 2432522 2026-07-01T18:09:04Z Stashbot 7414 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306950|PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910|PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] 2432526 wikitext text/x-wiki evdx8worut45e8jvhtfb7ckv7ri1nfb 2432527 2432526 2026-07-01T18:11:05Z Stashbot 7414 jgiannelos@deploy1003: jgiannelos, neriah: Backport for [[gerrit:1306950|PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910|PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432527 wikitext text/x-wiki sdhirhi9kp1viw4a1d3xbx6fsazn8qv 2432529 2432527 2026-07-01T18:13:59Z Stashbot 7414 jgiannelos@deploy1003: jgiannelos, neriah: Continuing with deployment 2432529 wikitext text/x-wiki nflcjwfo2b0klgxxrbqh39uc00rclrc 2432530 2432529 2026-07-01T18:18:19Z Stashbot 7414 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306950|PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910|PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] (duration: 09m 15s) 2432530 wikitext text/x-wiki 03g41xbghiba9rdz28vhrsskaxglvb1 2432532 2432530 2026-07-01T18:27:05Z Stashbot 7414 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9 refs T423918 2432532 wikitext text/x-wiki 5n8068nlzn9t4koph6ikxg6wi389q1x 2432533 2432532 2026-07-01T18:50:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2080.codfw.wmnet with OS trixie 2432533 wikitext text/x-wiki 855ll7tmo3d8280jv0w0vweqi933uj5 2432534 2432533 2026-07-01T18:57:07Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2093.codfw.wmnet with OS trixie 2432534 wikitext text/x-wiki jd2esmfcky0vr8zd3rziizx7o9n1dbp 2432535 2432534 2026-07-01T19:07:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage 2432535 wikitext text/x-wiki rpp493k4xditov6h1ao3p0extk4pdli 2432536 2432535 2026-07-01T19:15:12Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage 2432536 wikitext text/x-wiki itc5swa3gtt3ztgbbvmc27h3mztqxzv 2432537 2432536 2026-07-01T19:17:21Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage 2432537 wikitext text/x-wiki cs9ed4hwazbz0bgfoepn6d4lntq4m4e 2432538 2432537 2026-07-01T19:18:54Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage 2432538 wikitext text/x-wiki t0gimbit55baeyfi1k9isqx9vsivd7q 2432539 2432538 2026-07-01T19:24:29Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage 2432539 wikitext text/x-wiki 7u5le1sb2a7cj9x1uc3kqfuowz9angm 2432540 2432539 2026-07-01T19:28:28Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage 2432540 wikitext text/x-wiki 85eeopamsv9xtzt6qkn2rya1ug6cqqc 2432541 2432540 2026-07-01T19:42:12Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2080.codfw.wmnet with OS trixie 2432541 wikitext text/x-wiki ov2poyi52120jnyfwrn3k3z1a42f4px 2432542 2432541 2026-07-01T19:43:43Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002" 2432542 wikitext text/x-wiki ntny8w3btfneoqmokbo5kkme9netb8y 2432543 2432542 2026-07-01T19:44:14Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002" 2432543 wikitext text/x-wiki avr46a5th3p3zslds7halwsau5r5o1e 2432544 2432543 2026-07-01T19:44:16Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie 2432544 wikitext text/x-wiki l1309fzpzchgw0ji20h37ybt3v9yufg 2432545 2432544 2026-07-01T19:46:44Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2093.codfw.wmnet with OS trixie 2432545 wikitext text/x-wiki sbks42gtd0t1ztjxcbwuxxw2q5hr1ie 2432546 2432545 2026-07-01T19:59:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2108.codfw.wmnet with OS trixie 2432546 wikitext text/x-wiki dpyphd48u6p6m50oa6i5ouhil2xq5h0 2432549 2432546 2026-07-01T20:19:40Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage 2432549 wikitext text/x-wiki m4uz0pggq612ddlotsx9ir6jw8cu5wv 2432550 2432549 2026-07-01T20:24:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage 2432550 wikitext text/x-wiki kybfa36ks5oxgdqjlqp2o4h8h8l0i8w 2432551 2432550 2026-07-01T20:28:34Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2081.codfw.wmnet with OS trixie 2432551 wikitext text/x-wiki 0oelsy93m77ao8no4cswwmz6fjtez8o 2432552 2432551 2026-07-01T20:45:55Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage 2432552 wikitext text/x-wiki b9ff9534f1htrprfdpf5qlj6xus8afj 2432553 2432552 2026-07-01T20:50:19Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage 2432553 wikitext text/x-wiki 9oh3bu1m5dz27k8m7yip457lzcszzxl 2432554 2432553 2026-07-01T20:50:47Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2108.codfw.wmnet with OS trixie 2432554 wikitext text/x-wiki go8u4jxlszpwsaty0d6y23i3r7ouolq 2432555 2432554 2026-07-01T21:15:24Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2081.codfw.wmnet with OS trixie 2432555 wikitext text/x-wiki blim7h11sd6mzb2weoyroi0pjxa4271 2432556 2432555 2026-07-01T21:19:48Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2084.codfw.wmnet with OS trixie 2432556 wikitext text/x-wiki 76nsqgobowz0c2x77bgdlb6xa6z1ext 2432557 2432556 2026-07-01T21:34:21Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2003.codfw.wmnet with OS bookworm 2432557 wikitext text/x-wiki 9slx8a4a3nf7ygxf7mqwqmtn3lyg1tl 2432558 2432557 2026-07-01T21:34:40Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2003 2432558 wikitext text/x-wiki 6m8eiqu5m7g44jrscu3u5d93xkml4vz 2432559 2432558 2026-07-01T21:35:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2432559 wikitext text/x-wiki neeyziv4ltone412fdykzlqqxlxr8zg 2432560 2432559 2026-07-01T21:36:57Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage 2432560 wikitext text/x-wiki 8ja7bjgvxp1y6z5256lytsx2tgwsxzd 2432561 2432560 2026-07-01T21:42:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003" 2432561 wikitext text/x-wiki 60emvobc50lhz5md0be9ymosucnbwic 2432562 2432561 2026-07-01T21:42:36Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003" 2432562 wikitext text/x-wiki 7ujrticulwozms72c7hc9qtxlq0vobj 2432563 2432562 2026-07-01T21:42:36Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2432563 wikitext text/x-wiki rvm1fpp5k150t6h4y44spwld2k2d05f 2432564 2432563 2026-07-01T21:42:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2432564 wikitext text/x-wiki mqcwtzqgs1y0j6zdvbingro3l8xtdde 2432565 2432564 2026-07-01T21:42:40Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2432565 wikitext text/x-wiki ab4tlfahk1zap88tio08fcgt610xbev 2432566 2432565 2026-07-01T21:42:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2003 2432566 wikitext text/x-wiki ripdq0p89uepvrw6ydx51iydscaphmr 2432567 2432566 2026-07-01T21:42:53Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2003 2432567 wikitext text/x-wiki jy5fapsrt3cyhc023wanr73sdnxhp3f 2432568 2432567 2026-07-01T21:42:54Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2003 2432568 wikitext text/x-wiki 03uadju7h9impowbjva9m8aobvnl6uf 2432569 2432568 2026-07-01T21:43:12Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage 2432569 wikitext text/x-wiki nwfxezei9b3j7pl140dxjbzlrb79fvd 2432570 2432569 2026-07-01T21:50:36Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie 2432570 wikitext text/x-wiki a2amad9pqmg1iw5018mjrcbbedm69xq 2432571 2432570 2026-07-01T22:01:58Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage 2432571 wikitext text/x-wiki guenotozw148076rhgjstck6d4kqcim 2432572 2432571 2026-07-01T22:03:36Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage 2432572 wikitext text/x-wiki 022rpgqejrsjf4ptoei4dvmup9drw3w 2432573 2432572 2026-07-01T22:09:13Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage 2432573 wikitext text/x-wiki 982eyu4keuecq254hyurkk85uqmvzya 2432574 2432573 2026-07-01T22:10:20Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2084.codfw.wmnet with OS trixie 2432574 wikitext text/x-wiki s6g0duidqn190jqgygib1lvpxu8ykvx 2432575 2432574 2026-07-01T22:13:00Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage 2432575 wikitext text/x-wiki 3g8z12u2ggss7rm550cy5owaicmzn5c 2432576 2432575 2026-07-01T22:29:16Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie 2432576 wikitext text/x-wiki iqhttmkw62cw5bh2ctpvwkt4qqx6n9k 2432577 2432576 2026-07-01T22:37:08Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2003.codfw.wmnet with OS bookworm 2432577 wikitext text/x-wiki qrp56bqg6oqfwfo2lv3uue35phtt069 2432580 2432577 2026-07-01T23:50:59Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2432580 wikitext text/x-wiki gkev77dknnc58582in08fh801h259sn 2432581 2432580 2026-07-01T23:51:29Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2432581 wikitext text/x-wiki mjwi0lad62x2jel4fgto89qvpt0vwx3 2432582 2432581 2026-07-01T23:51:30Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2432582 wikitext text/x-wiki 9s1aalj4k4sayq34czzub7677huwdaq 2432583 2432582 2026-07-01T23:52:01Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2432583 wikitext text/x-wiki mu6ybxp2dvszc5tfuh9qh5p5qb4wr1b 2432584 2432583 2026-07-01T23:59:01Z Stashbot 7414 ryankemper: T429844 [opensearch] stopped `opensearch_1@production-search-codfw` on `cirrussearch2111` after chi cluster-manager election churn following `voting_config_exclusions` POST; hoping this triggers a re-election 2432584 wikitext text/x-wiki rweua848mwuxlbrupwft8clbearae3s 2432586 2432584 2026-07-02T00:00:59Z Stashbot 7414 ryankemper: T429844 [opensearch] chi cluster recovered after stopping `opensearch_1@production-search-codfw` on `cirrussearch2111` 2432586 wikitext text/x-wiki g1arjkd4oh2smp6gxxl5mkyxcro1c8g 2432587 2432586 2026-07-02T00:01:52Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2111.codfw.wmnet with OS trixie 2432587 wikitext text/x-wiki 86vtbb1qrmqs16o38jv5bty9mgw1onp 2432588 2432587 2026-07-02T00:22:18Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage 2432588 wikitext text/x-wiki mdyojxxti5t31ovvmk3ec5n1vyyjf22 2432589 2432588 2026-07-02T00:29:24Z Stashbot 7414 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw 2432589 wikitext text/x-wiki 59faiv7p1laqo5j9lg893xzmuerv1w4 2432590 2432589 2026-07-02T00:29:25Z Stashbot 7414 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw 2432590 wikitext text/x-wiki 5h434ob51g1ij2c9r55vjwttmaz0dr7 2432591 2432590 2026-07-02T00:29:26Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage 2432591 wikitext text/x-wiki 5b9vxivnj9ci8b49f0z7n8oe9q3akvy 2432592 2432591 2026-07-02T00:29:41Z Stashbot 7414 ryankemper: T429844 [opensearch] depooled codfw search-omega/search-psi discovery records to match existing codfw search depool during OpenSearch 2.19 migration 2432592 wikitext text/x-wiki tecvns8z39090o5gcdg0c3usbdnffil 2432593 2432592 2026-07-02T00:57:40Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2111.codfw.wmnet with OS trixie 2432593 wikitext text/x-wiki 6pfxqfadwizd9jad864v15yn8g2nn2f 2432594 2432593 2026-07-02T01:16:40Z Stashbot 7414 ryankemper: T429844 [opensearch] completed `cirrussearch2111` reimage; all codfw search clusters are green, all nodes now report `OpenSearch 2.19.5`, and the temporary chi voting exclusion has been removed 2432594 wikitext text/x-wiki gn8gak6kiufi3r8vz9jeudwjq3bb51e 2432597 2432594 2026-07-02T02:00:50Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2432597 wikitext text/x-wiki g2ayoo4o4ij4wfgkqtew36hf8qa088z 2432598 2432597 2026-07-02T02:07:50Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 59s) 2432598 wikitext text/x-wiki jutvangzwufpmiqqi9idg1msl4umahe 2432601 2432598 2026-07-02T04:36:26Z Stashbot 7414 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1298928|CommonSettings: Set $wgScoreUseSvg = true (T49578)]] 2432601 wikitext text/x-wiki n0e52z8d7u1j73jdccmjp6nkmwzpg3z 2432602 2432601 2026-07-02T04:38:48Z Stashbot 7414 tstarling@deploy1003: tstarling, reedy: Backport for [[gerrit:1298928|CommonSettings: Set $wgScoreUseSvg = true (T49578)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432602 wikitext text/x-wiki 99onq4h8xmdjsqzw2vr8wkbkl5bmn4e 2432603 2432602 2026-07-02T04:41:08Z Stashbot 7414 tstarling@deploy1003: tstarling, reedy: Continuing with deployment 2432603 wikitext text/x-wiki 4srwezacc0y1cbqbygmrmfmvpcz4lvr 2432604 2432603 2026-07-02T04:45:34Z Stashbot 7414 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298928|CommonSettings: Set $wgScoreUseSvg = true (T49578)]] (duration: 09m 08s) 2432604 wikitext text/x-wiki 667zyhrfwmpbvc9y1dzjb0pg54o2xe4 2432605 2432604 2026-07-02T05:32:06Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: m3 master switchover T430158 2432605 wikitext text/x-wiki n78qsapn5emjy79cacz85l5nwjwwpel 2432606 2432605 2026-07-02T05:39:32Z Stashbot 7414 marostegui: Failover m3 (phabricator) from db1250 to db1228 - T430158 2432606 wikitext text/x-wiki 2mlx65yuodg4d9ila6nmfe8attooq2i 2432607 2432606 2026-07-02T05:44:06Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1250.eqiad.wmnet with reason: m3 master switchover T430158 2432607 wikitext text/x-wiki 93ewrx3saje7na9cwykao1wzf95jhh4 2432608 2432607 2026-07-02T05:55:25Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1250.eqiad.wmnet with OS trixie 2432608 wikitext text/x-wiki hc6qjh0xn8m518pjf236u1x3yexk7vr 2432609 2432608 2026-07-02T05:59:01Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 T430817 2432609 wikitext text/x-wiki 4mld6fsbds0dsnd7wqy5bs2czfx9fpx 2432610 2432609 2026-07-02T05:59:30Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1160 with weight 0 T430817', diff saved to https://phabricator.wikimedia.org/P94673 and previous config saved to /var/cache/conftool/dbconfig/20260702-055927-cwilliams.json 2432610 wikitext text/x-wiki ne336w4nf30xx4aod6gmx6iuzavj95l 2432612 2432610 2026-07-02T06:04:51Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet 2432612 wikitext text/x-wiki bm0lo1v3f1xs4lizg7lxovzd9u3zs50 2432613 2432612 2026-07-02T06:06:51Z Stashbot 7414 cezmunsta: Starting s4 eqiad failover from db1244 to db1160 - T430817 2432613 wikitext text/x-wiki bl7xeowop7ah2xinee8iubj5z6836jl 2432614 2432613 2026-07-02T06:07:07Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - T430817', diff saved to https://phabricator.wikimedia.org/P94674 and previous config saved to /var/cache/conftool/dbconfig/20260702-060704-cwilliams.json 2432614 wikitext text/x-wiki 3we563et856bbs5ie1rjo3dy2vdd8dt 2432615 2432614 2026-07-02T06:07:49Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1160 to s4 primary and set section read-write T430817', diff saved to https://phabricator.wikimedia.org/P94675 and previous config saved to /var/cache/conftool/dbconfig/20260702-060746-cwilliams.json 2432615 wikitext text/x-wiki r5hifdsqyorz31liq73o9bjculgjgpj 2432616 2432615 2026-07-02T06:08:12Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet 2432616 wikitext text/x-wiki ifmqpc3wkunyatogdwuriuwunz8gi9o 2432617 2432616 2026-07-02T06:09:37Z Stashbot 7414 cwilliams@dns1006: START - running authdns-update 2432617 wikitext text/x-wiki 9untd4bfonjldjts6zdvxkq6aio7fhm 2432618 2432617 2026-07-02T06:09:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage 2432618 wikitext text/x-wiki eme5wzs53ide8zwpyi3myawsih9rxmu 2432619 2432618 2026-07-02T06:11:02Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1244 T430817', diff saved to https://phabricator.wikimedia.org/P94676 and previous config saved to /var/cache/conftool/dbconfig/20260702-061059-cwilliams.json 2432619 wikitext text/x-wiki raw0xiu83mjmliqwyzqde39zr585vpu 2432620 2432619 2026-07-02T06:11:38Z Stashbot 7414 cwilliams@dns1006: END - running authdns-update 2432620 wikitext text/x-wiki 03sa2vwm4y844jvva1nb3w18ysmtfq2 2432621 2432620 2026-07-02T06:12:20Z Stashbot 7414 cwilliams@dns1006: START - running authdns-update 2432621 wikitext text/x-wiki d8ckrl55mh539kkqs8m8j29lq3k0jlo 2432622 2432621 2026-07-02T06:14:30Z Stashbot 7414 cwilliams@dns1006: END - running authdns-update 2432622 wikitext text/x-wiki fc5isn39s3lfgdytkw17638jqpkt3w2 2432623 2432622 2026-07-02T06:15:31Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage 2432623 wikitext text/x-wiki c2lw1g4kyjilexio23kvb6vstendwku 2432624 2432623 2026-07-02T06:25:36Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2432624 wikitext text/x-wiki g7oglulxorkivfjei73koo55jkby5ft 2432625 2432624 2026-07-02T06:25:39Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@eqiad T429893 2432625 wikitext text/x-wiki tc0walxku396zn026ls43q912y2i6rq 2432626 2432625 2026-07-02T06:25:45Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1244: Upgrading db1244.eqiad.wmnet 2432626 wikitext text/x-wiki jm1sq56b5j8ovi4k0hnmc5zxev5kq9z 2432627 2432626 2026-07-02T06:25:56Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1244: Upgrading db1244.eqiad.wmnet 2432627 wikitext text/x-wiki egnc7ewfzwsc1jeeg7f1v37kvwr3byr 2432628 2432627 2026-07-02T06:34:53Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1244.eqiad.wmnet with OS trixie 2432628 wikitext text/x-wiki sz0ov2evkt000wy4ctyt205s7fc5xy1 2432629 2432628 2026-07-02T06:38:32Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1250.eqiad.wmnet with OS trixie 2432629 wikitext text/x-wiki 5v33eb2sy1bdw4ca9bczlum6mwzojnw 2432630 2432629 2026-07-02T06:50:16Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage 2432630 wikitext text/x-wiki 9nbj75sjbudtfypcxbopg2avwn4j3c0 2432631 2432630 2026-07-02T06:54:34Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage 2432631 wikitext text/x-wiki oslqsspq3l4kmg47sma4tj3ickch4l6 2432632 2432631 2026-07-02T07:09:22Z Stashbot 7414 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1306970|Fix how to check the treatment group (T415904)]], [[gerrit:1306971|Fix how to check the treatment group (T415904)]] 2432632 wikitext text/x-wiki i0koclaejvwcgqc5ejm5iliuufabu88 2432633 2432632 2026-07-02T07:11:30Z Stashbot 7414 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1306970|Fix how to check the treatment group (T415904)]], [[gerrit:1306971|Fix how to check the treatment group (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432633 wikitext text/x-wiki m1arxhre8te32jwfi49mp0usjcydenn 2432634 2432633 2026-07-02T07:11:53Z Stashbot 7414 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment 2432634 wikitext text/x-wiki 9ejd91jyzub924em07djv6glu4dn7xp 2432636 2432634 2026-07-02T07:13:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1244.eqiad.wmnet with OS trixie 2432636 wikitext text/x-wiki ofdi8f66tvuosekh49d44mj17zem8ir 2432637 2432636 2026-07-02T07:16:17Z Stashbot 7414 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306970|Fix how to check the treatment group (T415904)]], [[gerrit:1306971|Fix how to check the treatment group (T415904)]] (duration: 06m 55s) 2432637 wikitext text/x-wiki 48l2kfmkklfgzwipwq1wvb4x9z1c5c3 2432638 2432637 2026-07-02T07:22:59Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1306985|Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] 2432638 wikitext text/x-wiki d00ky2z4lsiusjxkp6zrmk83pn37a2s 2432639 2432638 2026-07-02T07:23:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1244: Migration of db1244.eqiad.wmnet completed 2432639 wikitext text/x-wiki q1d3c6n5nocs178gnw3z9airahf396e 2432640 2432639 2026-07-02T07:25:08Z Stashbot 7414 cscott@deploy1003: ssastry, cscott: Backport for [[gerrit:1306985|Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432640 wikitext text/x-wiki cez61iklej1u9pb7mywz62vff6gdt71 2432641 2432640 2026-07-02T07:26:07Z Stashbot 7414 cscott@deploy1003: ssastry, cscott: Continuing with deployment 2432641 wikitext text/x-wiki ot5c1l9sza2dd90hqty42jat8ap78q9 2432643 2432641 2026-07-02T07:30:28Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306985|Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] (duration: 07m 28s) 2432643 wikitext text/x-wiki eq69rp0oox48j4mnkxlqmty16bd60iw 2432644 2432643 2026-07-02T07:39:52Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org 2432644 wikitext text/x-wiki 0s9856qjmcfzqknhnbb9gfri9lhb23c 2432647 2432644 2026-07-02T07:42:51Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307058|[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996|[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] 2432647 wikitext text/x-wiki nt1q6ayfrth0jbbrugr8ol8rs43dua7 2432648 2432647 2026-07-02T07:44:15Z Stashbot 7414 moritzm: installing node-lodash security updates 2432648 wikitext text/x-wiki t3vivwrtmx88xxthdnqpai292imbewd 2432649 2432648 2026-07-02T07:44:22Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org 2432649 wikitext text/x-wiki 6pe9586bnuo20a7fsl23wlav9ad17ur 2432650 2432649 2026-07-02T07:44:56Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1307058|[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996|[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432650 wikitext text/x-wiki ic1vntw1urrvdoyya4ysj43ju3hq4rx 2432651 2432650 2026-07-02T07:49:41Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432651 wikitext text/x-wiki qcvz9zn7qtdrlimp5u4p1kjbemlwhbf 2432652 2432651 2026-07-02T07:54:46Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2432652 wikitext text/x-wiki tg64wcp8c3libwgkc14g3v0dc4nyvwe 2432653 2432652 2026-07-02T07:54:58Z Stashbot 7414 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2432653 wikitext text/x-wiki sq2zw8sk6p3mni7v6p804ooxftq8pdz 2432654 2432653 2026-07-02T07:54:59Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2006.wikimedia.org 2432654 wikitext text/x-wiki dgzu84j78xa080dwjpew7e9gpvikfci 2432655 2432654 2026-07-02T07:55:17Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2432655 wikitext text/x-wiki cimukzpb5p7in53dr72xlt6t6e4aerr 2432656 2432655 2026-07-02T07:55:28Z Stashbot 7414 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2432656 wikitext text/x-wiki cyz90yzwq758ki3l3th1h8oyv2np9b3 2432657 2432656 2026-07-02T07:55:50Z Stashbot 7414 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2432657 wikitext text/x-wiki i2u1bbutmugfqxdswxxaj9ixrekt3hn 2432658 2432657 2026-07-02T07:56:02Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2432658 wikitext text/x-wiki ifui5bd78h201bzv39czeansf5ni8ti 2432659 2432658 2026-07-02T07:56:13Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2432659 wikitext text/x-wiki qlkdvebi4ek7bue3pxv6sl6c7cmdii9 2432660 2432659 2026-07-02T07:56:46Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2432660 wikitext text/x-wiki 9irxdn9tqi6vadjizknza7wy0vtmvnq 2432661 2432660 2026-07-02T07:57:25Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2432661 wikitext text/x-wiki 04kl56bypjoas4bsy6rvngvp3g8lj5g 2432662 2432661 2026-07-02T07:58:08Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2432662 wikitext text/x-wiki j5u0gif2tprkljj9rfc954xg8x240zh 2432664 2432662 2026-07-02T07:59:27Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2006.wikimedia.org 2432664 wikitext text/x-wiki jmw3mib32iovheb37rv0194lefb4697 2432665 2432664 2026-07-02T07:59:33Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2432665 wikitext text/x-wiki fboy46ivpn3f8ouup8monm73qymy1pu 2432666 2432665 2026-07-02T07:59:42Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2432666 wikitext text/x-wiki 82xmmifssvurtzmoyu026rn3n5y64m4 2432667 2432666 2026-07-02T07:59:51Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432667 wikitext text/x-wiki 70uiim1knt7cf7g24h5kolqbahfumak 2432668 2432667 2026-07-02T08:00:02Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2432668 wikitext text/x-wiki 9zn2g1euanv459nz48lkxxn9tnzfzed 2432669 2432668 2026-07-02T08:00:10Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2432669 wikitext text/x-wiki 5jsjohlie06z383eeooo6sqiyv6yifm 2432670 2432669 2026-07-02T08:00:43Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2432670 wikitext text/x-wiki p7f9rbl6jl2jmsbhuadi8svq1m8vpb8 2432671 2432670 2026-07-02T08:00:57Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2432671 wikitext text/x-wiki 1ricocdjum6gpy39t3yttn27049bc6x 2432672 2432671 2026-07-02T08:01:19Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2432672 wikitext text/x-wiki r7wllddog2ue82h085xvmoqtdrqmn4y 2432673 2432672 2026-07-02T08:01:30Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2432673 wikitext text/x-wiki kgq9dcl41zulaypya93b6hg1prot1he 2432674 2432673 2026-07-02T08:01:48Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307058|[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996|[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] (duration: 18m 58s) 2432674 wikitext text/x-wiki lj9tvt71owt609o8jz8kcog56c4oia4 2432675 2432674 2026-07-02T08:02:11Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2432675 wikitext text/x-wiki 6mq84aao7edubijnr1644hpkjpaup7j 2432676 2432675 2026-07-02T08:02:20Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2432676 wikitext text/x-wiki 0l6xqbfpvctkd98tqujzgi7u66bkmtq 2432677 2432676 2026-07-02T08:08:37Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1244: Migration of db1244.eqiad.wmnet completed 2432677 wikitext text/x-wiki 61yomptxi3geo9p0rjsvg6pdfdtlzw5 2432678 2432677 2026-07-02T08:08:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2432678 wikitext text/x-wiki mmrldz2lslz8j3ruti540n5t9cadzcs 2432679 2432678 2026-07-02T08:14:08Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307059|Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061|Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] 2432679 wikitext text/x-wiki r7bj2uqhc311exear6bner1xdo6vxfi 2432680 2432679 2026-07-02T08:16:11Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1307059|Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061|Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432680 wikitext text/x-wiki ebnwmuomz71krz667mm0gqf0kkalt37 2432682 2432680 2026-07-02T08:21:27Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2432682 wikitext text/x-wiki ahlmuf6z160koo96f2anmhta0uzjpat 2432683 2432682 2026-07-02T08:25:50Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307059|Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061|Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] (duration: 11m 44s) 2432683 wikitext text/x-wiki 0zn5lpnufepjpltnqjg7strvos4kogs 2432686 2432683 2026-07-02T08:40:43Z Stashbot 7414 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.9 refs T423918 2432686 wikitext text/x-wiki 7d4nl0fl65gco569ksv140s7f1e410j 2432689 2432686 2026-07-02T08:43:36Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432689 wikitext text/x-wiki k715q1hfeybry7jwlz59rx5jpw61ied 2432690 2432689 2026-07-02T08:52:30Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance 2432690 wikitext text/x-wiki m4eixsx0n3e3xgd7um3iq0to04l6n3x 2432691 2432690 2026-07-02T08:52:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2220 (T426633)', diff saved to https://phabricator.wikimedia.org/P94681 and previous config saved to /var/cache/conftool/dbconfig/20260702-085237-fceratto.json 2432691 wikitext text/x-wiki 0tm0uoue12s5myjp4yiqpuv0cyu8yp9 2432692 2432691 2026-07-02T08:52:44Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432692 wikitext text/x-wiki edgqka3zvnm2vzrgpbkmdcvdcmwon96 2432693 2432692 2026-07-02T08:52:56Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432693 wikitext text/x-wiki q64o8xy4x8tyo8clivkncwq5s0sjo5w 2432694 2432693 2026-07-02T08:54:25Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307075|build: Update required Node version from 24.14.1 to 24.18.0]] 2432694 wikitext text/x-wiki qvs5iwwc23j9kp6o693669taily68ck 2432697 2432694 2026-07-02T08:56:17Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307075|build: Update required Node version from 24.14.1 to 24.18.0]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432697 wikitext text/x-wiki bl3t2nqe8fcmfc5o0f76cm4bqv2dz0u 2432698 2432697 2026-07-02T08:57:10Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2432698 wikitext text/x-wiki atq38jtj8asn9kheqxsqpkmjurmcm7p 2432699 2432698 2026-07-02T08:59:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T426633)', diff saved to https://phabricator.wikimedia.org/P94682 and previous config saved to /var/cache/conftool/dbconfig/20260702-085942-fceratto.json 2432699 wikitext text/x-wiki ifmgl6kk9pxl7b7g1j2rv8w3wu5r9yv 2432700 2432699 2026-07-02T09:01:32Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307075|build: Update required Node version from 24.14.1 to 24.18.0]] (duration: 07m 07s) 2432700 wikitext text/x-wiki ke8djoz182hgyf5glt1v00w41eeamzp 2432701 2432700 2026-07-02T09:03:06Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432701 wikitext text/x-wiki rl0ny6gb69zrrp41p18kv73r823sogy 2432702 2432701 2026-07-02T09:09:52Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307076|SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] 2432702 wikitext text/x-wiki tnkjbdz228htq35970n6u3nyslxycgo 2432703 2432702 2026-07-02T09:09:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94683 and previous config saved to /var/cache/conftool/dbconfig/20260702-090950-fceratto.json 2432703 wikitext text/x-wiki ok3spx3tgnt0cx1g7vc8z3rbhbh6hq0 2432704 2432703 2026-07-02T09:11:43Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307076|SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432704 wikitext text/x-wiki cpy6cppy1te7dllp1mrzvx4iamvlvpc 2432705 2432704 2026-07-02T09:12:32Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2432705 wikitext text/x-wiki 9c271qshlm3aokps3nfc740g73713oe 2432706 2432705 2026-07-02T09:13:25Z Stashbot 7414 moritzm: installing libgcrypt20 security updates 2432706 wikitext text/x-wiki njjbs3ks460lxg97ei0wqgq0t954ayo 2432708 2432706 2026-07-02T09:16:49Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307076|SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] (duration: 06m 57s) 2432708 wikitext text/x-wiki r2rlh8ysjz2ohy83a88jcgc8vz99sav 2432709 2432708 2026-07-02T09:19:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94684 and previous config saved to /var/cache/conftool/dbconfig/20260702-091957-fceratto.json 2432709 wikitext text/x-wiki b07s2wb0hk397kqw7itl3c47u0uxiuc 2432710 2432709 2026-07-02T09:24:38Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s5 T430923 2432710 wikitext text/x-wiki idjzuwkc6bhk4bmwkx1ma4dzrjerguo 2432711 2432710 2026-07-02T09:24:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2192 with weight 0 T430923', diff saved to https://phabricator.wikimedia.org/P94685 and previous config saved to /var/cache/conftool/dbconfig/20260702-092455-fceratto.json 2432711 wikitext text/x-wiki detw3806lc4pa8crumenfyxwq7c3g61 2432713 2432711 2026-07-02T09:30:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T426633)', diff saved to https://phabricator.wikimedia.org/P94686 and previous config saved to /var/cache/conftool/dbconfig/20260702-093004-fceratto.json 2432713 wikitext text/x-wiki b33hw73konslj4mjnpxqtbxev88s9vr 2432716 2432713 2026-07-02T09:36:06Z Stashbot 7414 federico3: Starting s5 codfw failover from db2213 to db2192 - T430923 2432716 wikitext text/x-wiki 7sx7ucyqs53es2lwpiuwsnm8r84ymwa 2432717 2432716 2026-07-02T09:36:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2192 to s5 primary T430923', diff saved to https://phabricator.wikimedia.org/P94687 and previous config saved to /var/cache/conftool/dbconfig/20260702-093650-fceratto.json 2432717 wikitext text/x-wiki 1zb2luej2405pqe63kojksu7teuaoxn 2432718 2432717 2026-07-02T09:39:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2213 T430923', diff saved to https://phabricator.wikimedia.org/P94688 and previous config saved to /var/cache/conftool/dbconfig/20260702-093859-fceratto.json 2432718 wikitext text/x-wiki kk3vy69d8hn8n2283z14ci5nxff2ab9 2432719 2432718 2026-07-02T09:39:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover 2432719 wikitext text/x-wiki oqch7vwd8zydbc8g2asdri44qfwg12e 2432720 2432719 2026-07-02T09:44:41Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2213: Repooling after switchover 2432720 wikitext text/x-wiki 8m8ggroux6a2vprccr4adzjpfqwpwf8 2432721 2432720 2026-07-02T09:51:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover 2432721 wikitext text/x-wiki 70ec3w79shyihyhl3kijmhuntetw41b 2432722 2432721 2026-07-02T09:53:00Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2213: Repooling after switchover 2432722 wikitext text/x-wiki jpxhg5oqosus0kwspjlrasskgjiiw5v 2432723 2432722 2026-07-02T09:54:23Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432723 wikitext text/x-wiki f1lqpa7xief0g3gl7ka1jh3he0a3db4 2432724 2432723 2026-07-02T09:55:22Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance 2432724 wikitext text/x-wiki 3p96qqnw14nlhnhtyapijmy7vhnly7h 2432725 2432724 2026-07-02T09:55:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2213 (T426633)', diff saved to https://phabricator.wikimedia.org/P94690 and previous config saved to /var/cache/conftool/dbconfig/20260702-095529-fceratto.json 2432725 wikitext text/x-wiki 8rz6nef6bc6ob81xmcgt58whvxnc0u1 2432727 2432725 2026-07-02T10:01:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T426633)', diff saved to https://phabricator.wikimedia.org/P94691 and previous config saved to /var/cache/conftool/dbconfig/20260702-100122-fceratto.json 2432727 wikitext text/x-wiki nndxf667b4yaxdnlqi1wd8o6xucmsoc 2432728 2432727 2026-07-02T10:03:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2432728 wikitext text/x-wiki fmg90oe3sqg6hdhzevzvnwk4w4t8a73 2432729 2432728 2026-07-02T10:03:57Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts es1033.eqiad.wmnet 2432729 wikitext text/x-wiki nm5axm0fynmgmsgppwjvra0ipdatu3n 2432730 2432729 2026-07-02T10:04:34Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432730 wikitext text/x-wiki pcbizkrobgaljwtgmv7icwji0nm7qtr 2432731 2432730 2026-07-02T10:10:12Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2432731 wikitext text/x-wiki 1futlblm9byo5ktffe0iewetn8ul9ru 2432732 2432731 2026-07-02T10:10:44Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432732 wikitext text/x-wiki q1y6jkhufqxqodnuuonym4n46jxhumi 2432733 2432732 2026-07-02T10:11:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94693 and previous config saved to /var/cache/conftool/dbconfig/20260702-101130-fceratto.json 2432733 wikitext text/x-wiki 64k8vpb81ciibtq0yuktc8gwuyy6wje 2432734 2432733 2026-07-02T10:12:23Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling 2432734 wikitext text/x-wiki d7m8kldm8pwqad8lt80a4ohlq392zt6 2432735 2432734 2026-07-02T10:12:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2214.codfw.wmnet 2432735 wikitext text/x-wiki bwpj3ol2aahpfbfywy8bf6c4jb5mzpg 2432736 2432735 2026-07-02T10:12:55Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2214.codfw.wmnet 2432736 wikitext text/x-wiki lsa8y3j8n2p2ewp42vsc5udsf1ycxge 2432737 2432736 2026-07-02T10:13:34Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for clouddb1017.eqiad.wmnet 2432737 wikitext text/x-wiki quxv103dkzd24wabhhbhimx3pp9pdz6 2432738 2432737 2026-07-02T10:14:25Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2432738 wikitext text/x-wiki j4lxmbimtw8h4d7uz6p8ayor19363bv 2432739 2432738 2026-07-02T10:14:58Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2432739 wikitext text/x-wiki 7zx8sb9z3p8ib6ufduj2eaebujfzdlw 2432740 2432739 2026-07-02T10:14:58Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2432740 wikitext text/x-wiki 98hp0rs806ozgjo4pfav9sca6wx0lcl 2432741 2432740 2026-07-02T10:14:59Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1033.eqiad.wmnet 2432741 wikitext text/x-wiki h9gsce9mthia2e3j4hp5kd579uxy7au 2432742 2432741 2026-07-02T10:18:08Z Stashbot 7414 fceratto@cumin1003: Removing es1033 from zarcillo T408772 2432742 wikitext text/x-wiki j2ha8l30j8lw00caehgv9ofszdjw9ww 2432743 2432742 2026-07-02T10:18:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) 2432743 wikitext text/x-wiki 3bes2ckwmu0kp4asqba874vlnfhn8hz 2432745 2432743 2026-07-02T10:19:54Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for clouddb1017.eqiad.wmnet 2432745 wikitext text/x-wiki kvf579f8d6lak8lntr58zrmqi0rs1ql 2432746 2432745 2026-07-02T10:20:51Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432746 wikitext text/x-wiki 794z530xbmlafqnkpmeezcjtbcwdj1e 2432747 2432746 2026-07-02T10:21:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94696 and previous config saved to /var/cache/conftool/dbconfig/20260702-102137-fceratto.json 2432747 wikitext text/x-wiki l2fr5b3tgkvcl790zmte4rnr9ie7ate 2432748 2432747 2026-07-02T10:31:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T426633)', diff saved to https://phabricator.wikimedia.org/P94698 and previous config saved to /var/cache/conftool/dbconfig/20260702-103146-fceratto.json 2432748 wikitext text/x-wiki qjucb1m2sy9ksf8ciib96z8k1ekoqdc 2432749 2432748 2026-07-02T10:47:04Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2432749 wikitext text/x-wiki oxiuazp59oa8gluvkfukwxrnrx9qlbb 2432752 2432749 2026-07-02T10:49:10Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2432752 wikitext text/x-wiki smdhn7l83khchjfxp6j5uf0vf81esy5 2432755 2432752 2026-07-02T10:57:51Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2214: Repooling 2432755 wikitext text/x-wiki 4wdds6r7w9icb178panshzdh7hszp7x 2432756 2432755 2026-07-02T11:29:19Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges 2432756 wikitext text/x-wiki 36dg236absgyirfmcgt8w108vhg7zx9 2432757 2432756 2026-07-02T11:29:21Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) 2432757 wikitext text/x-wiki hy1yfrrcttr3zw5sdv00z6f21viliaj 2432758 2432757 2026-07-02T11:30:45Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet 2432758 wikitext text/x-wiki 1p9jfh1vep2f1zexm2674w06fprahww 2432759 2432758 2026-07-02T11:36:45Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet 2432759 wikitext text/x-wiki rz67mqou8a6lxefganccq862xnzselc 2432760 2432759 2026-07-02T11:37:27Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet 2432760 wikitext text/x-wiki orefvcwtfv0q6htuh1tlx78jqujcd9z 2432761 2432760 2026-07-02T11:39:29Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.decommission for hosts wdqs-categories1001.eqiad.wmnet 2432761 wikitext text/x-wiki sugnbdhkwhtrptw6vljb9qdfw9gpkoj 2432762 2432761 2026-07-02T11:41:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet 2432762 wikitext text/x-wiki do9g9im5wo21kq90srwctin9kt2zeel 2432763 2432762 2026-07-02T11:42:08Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges 2432763 wikitext text/x-wiki 8ptf5eyckt7edagu0mt20kgqio7fs03 2432764 2432763 2026-07-02T11:42:10Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) 2432764 wikitext text/x-wiki o7a0zst22v7t9nby6uqzw8jtvu297po 2432765 2432764 2026-07-02T11:44:18Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2432765 wikitext text/x-wiki 683sde864zq3f9ciu8f6o0z5w00qkgv 2432766 2432765 2026-07-02T11:51:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" 2432766 wikitext text/x-wiki jomxdo0sicq8ooxggn6bm2agrhs422w 0 17290 2432392 2432182 2026-07-01T13:39:12Z Stashbot 7414 hashar: integration: added timestamping to operations-puppet-catalog-compiler and operations-puppet-catalog-compiler-puppet7-test jobs 2432392 wikitext text/x-wiki 1836fdqg5zvr3dojsdoihfcunx609gk 2432750 2432392 2026-07-02T10:47:19Z Stashbot 7414 hashar: zuul1002: running Puppet agent to drop `wikimediacloud.org` from `no_proxy` | T430479 2432750 wikitext text/x-wiki nesj1ga8nyw15wgenob51qqyecb41qw 0 202567 2432502 2414328 2026-07-01T16:22:38Z HNowlan (WMF) 15760 /* Meta-Monitoring */ document downtiming 2432502 wikitext text/x-wiki odwy3q3fo6oiy1181hp524jxgs87627 0 440900 2432751 2431871 2026-07-02T10:48:11Z FGiunchedi (WMF) 39205 /* new cloudvirt install */ hardcoding nova id is no longer needed 2432751 wikitext text/x-wiki meax17y1hznai9tync5kmbkoo4ssstc 498 443628 2432341 2432009 2026-07-01T12:17:49Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53) 2432341 wikitext text/x-wiki favmv960nga1dmt8m3evz3daf4a57kc 2432382 2432341 2026-07-01T13:24:39Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506476 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432382 wikitext text/x-wiki 4wu0rlwcq38he54tp18w5mhdpdurstu 2432446 2432382 2026-07-01T15:00:37Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28525308526 (https://github.com/cluebotng/component-configs/commits/4d35560ba5b135c6dff161ca5a7ed6c393edd180) 2432446 wikitext text/x-wiki 4jxn64gyq0ybnr0j0ls1t08nkkdz9jy 2432523 2432446 2026-07-01T17:42:31Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28536333606 (https://github.com/cluebotng/component-configs/commits/fc7f0d0647c84e0e5c9f02d279015dc3183f3151) 2432523 wikitext text/x-wiki nuuu7228jatl660d15kr999i51iy45d 2432531 2432523 2026-07-01T18:26:51Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28538748543 (https://github.com/cluebotng/component-configs/commits/6c7bcbaa2a10eb01c07a4df9fe5c1ea3b571b351) 2432531 wikitext text/x-wiki ntcfzyfl02r8cv4x4qpcfpc3dtliszb 2432547 2432531 2026-07-01T20:12:03Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28544657461 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861) 2432547 wikitext text/x-wiki mdvz7aka8r39rhju8azcguw3tjrnjsx 498 446211 2432378 2427365 2026-07-01T13:21:39Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506454 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432378 wikitext text/x-wiki k6han8cr2y1chfr6485p42vwtwfexf1 0 449031 2432600 2398747 2026-07-02T03:41:30Z CStone (WMF) 38439 2432600 wikitext text/x-wiki 56srmt61urql1c8mp4qihxa0rytns6l 0 449160 2432350 2432284 2026-07-01T12:54:22Z Dexbot 30554 Bot: Updating the report 2432350 wikitext text/x-wiki rlva314531yy3vdfz06yrfbdnakmws3 2432404 2432350 2026-07-01T14:00:35Z Dexbot 30554 Bot: Updating the report 2432404 wikitext text/x-wiki 5h1r1jds016ffc36uur0pd18fr32lmv 2432585 2432404 2026-07-02T00:00:35Z Dexbot 30554 Bot: Updating the report 2432585 wikitext text/x-wiki tku5slp3itcd2lhx67ty2x83tpjsyo3 2432611 2432585 2026-07-02T06:01:14Z Dexbot 30554 Bot: Updating the report 2432611 wikitext text/x-wiki g9czvewy2hfooy8pgts4ec6wedju6yo 2432696 2432611 2026-07-02T08:55:34Z Dexbot 30554 Bot: Updating the report 2432696 wikitext text/x-wiki b2nkiam81ft3eo5thy8apuiuw34yyd8 2432712 2432696 2026-07-02T09:25:39Z Dexbot 30554 Bot: Updating the report 2432712 wikitext text/x-wiki nieicy8cy1buu9kizjqoitq306l1vrn 2432726 2432712 2026-07-02T09:55:44Z Dexbot 30554 Bot: Updating the report 2432726 wikitext text/x-wiki awuf2pswkj7h7nxa6e8w042rej9l3v3 0 451448 2432521 2429019 2026-07-01T17:14:33Z CMassaro (WMF) 40102 Update kubectl commands 2432521 wikitext text/x-wiki 8wikd25nqmfnhnvdrf2y6gnwzlpy1l8 0 451533 2432353 2432031 2026-07-01T12:57:06Z APuthin-WMF 52425 /* Next meeting agenda */ Updating next agenda & date 2432353 wikitext text/x-wiki 6d1rs0386opw8obvzqwbg4kkcxbnavg 498 452890 2432373 2431189 2026-07-01T13:19:58Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506430 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432373 wikitext text/x-wiki pxcmn8c0j4h73iv5hqkdbt12if12rmq 2432548 2432373 2026-07-01T20:15:02Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28544657044 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861) 2432548 wikitext text/x-wiki 5m68365qlkfu1xxkgt5cekppo772d2x 116 453906 2432436 2432278 2026-07-01T14:15:23Z Gitlabaccountapprovalbot 37332 @vadymts1 was approved. 2432436 wikitext text/x-wiki 0fowtd4r65uvees5se0lu92zr33c9yh 2432454 2432436 2026-07-01T15:12:23Z Gitlabaccountapprovalbot 37332 @mugurolevy was approved. 2432454 wikitext text/x-wiki jd9viedendrrjvojcvg7irmtsaq9kz1 0 455726 2432635 2430086 2026-07-02T07:12:22Z MMuhlenhoff (WMF) 5268 /* Bots still using the legacy setup */ Cluebot fixed 2432635 wikitext text/x-wiki rxynt4dq6ooebkgfdefg02zu9na7h9c 498 458787 2432380 2427413 2026-07-01T13:22:48Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506690 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432380 wikitext text/x-wiki 0mj90l7gvo42ox6udtlrnu9m1ple9jf 498 459243 2432372 2428856 2026-07-01T13:19:19Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432372 wikitext text/x-wiki r52w6zm256etlvfn4iurvlcvy6tbazk 2432445 2432372 2026-07-01T15:00:26Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432445 wikitext text/x-wiki 1caztgrxxfpcujyrtx1hplua87r6aqo 498 459244 2432381 2431190 2026-07-01T13:22:54Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506484 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432381 wikitext text/x-wiki t8ij0waaecgey5zm3lbmprmtb5cheds 498 459255 2432379 2427366 2026-07-01T13:21:59Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506563 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432379 wikitext text/x-wiki hcyok947oav76tdd2u54nb0bp9snfbm 498 459263 2432371 2431187 2026-07-01T13:19:18Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432371 wikitext text/x-wiki butvv7g9ka2b6o5dbau9g2y92vlzqzs 498 459362 2432376 2427409 2026-07-01T13:21:24Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506415 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) 2432376 wikitext text/x-wiki 6r84ankv0kvaxhnof0olorm15wxf9pb 0 459699 2432595 2432126 2026-07-02T01:46:06Z Stashbot 7414 eileen: config revision changed from 92ac127c to d0a8b49f 2432595 wikitext text/x-wiki jk1m9q3l7edlccgiwiqi0be5ig2xhlk 2432596 2432595 2026-07-02T01:54:50Z Stashbot 7414 eileen: * civicrm upgraded from 13584dc4 to 7288da0a 2432596 wikitext text/x-wiki pso2i94vcpuanhm7oxzubrnfaujtjxq 2432599 2432596 2026-07-02T03:36:42Z Stashbot 7414 eileen: civicrm upgraded from 7288da0a to d44caa2a 2432599 wikitext text/x-wiki 6vo0geboywbnr5ia8h0h1lwjsulca28 0 460052 2432401 2432010 2026-07-01T13:58:58Z MPopov (WMF) 5891 MPopov (WMF) moved page [[Data Platform/Systems/Growthbook]] to [[Data Platform/Systems/GrowthBook]]: Incorrect capitalization of software name 2432010 wikitext text/x-wiki 3zs1tld19khioqpy9rf9rkflg9yssyw 0 460232 2432414 2432025 2026-07-01T14:04:27Z MPopov (WMF) 5891 2432414 wikitext text/x-wiki 1jylcfs3iw32hqfmo1gv696ffh0lg2a 0 460234 2432377 2431610 2026-07-01T13:21:26Z MPopov (WMF) 5891 /* Create/schedule an experiment */ 2432377 wikitext text/x-wiki 08gdivy7sf6yn2j7n8ap3f96a4mnwl7 2432383 2432377 2026-07-01T13:26:45Z MPopov (WMF) 5891 Formatting improvements 2432383 wikitext text/x-wiki hxh5r5hyfdlm7jg71d3iofxatbpostp 0 460398 2432351 2432032 2026-07-01T12:55:01Z APuthin-WMF 52425 Formatting + updating attendees + adding phab link 2432351 wikitext text/x-wiki 03kqzb037vu50wau4hfxb74p3fth66u 0 460402 2432402 2026-07-01T13:58:58Z MPopov (WMF) 5891 MPopov (WMF) moved page [[Data Platform/Systems/Growthbook]] to [[Data Platform/Systems/GrowthBook]]: Incorrect capitalization of software name 2432402 wikitext text/x-wiki f5e5uxlkw99fin9b5a6jtj8dqx4iafb 0 460403 2432406 2026-07-01T14:01:00Z MPopov (WMF) 5891 Create redirect 2432406 wikitext text/x-wiki f5e5uxlkw99fin9b5a6jtj8dqx4iafb 3 460404 2432707 2026-07-02T09:16:33Z StrikerBot 8475 Welcome to Toolforge! 2432707 wikitext text/x-wiki lik68osu19og49oncbq16uwa72gfcm3 3 460405 2432714 2026-07-02T09:34:23Z StrikerBot 8475 Welcome to Toolforge! 2432714 wikitext text/x-wiki 12v462k3mf9t0v2xmpvs3oiliowy6uo