Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.15 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2445911 2445030 2026-08-12T12:26:27Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli 2445911 wikitext text/x-wiki oui3tpsv24z7xj3cfdhmsima84blsxh 2445912 2445911 2026-08-12T12:27:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli 2445912 wikitext text/x-wiki c2vxiem5a3cqcjb3ifxpikmqewvakfb 2445950 2445912 2026-08-12T13:15:20Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli 2445950 wikitext text/x-wiki frxadlnm2qe5yisvzgqf4uwo1ehdsqx 2445992 2445950 2026-08-12T13:28:58Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli 2445992 wikitext text/x-wiki so7aglehrt7itw762i69livh0tfhxdc 2446145 2445992 2026-08-12T15:17:03Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2446145 wikitext text/x-wiki ewkag2lr1rr0exxg5jyc8od4jhi73ul 2446161 2446145 2026-08-12T15:26:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2446161 wikitext text/x-wiki mmo4kzvspu23ifuo51cdyzb5uuordu1 2446187 2446161 2026-08-12T15:54:43Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli 2446187 wikitext text/x-wiki tb1p4kjawzwrly6tbpgukm0b0ohqz62 2446189 2446187 2026-08-12T15:56:48Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli 2446189 wikitext text/x-wiki cq9sfnsyn12r6t8im64kx56kjb0rdbq 2446542 2446189 2026-08-13T09:06:12Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli 2446542 wikitext text/x-wiki s953324r98m5chqn9x7lp81c5kykge1 2446559 2446542 2026-08-13T09:21:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli 2446559 wikitext text/x-wiki f7o4vikqwkj1jc0waxh50fveasily4a 2446564 2446559 2026-08-13T09:26:49Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers 2446564 wikitext text/x-wiki lfb3gplkllx13ov84btia0qp7aqffxh 2446604 2446564 2026-08-13T09:58:51Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2446604 wikitext text/x-wiki jd1nuaclqo78mqna7z542lytomnqx5g 2446610 2446604 2026-08-13T10:00:08Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli 2446610 wikitext text/x-wiki imjhtt5hgizv7cz8ppus739vhjdkrky 2446624 2446610 2026-08-13T10:35:22Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2446624 wikitext text/x-wiki m95x54z02tyo2lxfxho7ifoj5ay0idf 2446626 2446624 2026-08-13T10:37:04Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli 2446626 wikitext text/x-wiki 2aehd1p5qamf1ibm4e2ub2tlbag5poa 2446629 2446626 2026-08-13T10:39:40Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2446629 wikitext text/x-wiki qw6fsgw957ekk2k055sy0tnlllb1z0h 2446631 2446629 2026-08-13T10:40:56Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2446631 wikitext text/x-wiki ki75m00uhmuml656ex9h2c37o4y2qtn 2446649 2446631 2026-08-13T11:25:08Z Stashbot 7414 taavi: update striker to 2026-08-13-111038-production 2446649 wikitext text/x-wiki iz6n8airp2pm7swhrx8vanmxoodk1cu 0 4108 2445903 2445712 2026-08-12T12:21:16Z ScheduleDeploymentBot 37566 Add [[gerrit:1324700]] to Wednesday, August 12 UTC afternoon backport window 2445903 wikitext text/x-wiki qwgy6c6wrtwgba3ncrserxy395r27bo 2445941 2445903 2026-08-12T13:11:55Z ScheduleDeploymentBot 37566 Add [[gerrit:1324707]] to Wednesday, August 12 UTC afternoon backport window 2445941 wikitext text/x-wiki 4orjp11rq2b52o2o87aadckgg6o2ih4 2445945 2445941 2026-08-12T13:13:09Z ScheduleDeploymentBot 37566 Add [[gerrit:1324713]] to Wednesday, August 12 UTC afternoon backport window 2445945 wikitext text/x-wiki len8ip99m1vcxxtw0vxkdtlsg29mupc 2445990 2445945 2026-08-12T13:27:38Z ScheduleDeploymentBot 37566 Add [[gerrit:1324719]] to Wednesday, August 12 UTC afternoon backport window 2445990 wikitext text/x-wiki 215j7eo3c8cs7i3brh1wb65kk3kaqtp 2445993 2445990 2026-08-12T13:31:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1324720]] to Wednesday, August 12 UTC afternoon backport window 2445993 wikitext text/x-wiki rjxg0fna37bwhr5wxesa72d8ai1n0bj 2445994 2445993 2026-08-12T13:32:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1324722]] to Wednesday, August 12 UTC afternoon backport window 2445994 wikitext text/x-wiki 5fd5496xtx3na5ad5u8sbgtofxzqg5e 2446334 2445994 2026-08-12T19:59:26Z ScheduleDeploymentBot 37566 Add [[gerrit:1324794]] to Wednesday, August 12 UTC late backport window 2446334 wikitext text/x-wiki 6ktolmvrg8mwn3j6j0l13mgthsmvmge 2446346 2446334 2026-08-12T20:14:16Z ScheduleDeploymentBot 37566 Add [[gerrit:1324277]] to Wednesday, August 12 UTC late backport window 2446346 wikitext text/x-wiki ey5rlmkbp1bgcm1kpmhh8dir1ctvqlj 2446404 2446346 2026-08-12T23:32:10Z ScheduleDeploymentBot 37566 Add [[gerrit:1216721]] to Monday, August 17 UTC morning backport window 2446404 wikitext text/x-wiki 97oy8d0ehu8od3uhy466643rbbj50bm 2446405 2446404 2026-08-12T23:33:51Z ScheduleDeploymentBot 37566 Add [[gerrit:1216721]] to Monday, August 17 UTC morning backport window 2446405 wikitext text/x-wiki 9galtm6hrvy3j8ht9jwd0i9h0dd0npx 2446406 2446405 2026-08-12T23:36:29Z Nvdtn19 47447 Restore revision [[Special:Permalink/2446404|2446404]] by [[Special:Contributions/ScheduleDeploymentBot|ScheduleDeploymentBot]] ([[User talk:ScheduleDeploymentBot|talk]]) 2446406 wikitext text/x-wiki 97oy8d0ehu8od3uhy466643rbbj50bm 2446539 2446406 2026-08-13T09:05:44Z ScheduleDeploymentBot 37566 Add [[gerrit:1324731]] to Thursday, August 13 UTC afternoon backport window 2446539 wikitext text/x-wiki 7r8k31dl9jfe7jwy6i3t10l3p0sdi8w 498 6726 2445898 2445889 2026-08-12T12:15:03Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli 2445898 wikitext text/x-wiki bdypbnezytomif3nw2q6vg209211qfv 2445899 2445898 2026-08-12T12:15:08Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli 2445899 wikitext text/x-wiki ls60kwkf7r909uccavw5azl571xppsu 2445901 2445899 2026-08-12T12:20:24Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli 2445901 wikitext text/x-wiki 8s49y1geqd16y76fkdohe6uz4ib01j4 2445902 2445901 2026-08-12T12:20:30Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli 2445902 wikitext text/x-wiki mw97k4bom9ywozn1zkgttr2muov8ska 2445906 2445902 2026-08-12T12:23:05Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli 2445906 wikitext text/x-wiki khpe0iriyloq5omxiubcb11dqx9wpgo 2445907 2445906 2026-08-12T12:23:11Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli 2445907 wikitext text/x-wiki 6it4qdnf6bah295gq9u0dvkak357y2g 2445908 2445907 2026-08-12T12:23:37Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli 2445908 wikitext text/x-wiki q4r9ouluiby3i3vw8ymrphsa0jngp9q 2445910 2445908 2026-08-12T12:24:20Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli 2445910 wikitext text/x-wiki 1iezza1mzqq1l7t0ipzt8ytzi62b2tl 2445915 2445910 2026-08-12T12:32:32Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli 2445915 wikitext text/x-wiki a7n8hzlbedq30snwuclickf6j8wy3oy 2445916 2445915 2026-08-12T12:32:49Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli 2445916 wikitext text/x-wiki eys3xpuq34hcxa7qt1hgafwyu6kk4e8 2445922 2445916 2026-08-12T12:44:54Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-bastion-6 (T401804) 2445922 wikitext text/x-wiki 29pjvy4m2cw202xpnvyyeh1fju8u4di 2445923 2445922 2026-08-12T12:45:47Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-bastion-6 (T401804) 2445923 wikitext text/x-wiki mtjmfb23y1em68satxyc4bq9t6wm5cx 2445924 2445923 2026-08-12T12:46:04Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli 2445924 wikitext text/x-wiki hdiy18viwrey96oy374x7p7cyh3mvft 2445925 2445924 2026-08-12T12:46:18Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli 2445925 wikitext text/x-wiki ogjd3kfly7gwz6wm0pfpx79cbhq3y46 2445927 2445925 2026-08-12T12:48:57Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli 2445927 wikitext text/x-wiki gq55sfwguv209p77j4g0e74alhslihc 2445928 2445927 2026-08-12T12:49:09Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli 2445928 wikitext text/x-wiki jzverlpjn4fgfndlgpb16243jiii3r1 2445929 2445928 2026-08-12T12:54:54Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli 2445929 wikitext text/x-wiki 8bj5iix5vv4gx4njlccjp64w2ealzsb 2445938 2445929 2026-08-12T13:07:50Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli 2445938 wikitext text/x-wiki 7ho2vb1ijrreccyjgh8ssisntmeao0q 2446106 2445938 2026-08-12T14:56:18Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2446106 wikitext text/x-wiki nk8n5r8vir24een0ac5v147169nl5qz 2446108 2446106 2026-08-12T14:56:22Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli 2446108 wikitext text/x-wiki 7u9qmksyd8mkljz0ps7a2dja8twuug5 2446116 2446108 2026-08-12T14:57:56Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2446116 wikitext text/x-wiki 7jrht63u0wjkoyp18imim64zlt7fn3q 2446117 2446116 2026-08-12T14:58:02Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli 2446117 wikitext text/x-wiki n64el8lnj6182d32spwdcncp7e9xq6n 2446119 2446117 2026-08-12T15:01:07Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2446119 wikitext text/x-wiki kgcvwrmusvng1t2xpyxdd0ca5di8xrt 2446130 2446119 2026-08-12T15:09:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2446130 wikitext text/x-wiki 54eizo1f240urn1uxdyaw4sr30tv2qu 2446160 2446130 2026-08-12T15:24:30Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.33.13 (T433131) 2446160 wikitext text/x-wiki in2jgob1j7i3utwr7ijv0k9i74t1itn 2446177 2446160 2026-08-12T15:39:48Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli 2446177 wikitext text/x-wiki r5ha3w9maurmb34wwjni3l8y2ouy303 2446178 2446177 2026-08-12T15:40:07Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli 2446178 wikitext text/x-wiki afgwwvrhidgwd58qoe0s0jgr5cc271m 2446179 2446178 2026-08-12T15:40:49Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.33.13 (T433131) 2446179 wikitext text/x-wiki o6tsrra5tzw1omebyl5wbaubplk8uwu 2446182 2446179 2026-08-12T15:51:07Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli 2446182 wikitext text/x-wiki le0yor6dpur000460dvgltiwd4krbd5 2446185 2446182 2026-08-12T15:52:35Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli 2446185 wikitext text/x-wiki fktruaq0u57ftw2mwrz3auhtii3y3n5 2446192 2446185 2026-08-12T16:02:05Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446192 wikitext text/x-wiki li44losjy973vd8r4m9dzsuw2y5xrtc 2446193 2446192 2026-08-12T16:02:16Z Stashbot 7414 andrewbogott: upgrading to kubernetes 1.33.13 2446193 wikitext text/x-wiki 3bqs818jaid2qd7tdxndaa32h7hfilk 2446198 2446193 2026-08-12T16:08:40Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli 2446198 wikitext text/x-wiki ryo6ojmq4ixdevby14rl070f7hn4ng4 2446200 2446198 2026-08-12T16:09:25Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli 2446200 wikitext text/x-wiki l2jlm495n58ffs469pu5338e9zpq07t 2446212 2446200 2026-08-12T16:21:55Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli 2446212 wikitext text/x-wiki 7ay8aisdjzbv5chvcba3clusymb7qcz 2446213 2446212 2026-08-12T16:22:26Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli 2446213 wikitext text/x-wiki 36ahbhje2oghwzhq45gb1nnuebzt486 2446217 2446213 2026-08-12T16:30:01Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446217 wikitext text/x-wiki 6gre72prha3zbauili94vsxbdnjswav 2446225 2446217 2026-08-12T16:39:27Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446225 wikitext text/x-wiki a2fq2z2f4nx9tmtr0qrogl2apj5dwyu 2446226 2446225 2026-08-12T16:39:29Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446226 wikitext text/x-wiki 84t9c9r9c15jojdjatu6i9h8irp7ctp 2446228 2446226 2026-08-12T16:39:42Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) 2446228 wikitext text/x-wiki fmvowh9ipcf6ukafih9btddlix8pdtr 2446234 2446228 2026-08-12T16:46:21Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) 2446234 wikitext text/x-wiki 5u3nvtjcggaeywerz40vew3qyq036h4 2446235 2446234 2026-08-12T16:48:54Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) 2446235 wikitext text/x-wiki 0gl4e00iin5972rt9g7kyvhhlrs6hl9 2446238 2446235 2026-08-12T16:55:10Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) 2446238 wikitext text/x-wiki fn869l9b0vsqrj35p2009hv3jlhbl9e 2446241 2446238 2026-08-12T16:56:11Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446241 wikitext text/x-wiki q1wd2vyd61rqqm543kaw22y80rrpbfu 2446242 2446241 2026-08-12T16:56:12Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446242 wikitext text/x-wiki boh513fyv7z4haermw8mz5x0tjzgw33 2446244 2446242 2026-08-12T16:59:01Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) 2446244 wikitext text/x-wiki fy3shaa387v2zd66uop3xawk0k7o23e 2446245 2446244 2026-08-12T16:59:02Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) 2446245 wikitext text/x-wiki hwsyjv2isozuwxduhjylp1wqs238wxb 2446254 2446245 2026-08-12T17:17:32Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) 2446254 wikitext text/x-wiki m1liyjqihe9l19dmap0hrqabll9ryqv 2446255 2446254 2026-08-12T17:17:33Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) 2446255 wikitext text/x-wiki hof2qjx3lh9pysmqiavcggyamrxpasp 2446263 2446255 2026-08-12T17:36:44Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446263 wikitext text/x-wiki 2wq022vb13lattj3za41njok7r3uj19 2446264 2446263 2026-08-12T17:37:46Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446264 wikitext text/x-wiki 84xovdsa1528tfu6vwzmu9qlfldbstw 2446265 2446264 2026-08-12T17:39:02Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446265 wikitext text/x-wiki 4eatbeuvkni5oahbzaevwn2t61qhd3x 2446266 2446265 2026-08-12T17:40:05Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446266 wikitext text/x-wiki etwjfqawi2vn49r2omolx6pmahng04a 2446280 2446266 2026-08-12T18:15:34Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2446280 wikitext text/x-wiki hfrmtxdvjz36hhhl6g7zxumcu2q2epb 2446281 2446280 2026-08-12T18:15:42Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2446281 wikitext text/x-wiki qhxxign4w8bvdl5ymdjexdmwwu4j3yi 2446292 2446281 2026-08-12T18:53:50Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster 2446292 wikitext text/x-wiki 3tqi7ozrox6bk5v8ryf0ukc26qbr0k3 2446296 2446292 2026-08-12T19:05:37Z Stashbot 7414 andrew@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster 2446296 wikitext text/x-wiki 0c7sguopm25dzbi77opseymzvinnkim 2446297 2446296 2026-08-12T19:05:41Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster 2446297 wikitext text/x-wiki is2p086xn8rvm96lq0a8mqkkpscjnst 2446298 2446297 2026-08-12T19:07:33Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446298 wikitext text/x-wiki 8ty0agejvh7qaa3tsuwuexu1k6qrpjb 2446299 2446298 2026-08-12T19:08:36Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446299 wikitext text/x-wiki 80moujlk278whkhe5tsarw98md7wdkd 2446303 2446299 2026-08-12T19:10:52Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446303 wikitext text/x-wiki i1ge8mv2muol7vbmwiqijort0awqzh0 2446304 2446303 2026-08-12T19:17:46Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446304 wikitext text/x-wiki 8t19zwolxl0qxu2lobjxl9r58nbdc46 2446313 2446304 2026-08-12T19:34:46Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446313 wikitext text/x-wiki 3wwtsh97jpi9evs2hk8whzi32bwbqlf 2446316 2446313 2026-08-12T19:41:11Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446316 wikitext text/x-wiki bl3jz2zsvtrl0h2z5gcc0rc42qe1qxm 2446317 2446316 2026-08-12T19:42:21Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446317 wikitext text/x-wiki dp7g7y7kn10oklkb2izx29k4a6sb1jr 2446319 2446317 2026-08-12T19:50:15Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446319 wikitext text/x-wiki tv1gv6j5orssop7fhi41ohse3c5v561 2446320 2446319 2026-08-12T19:50:23Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446320 wikitext text/x-wiki rc689r1u81wpdd5tyyhyisv1867kte1 2446321 2446320 2026-08-12T19:50:25Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446321 wikitext text/x-wiki io67ry3fuxmk2mewslzlf3n1obiaqks 2446322 2446321 2026-08-12T19:50:42Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446322 wikitext text/x-wiki 8da4ei9gtaehqxff0noohxvea4t6qqn 2446323 2446322 2026-08-12T19:50:43Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446323 wikitext text/x-wiki kyknf621a7a2okcdiy4s8ojv3lgm9tk 2446324 2446323 2026-08-12T19:54:17Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446324 wikitext text/x-wiki qtuxhl6tmoo6sc4bh0ewj4efwbni7b3 2446325 2446324 2026-08-12T19:54:19Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) 2446325 wikitext text/x-wiki lnnbwtjy9e0xt93lifxc7w7s9gw76nf 2446326 2446325 2026-08-12T19:54:32Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) 2446326 wikitext text/x-wiki gfob3cpsimgwfmjzn6h2us1bqw0rht1 2446327 2446326 2026-08-12T19:54:33Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) 2446327 wikitext text/x-wiki ixpkh9g57w3mimfo9c59ewij4ujvdf3 2446329 2446327 2026-08-12T19:56:27Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446329 wikitext text/x-wiki ji8cehd37w0giojdyvf7lkrif04wryx 2446330 2446329 2026-08-12T19:56:29Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446330 wikitext text/x-wiki gf1dvlra40ixtw9ix3t5og5w5pevd52 2446347 2446330 2026-08-12T20:22:30Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446347 wikitext text/x-wiki 6hdwyz8ecto37cd50dth8r2a7j4y5n4 2446348 2446347 2026-08-12T20:22:38Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446348 wikitext text/x-wiki sm41isf2vrpa1awd86byhnf5zz6ehvr 2446349 2446348 2026-08-12T20:22:44Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446349 wikitext text/x-wiki 6b5gqn6373fv2tfjvrajyvnsde04u2i 2446350 2446349 2026-08-12T20:22:47Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446350 wikitext text/x-wiki ssoswxxi2wlmk3iiju0fi4ci0idmgtl 2446351 2446350 2026-08-12T20:22:53Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446351 wikitext text/x-wiki b539vbffgee6p2dyg22ayuld72r6rl8 2446352 2446351 2026-08-12T20:23:59Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446352 wikitext text/x-wiki lr31kh87t2luit4ayug7za559rfzik5 2446363 2446352 2026-08-12T20:46:48Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446363 wikitext text/x-wiki bchwoidwgqb2164z49vgnozx4ahqdl0 2446364 2446363 2026-08-12T20:47:53Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446364 wikitext text/x-wiki 8wsh99vsnhbu8go2v3z3wrmhkthbcfr 2446365 2446364 2026-08-12T20:48:57Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446365 wikitext text/x-wiki sl3dzdanh1nbnh65dxrr5mpxbe32k4b 2446367 2446365 2026-08-12T20:52:25Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446367 wikitext text/x-wiki q9228yvos7kr0paryjkic0gxc6th3li 2446368 2446367 2026-08-12T20:52:50Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446368 wikitext text/x-wiki qbuobz4ezisl6e1mo9njuyggpt5g0nh 2446370 2446368 2026-08-12T20:54:58Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) 2446370 wikitext text/x-wiki lzlfnz07jlbfgh69u0as2ya7nn12895 2446372 2446370 2026-08-12T21:01:01Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T433131) 2446372 wikitext text/x-wiki 9anrzockdbc29qq22uvas08gk1r5mas 2446373 2446372 2026-08-12T21:01:05Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T433131) 2446373 wikitext text/x-wiki n1cfcn58m8v5ysb3nqp9lc9koh1a4rg 2446415 2446373 2026-08-13T01:17:51Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) 2446415 wikitext text/x-wiki 8j3vuinxfh0pd7nrcdykbu01b6wnsf2 2446417 2446415 2026-08-13T01:47:52Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) 2446417 wikitext text/x-wiki pibry6mrwt8xiun8jezvrfvclx1xq89 2446418 2446417 2026-08-13T01:48:02Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446418 wikitext text/x-wiki 1mit6b444eurnna4ey66li4u3e4wc4l 2446419 2446418 2026-08-13T01:48:04Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) 2446419 wikitext text/x-wiki huv220fvkc106x4jowvscoo8ktslttw 2446420 2446419 2026-08-13T01:49:23Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-13 2446420 wikitext text/x-wiki kdvrr07ybsrlsdmcu1s6h1n8jsa81mj 2446421 2446420 2026-08-13T01:49:25Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-13 2446421 wikitext text/x-wiki ndyr8r2nx8bj6rpejt8p55mlswohve3 2446422 2446421 2026-08-13T01:50:22Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-13 2446422 wikitext text/x-wiki qghl4ry6of23c9lyge3921vl1wnta3l 2446423 2446422 2026-08-13T01:51:53Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-13 2446423 wikitext text/x-wiki gx3az7bp8nbisof70vwgk3wgls60ji6 2446424 2446423 2026-08-13T01:56:37Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131) 2446424 wikitext text/x-wiki h9hqi5nbs2razaxegu34vq8jopcesoe 2446425 2446424 2026-08-13T01:57:41Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131) 2446425 wikitext text/x-wiki hx4ortblpr9t5mvjlpj3r619wpdsl47 2446426 2446425 2026-08-13T01:58:39Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131) 2446426 wikitext text/x-wiki gt05a2y1c49yotzyomub2c89f8iin29 2446427 2446426 2026-08-13T01:59:40Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131) 2446427 wikitext text/x-wiki j49kfgr7l343emuflxz8igdvb5udo72 2446430 2446427 2026-08-13T02:01:04Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) 2446430 wikitext text/x-wiki 9a5gjy65knuo3ducg2aq2ybn9uvdw3m 2446431 2446430 2026-08-13T02:01:57Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) 2446431 wikitext text/x-wiki 1joqk2xqg5howrr0ixm5mu9sy81q2io 2446432 2446431 2026-08-13T02:02:06Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) 2446432 wikitext text/x-wiki 8p9q64vz55vf4hr4pdz5eq9jbw2pnbn 2446433 2446432 2026-08-13T02:03:00Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) 2446433 wikitext text/x-wiki etnbe9nd5qvvpr7bw1i16x5r3dt5lyp 2446434 2446433 2026-08-13T02:03:10Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) 2446434 wikitext text/x-wiki gyk8qjykdcdduw6mmurlsbcxe7ms4zw 2446436 2446434 2026-08-13T02:04:04Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) 2446436 wikitext text/x-wiki 62z6kr7p2vol805bkh3f4faccbawd4c 2446461 2446436 2026-08-13T08:09:31Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli 2446461 wikitext text/x-wiki sj74vjjxxzulrzlaecfq6fgrg02vizn 2446468 2446461 2026-08-13T08:21:55Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli 2446468 wikitext text/x-wiki bj8pb7u86weao696xx8cogy1a47w8vc 2446594 2446468 2026-08-13T09:41:54Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2446594 wikitext text/x-wiki bskuhw9stn04b5rvi4b3f9jlozakpk3 2446597 2446594 2026-08-13T09:43:09Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2446597 wikitext text/x-wiki lyi13603we3ifiywhwuqdeamxz7u3md 0 7478 2445904 2434891 2026-08-12T12:21:27Z Filippo Giunchedi 2939 Update page in light of https://phabricator.wikimedia.org/T362397 2445904 wikitext text/x-wiki cepc8x39couz4plv7q6198ys4cksvmr 0 7919 2445891 2445887 2026-08-12T12:00:01Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet 2445891 wikitext text/x-wiki sd7cjd62lsmbxopxlz2hdnqbi1a3dd9 2445892 2445891 2026-08-12T12:00:23Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet 2445892 wikitext text/x-wiki t90vgxqcuib6cqrd7d7dimdujyzx4h6 2445893 2445892 2026-08-12T12:01:19Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1017.eqiad.wmnet with OS bookworm 2445893 wikitext text/x-wiki duzuztnoppmt5un0yk343f3igtp3ubc 2445894 2445893 2026-08-12T12:02:26Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1009.eqiad.wmnet with OS bookworm 2445894 wikitext text/x-wiki 17ifwrrvt6n212qjfv3a2tf6hth7scr 2445895 2445894 2026-08-12T12:04:27Z Stashbot 7414 moritzm: failover ganeti master in drmrs02 to ganeti6004 2445895 wikitext text/x-wiki 6vd6xt242g3t826nzqar86jrdbmulq9 2445896 2445895 2026-08-12T12:07:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1015.eqiad.wmnet with reason: host reimage 2445896 wikitext text/x-wiki 5ipkfvktvy0rvagkn4mwxy4istg388t 2445897 2445896 2026-08-12T12:11:32Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1015.eqiad.wmnet with reason: host reimage 2445897 wikitext text/x-wiki 2pfo6jjyn4p3dtodubjs039y5yjusff 2445900 2445897 2026-08-12T12:18:07Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1017.eqiad.wmnet with reason: host reimage 2445900 wikitext text/x-wiki 6j5vdgcgh2py30kopjkt2to1q6rf1an 2445905 2445900 2026-08-12T12:22:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1158: Pool db1158.eqiad.wmnet in after cloning 2445905 wikitext text/x-wiki 8n0kcb9xa6b6mzgcqympt0bw0nip5sw 2445909 2445905 2026-08-12T12:24:18Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1017.eqiad.wmnet with reason: host reimage 2445909 wikitext text/x-wiki e34uj2jamsklxnn3ostfbna05ef1cvc 2445914 2445909 2026-08-12T12:31:37Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet 2445914 wikitext text/x-wiki hbbkgunzzivuc2k4g7ujepyphbvu59k 2445917 2445914 2026-08-12T12:33:59Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2445917 wikitext text/x-wiki 4r3t83v40gqopfxac419xvy83z0rhor 2445918 2445917 2026-08-12T12:34:37Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2445918 wikitext text/x-wiki fjflkj4e41bhruc8wlkduta3e0m9o9h 2445919 2445918 2026-08-12T12:35:10Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2445919 wikitext text/x-wiki kckgih05ubp276y0n9rela5918078hh 2445920 2445919 2026-08-12T12:36:22Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2445920 wikitext text/x-wiki ovsx3mkk8rpp6ntbm190q0wtxtb65ng 2445921 2445920 2026-08-12T12:44:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1017.eqiad.wmnet with OS bookworm 2445921 wikitext text/x-wiki c6wq2j542xnu35p5fu45vob1spfe3dm 2445930 2445921 2026-08-12T12:56:06Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet 2445930 wikitext text/x-wiki bdsb07vqey35v5smifnukgtz6utoty9 2445931 2445930 2026-08-12T12:59:18Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1015.eqiad.wmnet with OS bookworm 2445931 wikitext text/x-wiki lbq1aytsdii9v79conys4scq2cvsnza 2445934 2445931 2026-08-12T13:02:23Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet 2445934 wikitext text/x-wiki 69gnmhizoxor21szce9wvjrikn0mvys 2445935 2445934 2026-08-12T13:02:27Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet 2445935 wikitext text/x-wiki cxzfvdpzdenszplcnak7w44g97xjwsk 2445939 2445935 2026-08-12T13:08:00Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1158: Pool db1158.eqiad.wmnet in after cloning 2445939 wikitext text/x-wiki juhnyte2eyvypultm9i97i9mmq6dkta 2445940 2445939 2026-08-12T13:08:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1158.eqiad.wmnet onto db1273.eqiad.wmnet 2445940 wikitext text/x-wiki 6dwwx6mf9b8x9gv2xfdxls7ybftajim 2445942 2445940 2026-08-12T13:12:13Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet 2445942 wikitext text/x-wiki c6mmvunljaqk9l4xd8kv0903ezfd6sr 2445943 2445942 2026-08-12T13:12:15Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock1001.eqiad.wmnet 2445943 wikitext text/x-wiki 66qnvkt1wacviv9u5scal6ctgsfwdf3 2445944 2445943 2026-08-12T13:12:17Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2445944 wikitext text/x-wiki 9gkvn31vamfsdfouuwjoeb8agwunzqi 2445946 2445944 2026-08-12T13:14:16Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet 2445946 wikitext text/x-wiki gfdzcvzzl1kkg47xo1eeq8di5kn6j1i 2445947 2445946 2026-08-12T13:14:28Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply 2445947 wikitext text/x-wiki pajqu5bwma4xf81pxyfv70s5ofburl5 2445948 2445947 2026-08-12T13:15:05Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply 2445948 wikitext text/x-wiki tnml3ya2idrxl3em57ka626yxpzzjxe 2445949 2445948 2026-08-12T13:15:17Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply 2445949 wikitext text/x-wiki dv5yafdmnngl688of9n63rcazwg5qls 2445952 2445949 2026-08-12T13:15:41Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply 2445952 wikitext text/x-wiki qaf6vq6g5713yf6aznyaa9a312kusjp 2445953 2445952 2026-08-12T13:15:54Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1324713|thwiki: reinstate temporary wiki25 logos (T431094)]] 2445953 wikitext text/x-wiki 4zmjzj56ba6sndsc4i0hqitm9aor6dw 2445954 2445953 2026-08-12T13:17:10Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1001.eqiad.wmnet - cgoubert@cumin2003" 2445954 wikitext text/x-wiki 2a2030d1mtp0ur7mviu9511rj9rjle6 2445955 2445954 2026-08-12T13:17:15Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1001.eqiad.wmnet - cgoubert@cumin2003" 2445955 wikitext text/x-wiki 1bd3r102gti5wypta99n6elvxa8uam1 2445956 2445955 2026-08-12T13:17:18Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2445956 wikitext text/x-wiki 8c2844sm4t8l55hbsmsxw2tyygo9xu5 2445957 2445956 2026-08-12T13:17:20Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock1001.eqiad.wmnet on all recursors 2445957 wikitext text/x-wiki cvmkrtx7trfnl9og5up491lw9d6t2mt 2445958 2445957 2026-08-12T13:17:22Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock1001.eqiad.wmnet on all recursors 2445958 wikitext text/x-wiki miv7gycddnu05iv2c2v33zkj373ywem 2445959 2445958 2026-08-12T13:17:40Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply 2445959 wikitext text/x-wiki 85yyz6mt0xh39bw5iho3ce7uqtgnfpv 2445960 2445959 2026-08-12T13:17:45Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2001.codfw.wmnet 2445960 wikitext text/x-wiki at6c8uvrp61qxyk4xy4569ro5yzuefd 2445961 2445960 2026-08-12T13:17:47Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1001.eqiad.wmnet - cgoubert@cumin2003" 2445961 wikitext text/x-wiki ii0az3s370y54fz36h97vdj59dln6ve 2445962 2445961 2026-08-12T13:17:49Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2445962 wikitext text/x-wiki otbqrgtf85q19z3roh56wn0j44zkwjk 2445963 2445962 2026-08-12T13:17:51Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1001.eqiad.wmnet - cgoubert@cumin2003" 2445963 wikitext text/x-wiki slsmnph147q2ap95sl46s37ldbznte5 2445964 2445963 2026-08-12T13:18:00Z Stashbot 7414 kharlan@deploy1003: anzx, kharlan: Backport for [[gerrit:1324713|thwiki: reinstate temporary wiki25 logos (T431094)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2445964 wikitext text/x-wiki 7y4jd3sfllscol1m3tabfvbrtal0tl1 2445966 2445964 2026-08-12T13:18:39Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock1001.eqiad.wmnet with OS trixie 2445966 wikitext text/x-wiki 95kw6ystzxppxytk2yk4l1sxhjp8z1u 2445967 2445966 2026-08-12T13:18:41Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply 2445967 wikitext text/x-wiki 4zyqc4nqip4xkdye4hkezt6xqm5p70y 2445968 2445967 2026-08-12T13:18:59Z Stashbot 7414 kharlan@deploy1003: anzx, kharlan: Continuing with deployment 2445968 wikitext text/x-wiki 2o7fkmj07babdsxxfdqy0jyxwgd3w7o 2445969 2445968 2026-08-12T13:19:38Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply 2445969 wikitext text/x-wiki kgohad6mo1g5pwnbe6u5eketxt6dh0v 2445970 2445969 2026-08-12T13:20:16Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet 2445970 wikitext text/x-wiki 4urwcm402x902j6hzfzswy4g74y6uzv 2445971 2445970 2026-08-12T13:20:28Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet 2445971 wikitext text/x-wiki ggdk8ncrc8otjp8pyk0gsklmmrlfveo 2445972 2445971 2026-08-12T13:20:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1018.eqiad.wmnet with OS bookworm 2445972 wikitext text/x-wiki swavjngrpnd3nhnte07paoft9i10f99 2445973 2445972 2026-08-12T13:20:43Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply 2445973 wikitext text/x-wiki qalqhwck9ftnloemgbtfkl7gxp1plm3 2445975 2445973 2026-08-12T13:23:08Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324713|thwiki: reinstate temporary wiki25 logos (T431094)]] (duration: 07m 13s) 2445975 wikitext text/x-wiki i5ckflixquu8j6coa1k3m45u6bgv8yv 2445976 2445975 2026-08-12T13:23:21Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2001.codfw.wmnet - cgoubert@cumin2003" 2445976 wikitext text/x-wiki fzhebu21coy80crxy8ehm9axqs4a5d6 2445977 2445976 2026-08-12T13:23:50Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2001.codfw.wmnet - cgoubert@cumin2003" 2445977 wikitext text/x-wiki eefvzrzq9o3lyfbmvjyaj98wzd9rsl1 2445978 2445977 2026-08-12T13:23:51Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2445978 wikitext text/x-wiki o1a70esk27uuoyxo2vgguusxn5s11nq 2445979 2445978 2026-08-12T13:23:53Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2001.codfw.wmnet on all recursors 2445979 wikitext text/x-wiki 8qp8ysr0vzknp92rcguzd2fuoxodeq8 2445980 2445979 2026-08-12T13:23:55Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2001.codfw.wmnet on all recursors 2445980 wikitext text/x-wiki q8rum624iz386qddktupcahqhwr7qtt 2445981 2445980 2026-08-12T13:24:24Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2001.codfw.wmnet - cgoubert@cumin2003" 2445981 wikitext text/x-wiki g7ni9gbjf25cs7zhkomiyyiqdqv2b8z 2445982 2445981 2026-08-12T13:24:29Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2001.codfw.wmnet - cgoubert@cumin2003" 2445982 wikitext text/x-wiki d185dbfgft94mi6nx6nio4v2lxr6jif 2445983 2445982 2026-08-12T13:24:37Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock2001.codfw.wmnet with OS trixie 2445983 wikitext text/x-wiki 7fxwba63nzn06mix7h58cx6rpc8d9no 2445984 2445983 2026-08-12T13:25:12Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitize-wiki (exit_code=99) Managing sanitization for wikis testwiki in section s3 2445984 wikitext text/x-wiki hq039zmh7rz2if5coouchgr103lenol 2445985 2445984 2026-08-12T13:25:22Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1275938|Enable campaignEvents on bdwikimedia (T424016)]] 2445985 wikitext text/x-wiki sjbn8zhtgpd219nacfdhpfoa78ttqtr 2445986 2445985 2026-08-12T13:26:43Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2445986 wikitext text/x-wiki blb84noy51n5zi18tenlchv7wh21g4u 2445987 2445986 2026-08-12T13:26:54Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2445987 wikitext text/x-wiki hrofwsv5v8b1jyd3j0mkn116cxsf1si 2445988 2445987 2026-08-12T13:27:03Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2028.codfw.wmnet 2445988 wikitext text/x-wiki 1lrnj4bidds2ab4on8ryq91terl4q2b 2445989 2445988 2026-08-12T13:27:27Z Stashbot 7414 kharlan@deploy1003: kharlan, yahya: Backport for [[gerrit:1275938|Enable campaignEvents on bdwikimedia (T424016)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2445989 wikitext text/x-wiki 2no5yro0g43jp6tgiq7qkmlerd15qrf 2445991 2445989 2026-08-12T13:28:52Z Stashbot 7414 kharlan@deploy1003: kharlan, yahya: Continuing with deployment 2445991 wikitext text/x-wiki 6sh5rybbe26k4tes3fewp9m2cwe7gxe 2445995 2445991 2026-08-12T13:32:37Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb-lock1001.eqiad.wmnet with reason: host reimage 2445995 wikitext text/x-wiki 3uuctgiwsvrtc4om1j218vmsyjawuox 2445996 2445995 2026-08-12T13:32:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1019.eqiad.wmnet with OS bookworm 2445996 wikitext text/x-wiki opsy2uo6lazbxuj6p419jax9uz33t0l 2445997 2445996 2026-08-12T13:32:48Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1020.eqiad.wmnet with OS bookworm 2445997 wikitext text/x-wiki 7um1lum0u8156syrdwq6zvbj7ntzdgv 2445998 2445997 2026-08-12T13:32:56Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1275938|Enable campaignEvents on bdwikimedia (T424016)]] (duration: 07m 35s) 2445998 wikitext text/x-wiki baw29pbw5118z6540q7zkthsgpp9j82 2445999 2445998 2026-08-12T13:35:33Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet 2445999 wikitext text/x-wiki h8883q0ofaoen00o7d3zsexob6zmn6f 2446000 2445999 2026-08-12T13:35:47Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1324707|Backport all changes from wmf/1.47.0-wmf.15]] 2446000 wikitext text/x-wiki k980btz3ovviz3r0coyr2f3nxygwglh 2446001 2446000 2026-08-12T13:36:53Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1018.eqiad.wmnet with reason: host reimage 2446001 wikitext text/x-wiki edlhgtnuhlhx9uwqr380tubik6q4gv0 2446002 2446001 2026-08-12T13:38:51Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock1001.eqiad.wmnet with reason: host reimage 2446002 wikitext text/x-wiki 04w3xxw1p9y0mze4vn95xr4db7lgkj2 2446006 2446002 2026-08-12T13:41:53Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1018.eqiad.wmnet with reason: host reimage 2446006 wikitext text/x-wiki fceg33t2p1hhb5tesfu00fl6m9l28fq 2446007 2446006 2026-08-12T13:41:56Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet 2446007 wikitext text/x-wiki d5bmiaw3dyo4ac2oftksdabhsd8kn8j 2446008 2446007 2026-08-12T13:42:00Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet 2446008 wikitext text/x-wiki cestzfi0zo81vjjbtmpx2f19oab3ze8 2446009 2446008 2026-08-12T13:43:09Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb-lock2001.codfw.wmnet with reason: host reimage 2446009 wikitext text/x-wiki aba9elabvqlxchpedy031xc2rgik80i 2446010 2446009 2026-08-12T13:44:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host apifeatureusage2001.codfw.wmnet with OS bookworm 2446010 wikitext text/x-wiki 53zm0ov65gjezi514eln0f5gloasp9g 2446012 2446010 2026-08-12T13:47:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Configuring db1272 for s3 pooling', diff saved to https://phabricator.wikimedia.org/P96021 and previous config saved to /var/cache/conftool/dbconfig/20260812-134732-cwilliams.json 2446012 wikitext text/x-wiki kslvtb7yqdp072ehoef07eo3nmua6dl 2446013 2446012 2026-08-12T13:48:24Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet 2446013 wikitext text/x-wiki iipispgrlja3f8jrtolqcd70tgyt173 2446014 2446013 2026-08-12T13:49:32Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock2001.codfw.wmnet with reason: host reimage 2446014 wikitext text/x-wiki 942mswt6100lnbjxk9pywxcbjs92gsa 2446015 2446014 2026-08-12T13:49:34Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1020.eqiad.wmnet with reason: host reimage 2446015 wikitext text/x-wiki op7m1sdwtfmtbob7nhvxj9lilgkistt 2446016 2446015 2026-08-12T13:49:56Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1019.eqiad.wmnet with reason: host reimage 2446016 wikitext text/x-wiki kavdaa31x1msl4h2w9o01dxicll31ps 2446017 2446016 2026-08-12T13:52:38Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1020.eqiad.wmnet with reason: host reimage 2446017 wikitext text/x-wiki 391aw0dc31mxcrd67dn9pibx99urdxn 2446018 2446017 2026-08-12T13:52:50Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet 2446018 wikitext text/x-wiki 07cpzmi2ea97ew80zf167hz5h1tatlk 2446020 2446018 2026-08-12T13:53:19Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1324707|Backport all changes from wmf/1.47.0-wmf.15]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446020 wikitext text/x-wiki 5xm7bdft1xhgjpvy85viip9rncjxtu5 2446021 2446020 2026-08-12T13:53:34Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb-lock1001.eqiad.wmnet with OS trixie 2446021 wikitext text/x-wiki c1xo248ce3b1dkun1vt090rfnc9vpt2 2446022 2446021 2026-08-12T13:53:36Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rdb-lock1001.eqiad.wmnet 2446022 wikitext text/x-wiki 1y4o5ewg2p64ge5kdrc7ik2xjdn7n15 2446023 2446022 2026-08-12T13:54:04Z Stashbot 7414 btullis@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on archiva1002.wikimedia.org with reason: Upgrading in-place 2446023 wikitext text/x-wiki f3qjvlrgpv5whbho1g668rswvq66uis 2446024 2446023 2026-08-12T13:56:30Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1019.eqiad.wmnet with reason: host reimage 2446024 wikitext text/x-wiki nu7pohpx5zi09vchzqoxkqng8d3xxm2 2446025 2446024 2026-08-12T13:58:13Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock1002.eqiad.wmnet 2446025 wikitext text/x-wiki 894tw18h1pdnwmkd5o4wxsrwlpu6598 2446026 2446025 2026-08-12T13:58:16Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446026 wikitext text/x-wiki cazbf2lyybqamj80cpcphsb1xfl24on 2446027 2446026 2026-08-12T13:58:56Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet 2446027 wikitext text/x-wiki ddmibq4fnauyv6yg4j9ryyzo123lu4p 2446028 2446027 2026-08-12T13:59:22Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet 2446028 wikitext text/x-wiki laco0gglnpqbmorms4c0ysrzgs7ef23 2446029 2446028 2026-08-12T14:01:00Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet 2446029 wikitext text/x-wiki m6wfiaoqifrcy99isg4yq1jpxw8kp7w 2446030 2446029 2026-08-12T14:02:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on apifeatureusage2001.codfw.wmnet with reason: host reimage 2446030 wikitext text/x-wiki 0m4jefifsbysm4ttbcbojhczd8ttdl5 2446031 2446030 2026-08-12T14:02:42Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1002.eqiad.wmnet - cgoubert@cumin2003" 2446031 wikitext text/x-wiki 91ux9qrpt9hha0jbk70ccaiql6yyiq2 2446032 2446031 2026-08-12T14:03:38Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb-lock2001.codfw.wmnet with OS trixie 2446032 wikitext text/x-wiki n5bi7szo3q0r4x9fzmbphpfdi2ng74c 2446033 2446032 2026-08-12T14:03:41Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rdb-lock2001.codfw.wmnet 2446033 wikitext text/x-wiki 8v8pkgevro7qabpqqwoimzsx5qsvoer 2446034 2446033 2026-08-12T14:04:08Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2446034 wikitext text/x-wiki 0vn883hr0kr20uymj8f3p2gjlxzopg2 2446035 2446034 2026-08-12T14:04:33Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2446035 wikitext text/x-wiki a460jhc4137wvl9yygblp6sbuy4617w 2446036 2446035 2026-08-12T14:04:59Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2446036 wikitext text/x-wiki k5013pzuejwaiqlotqp7o6abe6upvom 2446037 2446036 2026-08-12T14:05:09Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1002.eqiad.wmnet - cgoubert@cumin2003" 2446037 wikitext text/x-wiki eqr51fj4nk3g7wkvbnehzatdzu4qpfx 2446038 2446037 2026-08-12T14:05:11Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446038 wikitext text/x-wiki bd1dnknyv3z3dq73fahu2b01q8hgt0n 2446039 2446038 2026-08-12T14:05:14Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock1002.eqiad.wmnet on all recursors 2446039 wikitext text/x-wiki 1kjbqhedmukn4jwt2wnbh8pvcv6k81d 2446040 2446039 2026-08-12T14:05:16Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock1002.eqiad.wmnet on all recursors 2446040 wikitext text/x-wiki najo889z9sju1nyge67zm48g18riwig 2446041 2446040 2026-08-12T14:05:36Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1002.eqiad.wmnet - cgoubert@cumin2003" 2446041 wikitext text/x-wiki qwihk3ahec45dhbugdxddfgjr3x303v 2446042 2446041 2026-08-12T14:05:40Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1002.eqiad.wmnet - cgoubert@cumin2003" 2446042 wikitext text/x-wiki ck7giu61zx4arvmfc0tfgthhxyk97sf 2446043 2446042 2026-08-12T14:05:54Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2002.codfw.wmnet 2446043 wikitext text/x-wiki h8e4oywronw6c2a3r8tz59y4wc8u9ie 2446044 2446043 2026-08-12T14:05:58Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446044 wikitext text/x-wiki df8h3riotrvldp6depur0jxxibqmyd3 2446045 2446044 2026-08-12T14:06:02Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock1002.eqiad.wmnet with OS trixie 2446045 wikitext text/x-wiki nc7wuwbjs8d1xc3cdbm8ucsck0sxsri 2446046 2446045 2026-08-12T14:06:35Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2446046 wikitext text/x-wiki ik68rz2x347j5z0nd12i83a759alj02 2446047 2446046 2026-08-12T14:06:37Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet 2446047 wikitext text/x-wiki sbstwrzuj54a6h60i5ersid304j9bel 2446048 2446047 2026-08-12T14:07:58Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie 2446048 wikitext text/x-wiki 6cya9ynr1vxgksd1gug47ptq4hrhhd6 2446049 2446048 2026-08-12T14:08:35Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2446049 wikitext text/x-wiki sqdgmmgjcr8xpq050bpumncs6cvr862 2446050 2446049 2026-08-12T14:09:24Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2446050 wikitext text/x-wiki n23apx78mo9bt9cqg03sdmunodurres 2446051 2446050 2026-08-12T14:09:30Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on apifeatureusage2001.codfw.wmnet with reason: host reimage 2446051 wikitext text/x-wiki njujsfx9xci8j0mqaixfz80coygvced 2446052 2446051 2026-08-12T14:11:31Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2446052 wikitext text/x-wiki imjpmq4n1umj5lch9wbv7utg8j8z8ie 2446053 2446052 2026-08-12T14:12:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet 2446053 wikitext text/x-wiki sxgp3x1yu41twuab2d1rz4h5xu1vi3d 2446054 2446053 2026-08-12T14:12:47Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet 2446054 wikitext text/x-wiki q9u9m4pkx751x4uklbsze0szgsy3o4o 2446055 2446054 2026-08-12T14:14:03Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2002.codfw.wmnet - cgoubert@cumin2003" 2446055 wikitext text/x-wiki i7u4tq3sr8ypfqcva36pp5fgbciso7u 2446056 2446055 2026-08-12T14:14:07Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2002.codfw.wmnet - cgoubert@cumin2003" 2446056 wikitext text/x-wiki 6cvaoztdiu5g7vle3sebpqer89pp1sr 2446057 2446056 2026-08-12T14:14:10Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446057 wikitext text/x-wiki n029vof5vv5qnwgrxu2ei6rnhlg258d 2446058 2446057 2026-08-12T14:14:12Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2002.codfw.wmnet on all recursors 2446058 wikitext text/x-wiki 2tnr4fa93gtyaomusqe8fybrv6ok2zz 2446059 2446058 2026-08-12T14:14:14Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2002.codfw.wmnet on all recursors 2446059 wikitext text/x-wiki 7h62m7ghnuu338dtbliwzdexy96qigk 2446060 2446059 2026-08-12T14:14:17Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2446060 wikitext text/x-wiki eka5yvyjvd5fsfbalj70d7qs2xqst28 2446061 2446060 2026-08-12T14:14:38Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet 2446061 wikitext text/x-wiki 4i12z080bxn0kfmsue7gi27gbwe5fps 2446062 2446061 2026-08-12T14:14:40Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2446062 wikitext text/x-wiki c98s37ggcet2k6n9v78qh2nggpikxcm 2446063 2446062 2026-08-12T14:14:43Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2002.codfw.wmnet - cgoubert@cumin2003" 2446063 wikitext text/x-wiki tlg0qywtelx495t0u2btq4488fc6j16 2446064 2446063 2026-08-12T14:14:47Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2002.codfw.wmnet - cgoubert@cumin2003" 2446064 wikitext text/x-wiki 96xumt4s0crne3w9mn8yghnf2pwpa53 2446065 2446064 2026-08-12T14:14:53Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2446065 wikitext text/x-wiki ezkqirbuquankvcpungmluokh0wxbg2 2446066 2446065 2026-08-12T14:15:35Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock2002.codfw.wmnet with OS trixie 2446066 wikitext text/x-wiki kf44049hsest5tlzjm79no4rrcn7wvl 2446068 2446066 2026-08-12T14:16:34Z Stashbot 7414 moritzm: installing Linux 6.1.180 on Bookworm hosts 2446068 wikitext text/x-wiki b2wodsx9sra5ay5gkefz7527o9t5wtb 2446069 2446068 2026-08-12T14:16:39Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324707|Backport all changes from wmf/1.47.0-wmf.15]] (duration: 40m 51s) 2446069 wikitext text/x-wiki 1u6j6vnq94fy4pi9s626kj54rxyuhe1 2446070 2446069 2026-08-12T14:16:44Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2446070 wikitext text/x-wiki imxmrlwrozak7rtu7fkbq1znczh4rhg 2446071 2446070 2026-08-12T14:16:53Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2446071 wikitext text/x-wiki 6svz3t2bwtm6fhltjnmt1r4upja5ygv 2446072 2446071 2026-08-12T14:17:28Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2446072 wikitext text/x-wiki i31bz30huf8hnpwfntrlv1csm4i8rsg 2446073 2446072 2026-08-12T14:18:01Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb-lock1002.eqiad.wmnet with reason: host reimage 2446073 wikitext text/x-wiki 5ufjyr8nh5yrh0a9y0sg0bfny4uchho 2446074 2446073 2026-08-12T14:20:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2045.codfw.wmnet 2446074 wikitext text/x-wiki 5brbg4yixzuesi41ve7zdo11xk870i9 2446075 2446074 2026-08-12T14:22:34Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis testwiki in section s3 2446075 wikitext text/x-wiki dxoh6berxooq8sn9es69d1pxtec7ai9 2446076 2446075 2026-08-12T14:23:07Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1019.eqiad.wmnet with OS bookworm 2446076 wikitext text/x-wiki s449cb5xa0ne4rl48r8vkr5clzdaugi 2446077 2446076 2026-08-12T14:23:44Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock1002.eqiad.wmnet with reason: host reimage 2446077 wikitext text/x-wiki 54oz7yuyf81ao0woaqh4loizqkybi79 2446078 2446077 2026-08-12T14:25:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2045.codfw.wmnet 2446078 wikitext text/x-wiki pw85idk6ddv7q8c9khz8g9jy7pghyzy 2446079 2446078 2026-08-12T14:25:48Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet 2446079 wikitext text/x-wiki ax3742xeyiismvum6scor0ofovv4pt0 2446080 2446079 2026-08-12T14:27:11Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2046.codfw.wmnet 2446080 wikitext text/x-wiki n2zv7b0mqdmpszpi4w1t1tbcetmln6o 2446082 2446080 2026-08-12T14:31:12Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1020.eqiad.wmnet with OS bookworm 2446082 wikitext text/x-wiki 9bx7emxibzam52au2lbec15yihajsdj 2446083 2446082 2026-08-12T14:32:26Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2046.codfw.wmnet 2446083 wikitext text/x-wiki 1ub32exsopmjlqet4kzqw7aa1b5r8av 2446084 2446083 2026-08-12T14:32:29Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1018.eqiad.wmnet with OS bookworm 2446084 wikitext text/x-wiki 7c3hp39m644x64nlglzermh88bsv5kt 2446085 2446084 2026-08-12T14:33:10Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1324719|Partial revert "Use LockManager service instead of Database::getScopedLockAndFlush()" (T366938 T434596)]], [[gerrit:1324720|Partial revert "Use LockManager service instead of Database::getScopedLockAndFlush()" (T366938 T434596)]], [[gerrit:1324722|Bump mediawiki/mediawiki-codesniffer to v52.0.0 (T434187)]] 2446085 wikitext text/x-wiki dqlwi6w9su3c0id6phyyore87z5krm5 2446086 2446085 2026-08-12T14:33:13Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb-lock2002.codfw.wmnet with reason: host reimage 2446086 wikitext text/x-wiki i7gju5euqh1g91k28kyob9cbvk6ncvq 2446087 2446086 2026-08-12T14:37:01Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1324719|Partial revert "Use LockManager service instead of Database::getScopedLockAndFlush()" (T366938 T434596)]], [[gerrit:1324720|Partial revert "Use LockManager service instead of Database::getScopedLockAndFlush()" (T366938 T434596)]], [[gerrit:1324722|Bump mediawiki/mediawiki-codesniffer to v52.0.0 (T434187)]] synced to the testservers (see https://wikitech. 2446087 wikitext text/x-wiki 2kju5y9yeu94w0ju2ki4hb361sxfz3w 2446088 2446087 2026-08-12T14:37:49Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb-lock1002.eqiad.wmnet with OS trixie 2446088 wikitext text/x-wiki tlrz7f2gn3gamojor8aqdicfpq6s6q4 2446089 2446088 2026-08-12T14:37:51Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rdb-lock1002.eqiad.wmnet 2446089 wikitext text/x-wiki c61v7fk5rx8zclvsffy0eda6nd17l8m 2446090 2446089 2026-08-12T14:37:53Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2446090 wikitext text/x-wiki b245uarprd0zy06rob4p4vkk9zw0wz2 2446091 2446090 2026-08-12T14:39:20Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock2002.codfw.wmnet with reason: host reimage 2446091 wikitext text/x-wiki fi69ek293hmeo1x56cscy424dzbwy5d 2446092 2446091 2026-08-12T14:39:30Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock1003.eqiad.wmnet 2446092 wikitext text/x-wiki lce177jy93si1qrn6qgei4bfs6vr2yb 2446093 2446092 2026-08-12T14:39:32Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446093 wikitext text/x-wiki 19uzviv2nvg9t2g8wl31gz91pw5stlt 2446094 2446093 2026-08-12T14:44:10Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324719|Partial revert "Use LockManager service instead of Database::getScopedLockAndFlush()" (T366938 T434596)]], [[gerrit:1324720|Partial revert "Use LockManager service instead of Database::getScopedLockAndFlush()" (T366938 T434596)]], [[gerrit:1324722|Bump mediawiki/mediawiki-codesniffer to v52.0.0 (T434187)]] (duration: 11m 02s) 2446094 wikitext text/x-wiki lyrivykfb1h1e7qlqt9rq3waz9f1v62 2446095 2446094 2026-08-12T14:44:39Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1003.eqiad.wmnet - cgoubert@cumin2003" 2446095 wikitext text/x-wiki pa3qkidj2uhq5g88irhczerjdd9pncx 2446096 2446095 2026-08-12T14:45:04Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Enabling notifications 2446096 wikitext text/x-wiki 5l0sktn41ituxtcp1oh38yhcm67vxmr 2446097 2446096 2026-08-12T14:45:39Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1003.eqiad.wmnet - cgoubert@cumin2003" 2446097 wikitext text/x-wiki kzv3otd2zdvsfyu24sq2x5tg4zddtck 2446098 2446097 2026-08-12T14:45:41Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446098 wikitext text/x-wiki jdmlrt7vrm2a2xdv1oc3l4mk2ywulvt 2446099 2446098 2026-08-12T14:45:43Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock1003.eqiad.wmnet on all recursors 2446099 wikitext text/x-wiki 7kfoe2rx9u0kuehkosn70yqgh28u03x 2446100 2446099 2026-08-12T14:45:45Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock1003.eqiad.wmnet on all recursors 2446100 wikitext text/x-wiki 17rar93o0ksznrbi18xou44soxj7wqx 2446101 2446100 2026-08-12T14:46:09Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1003.eqiad.wmnet - cgoubert@cumin2003" 2446101 wikitext text/x-wiki aulepidrstm9qw4gh09qin2gnjpi6uq 2446102 2446101 2026-08-12T14:46:14Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1003.eqiad.wmnet - cgoubert@cumin2003" 2446102 wikitext text/x-wiki 2z9crc98r3ipqxex9hcm78u83xzqf4d 2446103 2446102 2026-08-12T14:47:24Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock1003.eqiad.wmnet with OS trixie 2446103 wikitext text/x-wiki 1gxr13i9af7qvdgoehwk88k9jzm0i14 2446104 2446103 2026-08-12T14:55:16Z Stashbot 7414 moritzm: powercycle ganeti2046 2446104 wikitext text/x-wiki 0h204znkjgtv2hw1ibjvs14grcnwubu 2446105 2446104 2026-08-12T14:55:35Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2446105 wikitext text/x-wiki dghihfuvioyeff70em0oynd0184yow7 2446107 2446105 2026-08-12T14:56:20Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2446107 wikitext text/x-wiki bnsch4mz9bm72qmo5ip4yu7hpeip5hr 2446109 2446107 2026-08-12T14:56:27Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2446109 wikitext text/x-wiki tq5hmyhy2qr4mwptbnvwidfxixt4e3f 2446110 2446109 2026-08-12T14:56:49Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2446110 wikitext text/x-wiki dyscmoqe84x9oxgcqf5jbqcrzvia0rw 2446111 2446110 2026-08-12T14:57:20Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb-lock2002.codfw.wmnet with OS trixie 2446111 wikitext text/x-wiki pxuo6ceiqwsanuk00vdh8ooq8ve3ugi 2446112 2446111 2026-08-12T14:57:22Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rdb-lock2002.codfw.wmnet 2446112 wikitext text/x-wiki tgp97lk6qjxyl6g9qco4pvrscruvqqh 2446113 2446112 2026-08-12T14:57:38Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb-lock1003.eqiad.wmnet with reason: host reimage 2446113 wikitext text/x-wiki 21poqjvtpbqpjtf6xi257yd30t0nddo 2446114 2446113 2026-08-12T14:57:40Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet 2446114 wikitext text/x-wiki 33543x12p90n6yf5zwq1ay6a96hrw76 2446115 2446114 2026-08-12T14:57:42Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446115 wikitext text/x-wiki bs18wim1ga712casdlp04ukobvbxjqt 2446118 2446115 2026-08-12T15:00:45Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1324338|InitialiseSettings: Enable 2FA warnings on more private wikis (T428103)]] 2446118 wikitext text/x-wiki bgog152ge3g9bhk5w7upuuao0scu7no 2446120 2446118 2026-08-12T15:02:18Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446120 wikitext text/x-wiki 34mg19ffhr7tub3ki36d7qoosuuamh9 2446121 2446120 2026-08-12T15:02:59Z Stashbot 7414 reedy@deploy1003: reedy: Backport for [[gerrit:1324338|InitialiseSettings: Enable 2FA warnings on more private wikis (T428103)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446121 wikitext text/x-wiki 37wphujtggkyrm1jj75urpb5gr2shde 2446122 2446121 2026-08-12T15:03:42Z Stashbot 7414 reedy@deploy1003: reedy: Continuing with deployment 2446122 wikitext text/x-wiki d3m97ncmhcittg1wvhukrwsynnbcht6 2446123 2446122 2026-08-12T15:03:53Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock1003.eqiad.wmnet with reason: host reimage 2446123 wikitext text/x-wiki 1qefpnd15e197mm2y68cc917ol4nwdt 2446124 2446123 2026-08-12T15:04:04Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446124 wikitext text/x-wiki sa6b3gzax3xyz3027o36wiz4ubhtotp 2446125 2446124 2026-08-12T15:04:07Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446125 wikitext text/x-wiki qyl67wwbawp3f480i26joozjrrkdgst 2446126 2446125 2026-08-12T15:04:08Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2446126 wikitext text/x-wiki hsa6js7ciw7slv0m5n8h2hduxsyl8vm 2446127 2446126 2026-08-12T15:04:11Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2446127 wikitext text/x-wiki n3wiqjkqekjvq0mid0d78yvcq7gasrc 2446128 2446127 2026-08-12T15:04:26Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446128 wikitext text/x-wiki ovul82mxnr1vvg6ppcizc5tgwe7n5fx 2446129 2446128 2026-08-12T15:07:50Z Stashbot 7414 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324338|InitialiseSettings: Enable 2FA warnings on more private wikis (T428103)]] (duration: 07m 02s) 2446129 wikitext text/x-wiki q62es3ki1ah7ycdnjy0rrtbequxmwdf 2446131 2446129 2026-08-12T15:11:34Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446131 wikitext text/x-wiki jkxtn8advlb4u3dostwr78u1arxlvko 2446132 2446131 2026-08-12T15:11:38Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446132 wikitext text/x-wiki 1fdlyz3pearbg1wufsnw1f1i85gb2mi 2446133 2446132 2026-08-12T15:11:40Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446133 wikitext text/x-wiki bfjurdp26rj9qsv5kjq1pxnknanjqi0 2446134 2446133 2026-08-12T15:11:43Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2446134 wikitext text/x-wiki dabf4g8r1ehfal1e3086ooyii18ywt1 2446135 2446134 2026-08-12T15:11:45Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2446135 wikitext text/x-wiki 5uuxxa5c2cskvqtcwij4nwjwonrqs6z 2446136 2446135 2026-08-12T15:11:54Z Stashbot 7414 cgoubert@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet 2446136 wikitext text/x-wiki pmillxlg8iuv7zt7u4goejc7gglpn3m 2446137 2446136 2026-08-12T15:12:03Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet 2446137 wikitext text/x-wiki 5k0v2r2roolquupbe3snfvzzowgkgh8 2446138 2446137 2026-08-12T15:12:05Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446138 wikitext text/x-wiki 413qgss5eckfzyckl7g9qomkj3lchrt 2446139 2446138 2026-08-12T15:16:26Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446139 wikitext text/x-wiki j2icxns9gawtqj3lccq4ej29i64raum 2446140 2446139 2026-08-12T15:16:31Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446140 wikitext text/x-wiki snkydpsi8t3uc3272sqv29i6iynpa23 2446141 2446140 2026-08-12T15:16:32Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446141 wikitext text/x-wiki mop6vyiq72pshvme5beklu6oe5mhqyu 2446142 2446141 2026-08-12T15:16:34Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2446142 wikitext text/x-wiki 5utxz57nx1r09gs4yl6v0rocji55qqu 2446143 2446142 2026-08-12T15:16:36Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2446143 wikitext text/x-wiki oae0rv4zo8iod5moujy8l92kk5p4jwt 2446144 2446143 2026-08-12T15:16:53Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446144 wikitext text/x-wiki ela7ubthaz9kifn1yfcxuniypo5qjyi 2446146 2446144 2026-08-12T15:18:05Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb-lock1003.eqiad.wmnet with OS trixie 2446146 wikitext text/x-wiki kaj177q9tpm5eaw5ntdd8so7ap77hsr 2446147 2446146 2026-08-12T15:18:07Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rdb-lock1003.eqiad.wmnet 2446147 wikitext text/x-wiki 36ly35rv1ro220pzkbubg8gw8a5iudw 2446148 2446147 2026-08-12T15:18:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1178.eqiad.wmnet with OS bookworm 2446148 wikitext text/x-wiki gunf09l09ith2yrc3ow470y1t5zklgh 2446149 2446148 2026-08-12T15:20:43Z Stashbot 7414 brennen@deploy1003: Started scap sync-world: Backport for [[gerrit:1324700|EventDetailsParticipantsModule: populate cache with non-local users (T434597)]] 2446149 wikitext text/x-wiki 9l9b40kctdq375gqi33vio5f6l1hvbj 2446150 2446149 2026-08-12T15:21:49Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446150 wikitext text/x-wiki j01svnmduied3us1xeo9831pcwyzzt2 2446151 2446150 2026-08-12T15:21:53Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446151 wikitext text/x-wiki ajurf58sffz6tfqtq0po4wk55lt4kcq 2446152 2446151 2026-08-12T15:21:55Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446152 wikitext text/x-wiki qq5i2eipo9h9pz7b8efo7u3hsmij867 2446153 2446152 2026-08-12T15:21:57Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2446153 wikitext text/x-wiki iwujkgt0tacic1e9u01lmrhiq4fkdrz 2446154 2446153 2026-08-12T15:22:00Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2446154 wikitext text/x-wiki mh64kceh368xf6vujn0ejks8hdkhm9v 2446155 2446154 2026-08-12T15:22:09Z Stashbot 7414 cgoubert@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet 2446155 wikitext text/x-wiki oidhet11wvqmh0jl1z9onvrk2xjdmx9 2446156 2446155 2026-08-12T15:22:47Z Stashbot 7414 brennen@deploy1003: brennen, daimona: Backport for [[gerrit:1324700|EventDetailsParticipantsModule: populate cache with non-local users (T434597)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446156 wikitext text/x-wiki kunl048h5z86twwi66hsfql2ludv9jy 2446157 2446156 2026-08-12T15:23:06Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446157 wikitext text/x-wiki adc55ir0px7rf49p511q1ra93rshzdu 2446158 2446157 2026-08-12T15:23:15Z Stashbot 7414 brennen@deploy1003: brennen, daimona: Continuing with deployment 2446158 wikitext text/x-wiki ppdnkv9m0d3jfbqvazr50akrgxnyr5s 2446159 2446158 2026-08-12T15:23:45Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2024,1031]*.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003 2446159 wikitext text/x-wiki rmxby7qrynbr5061ieentqtxepb1kga 2446162 2446159 2026-08-12T15:27:20Z Stashbot 7414 brennen@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324700|EventDetailsParticipantsModule: populate cache with non-local users (T434597)]] (duration: 06m 38s) 2446162 wikitext text/x-wiki 8rd1xoohdydxeqq07hu5umi2p5wblsh 2446164 2446162 2026-08-12T15:29:25Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1324733|WikimediaAntiAbuse: Enable personal info tag display on enwiki (T431292)]] 2446164 wikitext text/x-wiki 40enheos3y6zuddlyms5sz4vxnyt49a 2446165 2446164 2026-08-12T15:29:37Z Stashbot 7414 mforns@deploy1003: Started deploy [analytics/refinery@49c336c] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@49c336cd] 2446165 wikitext text/x-wiki 2vou9vitm0awzxmkph2w1fh6ttm7j3w 2446166 2446165 2026-08-12T15:30:10Z Stashbot 7414 mforns@deploy1003: Finished deploy [analytics/refinery@49c336c] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@49c336cd] (duration: 00m 32s) 2446166 wikitext text/x-wiki geivt3o79uuqlvjcpclr95sp0jug0lm 2446167 2446166 2026-08-12T15:30:22Z Stashbot 7414 mforns@deploy1003: Started deploy [analytics/refinery@49c336c]: Regular analytics weekly train [analytics/refinery@49c336cd] 2446167 wikitext text/x-wiki bu7uyszzskw15bg3jf4ho84u4at3g6p 2446168 2446167 2026-08-12T15:30:53Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitize-wiki (exit_code=99) Checking sanitization for wikis testwiki in section s3 2446168 wikitext text/x-wiki 9ez1wm2tb9ectwj8uv0b7o9fuande6i 2446169 2446168 2026-08-12T15:31:31Z Stashbot 7414 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1324733|WikimediaAntiAbuse: Enable personal info tag display on enwiki (T431292)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446169 wikitext text/x-wiki gj9mxp61qj6fd0z4o4c1e4i28ns1yo4 2446170 2446169 2026-08-12T15:33:20Z Stashbot 7414 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment 2446170 wikitext text/x-wiki 3oi3sduepo8gbkemj37mnimsghaagbf 2446171 2446170 2026-08-12T15:33:44Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2024,1031]*.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003 2446171 wikitext text/x-wiki d53e0vifnckyezyo05gaeh86f45n65i 2446172 2446171 2026-08-12T15:34:43Z Stashbot 7414 mforns@deploy1003: Finished deploy [analytics/refinery@49c336c]: Regular analytics weekly train [analytics/refinery@49c336cd] (duration: 04m 20s) 2446172 wikitext text/x-wiki ain6ufvr2net9rzocjnl5oy68j587qw 2446173 2446172 2026-08-12T15:35:40Z Stashbot 7414 mforns@deploy1003: Started deploy [analytics/refinery@49c336c] (thin): Regular analytics weekly train THIN [analytics/refinery@49c336cd] 2446173 wikitext text/x-wiki sxzsj95zd7g3tycqpns4e5dy1fdbrds 2446174 2446173 2026-08-12T15:35:56Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446174 wikitext text/x-wiki lkhuwh62dfr9kdrefurrkrafpq5doyw 2446175 2446174 2026-08-12T15:37:38Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324733|WikimediaAntiAbuse: Enable personal info tag display on enwiki (T431292)]] (duration: 08m 12s) 2446175 wikitext text/x-wiki ew75vbybbcc5tkf4s6xqycg676sw3f5 2446176 2446175 2026-08-12T15:37:41Z Stashbot 7414 mforns@deploy1003: Finished deploy [analytics/refinery@49c336c] (thin): Regular analytics weekly train THIN [analytics/refinery@49c336cd] (duration: 01m 59s) 2446176 wikitext text/x-wiki ntb9vkcpn70lxg04m50t6weewa92zyg 2446180 2446176 2026-08-12T15:42:32Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446180 wikitext text/x-wiki gn4ssa9gh4z2yfilfjzu3sidk5ctmmc 2446183 2446180 2026-08-12T15:52:28Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ganeti2046.codfw.wmnet 2446183 wikitext text/x-wiki cjapf2mejqq1op316vbjudhl9xhth1u 2446184 2446183 2026-08-12T15:52:30Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2046.codfw.wmnet 2446184 wikitext text/x-wiki 4t8qs162ercfftq3lrdauph6gd3uwd0 2446188 2446184 2026-08-12T15:56:14Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1272: New host 2446188 wikitext text/x-wiki fmhdjvj40ha2gsfojfwkuniyd90xxjd 2446191 2446188 2026-08-12T15:57:38Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446191 wikitext text/x-wiki e70pzctl0q3wvx4eiuaz50qcinjxfr3 2446194 2446191 2026-08-12T16:02:41Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-magru and not (P{cp7001*} or P{cp7009*}) and A:cp - 9.2.15 upgrade (T434620) 2446194 wikitext text/x-wiki hdomoycwg7cwd1kibuad1nrn0i1x4r7 2446195 2446194 2026-08-12T16:07:49Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply 2446195 wikitext text/x-wiki k6fqxc1j9stupvanmj01h3w5r59lcg1 2446196 2446195 2026-08-12T16:08:01Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply 2446196 wikitext text/x-wiki cjirmdmasvxpwpjy4epqlpuoxda30li 2446197 2446196 2026-08-12T16:08:18Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply 2446197 wikitext text/x-wiki 87f87mvz9k11mjyx07hhkfja76gf1s2 2446199 2446197 2026-08-12T16:09:00Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply 2446199 wikitext text/x-wiki 001qm4pyu36tpcpxrl7spoftzaydnlb 2446201 2446199 2026-08-12T16:10:07Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply 2446201 wikitext text/x-wiki mcyejns81kwzwguz48vi4x4jwx2bgpw 2446202 2446201 2026-08-12T16:10:47Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply 2446202 wikitext text/x-wiki l9eugfgms7m98zven8enomiejpc8b6r 2446204 2446202 2026-08-12T16:12:23Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003 2446204 wikitext text/x-wiki 42a5f3vqj0022f4tow8gfndl07ufcoc 2446205 2446204 2026-08-12T16:15:10Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446205 wikitext text/x-wiki g4k6dnjeqcdbw3u4hkm8jqrsfb7zhov 2446207 2446205 2026-08-12T16:16:03Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cp5022'] 2446207 wikitext text/x-wiki 55837341o05kygsfr8flocoku23idsc 2446209 2446207 2026-08-12T16:17:42Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446209 wikitext text/x-wiki 110omykyslzvpkztyxvtug78xycv74r 2446210 2446209 2026-08-12T16:18:25Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp5022'] 2446210 wikitext text/x-wiki 06znrgpyfq56u4gvu76i1yzsnqxo4l9 2446211 2446210 2026-08-12T16:18:53Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446211 wikitext text/x-wiki 6cuxkxe57el0ml9rznoavd5c9auwksg 2446215 2446211 2026-08-12T16:27:25Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply 2446215 wikitext text/x-wiki b84js1m2k4se354wwzd452lsro4jhzp 2446216 2446215 2026-08-12T16:27:38Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply 2446216 wikitext text/x-wiki ankhhor55lu5o8z4tdqzwjpzcmdg90z 2446218 2446216 2026-08-12T16:33:48Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1324765|WikimediaAntiAbuse: Enable PersonalInfoFlagNotifications (T431292)]] 2446218 wikitext text/x-wiki hygz0owhpkxss16mmsdgcf88xlpb3wd 2446221 2446218 2026-08-12T16:35:50Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1324765|WikimediaAntiAbuse: Enable PersonalInfoFlagNotifications (T431292)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446221 wikitext text/x-wiki e5lungzgwjw3y18rmfl0y55aigkx3e0 2446222 2446221 2026-08-12T16:36:42Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2446222 wikitext text/x-wiki len5a865tjjrlo7ekpfka0rxxnr8qe9 2446223 2446222 2026-08-12T16:37:24Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply 2446223 wikitext text/x-wiki 0mw361sbapaaqvgy2n45szis5v09fn3 2446224 2446223 2026-08-12T16:38:08Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply 2446224 wikitext text/x-wiki s9iywvecucaogsv7hj94pt4swp47c89 2446227 2446224 2026-08-12T16:39:40Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply 2446227 wikitext text/x-wiki sdxzz773fganzdvvnbpvgw81guus3jk 2446229 2446227 2026-08-12T16:40:12Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply 2446229 wikitext text/x-wiki n9w18q6ag5ih31wr24jhca1j1f8r4h2 2446230 2446229 2026-08-12T16:40:51Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324765|WikimediaAntiAbuse: Enable PersonalInfoFlagNotifications (T431292)]] (duration: 07m 02s) 2446230 wikitext text/x-wiki 32hqsj9gbyzhicqqc74h0c3kwpp259j 2446231 2446230 2026-08-12T16:41:27Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1272: New host 2446231 wikitext text/x-wiki 72zzx52l8dm6t9des2szm1sr7gex1j6 2446232 2446231 2026-08-12T16:44:32Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-main: apply 2446232 wikitext text/x-wiki 820oxq8774tqkrbh8bfsvaq41je0kn7 2446233 2446232 2026-08-12T16:44:42Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply 2446233 wikitext text/x-wiki sjxtlkca23y88141tdxclbauwhea9mu 2446236 2446233 2026-08-12T16:52:34Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: apply 2446236 wikitext text/x-wiki qih5m4engmqv0nbk02zbq95vapy2mp6 2446237 2446236 2026-08-12T16:53:19Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply 2446237 wikitext text/x-wiki 4pwg8qys8aur7dj1lr45b5nuxqhxots 2446239 2446237 2026-08-12T16:55:26Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s5 T434635 2446239 wikitext text/x-wiki 3ed3kl9l7aqd1qw34i2yv9hrzaldlug 2446240 2446239 2026-08-12T16:55:48Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2213 with weight 0 T434635', diff saved to https://phabricator.wikimedia.org/P96028 and previous config saved to /var/cache/conftool/dbconfig/20260812-165544-cwilliams.json 2446240 wikitext text/x-wiki 3h4jtituyopwwh2anrid5frqq9k6ncv 2446246 2446240 2026-08-12T17:01:10Z Stashbot 7414 cezmunsta: Starting s5 codfw failover from db2192 to db2213 - T434635 2446246 wikitext text/x-wiki qka7eu8i64rrn1w3iwhsmr22gj6v4ze 2446247 2446246 2026-08-12T17:01:56Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2213 to s5 primary T434635', diff saved to https://phabricator.wikimedia.org/P96029 and previous config saved to /var/cache/conftool/dbconfig/20260812-170152-cwilliams.json 2446247 wikitext text/x-wiki b76rdh4seearha2ejmnto2j31bz0z8f 2446248 2446247 2026-08-12T17:03:42Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2192 T434635', diff saved to https://phabricator.wikimedia.org/P96030 and previous config saved to /var/cache/conftool/dbconfig/20260812-170338-cwilliams.json 2446248 wikitext text/x-wiki qrdubvy9ve6uwt4llqbl3qk5b4d1368 2446249 2446248 2026-08-12T17:03:51Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply 2446249 wikitext text/x-wiki 20erzo5tp5c8wssrhcwvlpaf3mvdnkc 2446250 2446249 2026-08-12T17:04:33Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply 2446250 wikitext text/x-wiki 5fnnn1k0hg1ajudbhkael3lt19dyzya 2446252 2446250 2026-08-12T17:11:57Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host apifeatureusage2001.codfw.wmnet with OS bookworm 2446252 wikitext text/x-wiki sok3omja4hge71xfhtmxrdyu9fukktl 2446256 2446252 2026-08-12T17:18:50Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance 2446256 wikitext text/x-wiki 2b2amfl6yqfvtgu516ioislqxlq5t6w 2446257 2446256 2026-08-12T17:22:24Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446257 wikitext text/x-wiki g5a2nqc1361sju6qwfe70oaajcxx5oo 2446258 2446257 2026-08-12T17:28:03Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cp5022'] 2446258 wikitext text/x-wiki q2d9lqsq5cg9rz3y3gzk6lfcfx2fzo0 2446259 2446258 2026-08-12T17:28:49Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446259 wikitext text/x-wiki rsmjyanfifkn6blqd2rgubhdn8i0rj8 2446260 2446259 2026-08-12T17:31:17Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-magru and not (P{cp7001*} or P{cp7009*}) and A:cp - 9.2.15 upgrade (T434620) 2446260 wikitext text/x-wiki rs9ob6tdqdabk3sjv5utnemx9o9766l 2446261 2446260 2026-08-12T17:35:24Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp5022'] 2446261 wikitext text/x-wiki 5hmsnqw37w4khotgltxb9dn39xil60c 2446262 2446261 2026-08-12T17:36:16Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446262 wikitext text/x-wiki a1qggy5rb0biijlb3wi08b9h3isosim 2446267 2446262 2026-08-12T17:41:50Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cp5022'] 2446267 wikitext text/x-wiki g4752dtq81dt6wzu0elp79zc9dmvogo 2446268 2446267 2026-08-12T17:42:02Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446268 wikitext text/x-wiki 8h6vf6v1judt9i8di90tl56c5jqproi 2446270 2446268 2026-08-12T17:47:42Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cp5022'] 2446270 wikitext text/x-wiki 0xx6b0pxm0nhgdgufx9h71wfomjv9q3 2446271 2446270 2026-08-12T17:47:58Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446271 wikitext text/x-wiki 7zaj4zt7gm6zhl7rurczong4jy9fqnt 2446272 2446271 2026-08-12T17:53:42Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp5022'] 2446272 wikitext text/x-wiki p2fgh9h2c952016jsq60mvli24nt9zq 2446273 2446272 2026-08-12T17:55:31Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum-eqsin and A:durum 2446273 wikitext text/x-wiki 9feya9anoza2bi76ndhc3hl6hjuhy8r 2446274 2446273 2026-08-12T17:55:42Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] 2446274 wikitext text/x-wiki hbkvikdkjrltrtkr4su4ps4sy882399 2446275 2446274 2026-08-12T17:56:19Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2192: Security update 2446275 wikitext text/x-wiki fcmj89rnanz25zjwar9k4cq57q95ny0 2446276 2446275 2026-08-12T18:05:16Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp5022'] 2446276 wikitext text/x-wiki 4t0ah6psb8ianqd115t7im1pkqe5t0p 2446277 2446276 2026-08-12T18:07:30Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003 2446277 wikitext text/x-wiki 548qmm2493njbmcsw58zd958hiu5ayi 2446278 2446277 2026-08-12T18:08:57Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum-eqsin and A:durum 2446278 wikitext text/x-wiki l4583t2o003huooew4azkcdzpayh96a 2446279 2446278 2026-08-12T18:11:23Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T434668 2446279 wikitext text/x-wiki nj6anjtbzqx7a751sghv3w473tmqe0c 2446282 2446279 2026-08-12T18:18:32Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Set storage compatability to NONE — T433028 - eevans@cumin1003 2446282 wikitext text/x-wiki flaz2v8lp4bcctmh3sjvxb9kg2hkdkg 2446283 2446282 2026-08-12T18:18:48Z Stashbot 7414 Amir1: migrated testwiki entries from ce_worklist_articles to ce_invitation_list_articles (T426102) 2446283 wikitext text/x-wiki stiz1kmu5hm0kgujjjz8oxmw25nfqoq 2446284 2446283 2026-08-12T18:21:07Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T434668 2446284 wikitext text/x-wiki 57ek281wrbbdqerol4s2ayknx14nk1h 2446285 2446284 2026-08-12T18:21:40Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T434668 2446285 wikitext text/x-wiki 9l7ar6xapchozcer5hu4r8z3ggzs6gv 2446286 2446285 2026-08-12T18:25:05Z Stashbot 7414 Amir1: ce_invitation_list_articles created as empty on wikishared (T426102) 2446286 wikitext text/x-wiki mznh44mhk96rri8ztftt2ddy8lmeu7s 2446287 2446286 2026-08-12T18:31:05Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T434668 2446287 wikitext text/x-wiki pvemiww67oz67m03qv3u59li5k174kj 2446288 2446287 2026-08-12T18:41:30Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2192: Security update 2446288 wikitext text/x-wiki ld057nydbageugyi3t3hj7h9bmg465v 2446290 2446288 2026-08-12T18:51:09Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1321224|Rename ce_worklist_articles table to ce_invitation_list_articles (T426102)]] 2446290 wikitext text/x-wiki 9ifbtfm2an4k78g4kqexjilzgltgenj 2446291 2446290 2026-08-12T18:53:17Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup, daimona: Backport for [[gerrit:1321224|Rename ce_worklist_articles table to ce_invitation_list_articles (T426102)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446291 wikitext text/x-wiki tryiw8nvoo0k3j4vt1fz0jf4fa6d8j8 2446293 2446291 2026-08-12T18:53:52Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup, daimona: Continuing with deployment 2446293 wikitext text/x-wiki j89ndc2968h8we4ons3i3cqfn4b8sab 2446294 2446293 2026-08-12T18:57:59Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1321224|Rename ce_worklist_articles table to ce_invitation_list_articles (T426102)]] (duration: 06m 50s) 2446294 wikitext text/x-wiki dbrxps651na3e23gjeu5fgw011uegh1 2446295 2446294 2026-08-12T19:00:03Z Stashbot 7414 Amir1: data migrated on wikishared (T426102) 2446295 wikitext text/x-wiki 34a9apzphqpiyejn43u1x5olo44qu96 2446300 2446295 2026-08-12T19:09:01Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2446300 wikitext text/x-wiki 12s2vhoy5a0ot9nqrwiv84bygz1dwrk 2446301 2446300 2026-08-12T19:09:17Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2446301 wikitext text/x-wiki a4u0rrgyg3kary0fti2jhg84zw0z9h2 2446302 2446301 2026-08-12T19:10:07Z Stashbot 7414 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.15 refs T430834 2446302 wikitext text/x-wiki onqqvud9he59ey1g3gk1wau5cs9iznf 2446306 2446302 2026-08-12T19:19:59Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Set storage compatability to NONE — T433028 - eevans@cumin1003 2446306 wikitext text/x-wiki 49ntpr7gam3sjsjh2pnprdqqw1ihbct 2446307 2446306 2026-08-12T19:23:06Z Stashbot 7414 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.15 refs T430834 2446307 wikitext text/x-wiki n1ytqri44zubvfra2u13e9enonvz8bb 2446308 2446307 2026-08-12T19:26:02Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2446308 wikitext text/x-wiki l1myflcdz9tgsvjcmnv8aj75b71j412 2446309 2446308 2026-08-12T19:29:19Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446309 wikitext text/x-wiki ntvojxdg8zptl2r79b0jdvsyfv8rorv 2446310 2446309 2026-08-12T19:30:16Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446310 wikitext text/x-wiki 4wfmym103acm5rzmnsr4awqo79khko2 2446312 2446310 2026-08-12T19:32:59Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446312 wikitext text/x-wiki 2iicrl06geyncx7pand96k3qq9w4v15 2446314 2446312 2026-08-12T19:35:24Z Stashbot 7414 cdobbins@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie 2446314 wikitext text/x-wiki 8cm0ynhfd3j25ziko8jln1gbzx3jds6 2446315 2446314 2026-08-12T19:35:46Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie 2446315 wikitext text/x-wiki ityok2i5q2yi3gn64zsb0arg9ouvkyc 2446318 2446315 2026-08-12T19:47:12Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2446318 wikitext text/x-wiki 7u7kqjmrwt8fqcviqxs42j229qghfzw 2446328 2446318 2026-08-12T19:55:16Z Stashbot 7414 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.15 refs T430834 2446328 wikitext text/x-wiki lg7hv1tvdccc40h0ibyuwkr135ydors 2446335 2446328 2026-08-12T20:01:58Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1324794|WikimediaAntiAbuse: Enable logging channel (T431292)]] 2446335 wikitext text/x-wiki fnt2xh2nhbinc2xqzi36rht6sk0onvo 2446336 2446335 2026-08-12T20:04:20Z Stashbot 7414 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1324794|WikimediaAntiAbuse: Enable logging channel (T431292)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446336 wikitext text/x-wiki hvtg55ux1px9h074atga296zh7np5r7 2446337 2446336 2026-08-12T20:04:42Z Stashbot 7414 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment 2446337 wikitext text/x-wiki ko23u9llcpy07mhc7ief2mysb78rspf 2446338 2446337 2026-08-12T20:05:13Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage 2446338 wikitext text/x-wiki 2fff9b8witdmeq7qpkzg4rgy6p3a29w 2446339 2446338 2026-08-12T20:08:37Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage 2446339 wikitext text/x-wiki g09cy14ci92zrv5gk08o708018qk1z5 2446340 2446339 2026-08-12T20:08:46Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324794|WikimediaAntiAbuse: Enable logging channel (T431292)]] (duration: 06m 48s) 2446340 wikitext text/x-wiki 6cn4fba7otfycvfzcb4iy8md3xjb8jq 2446341 2446340 2026-08-12T20:09:26Z Stashbot 7414 Dreamy_Jazz: Evening UTC backport window done 2446341 wikitext text/x-wiki 8usbbr539o6rtkm7p8jvhuvv5q13lcg 2446342 2446341 2026-08-12T20:09:40Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Set storage compatability to NONE — T433028 - eevans@cumin1003 2446342 wikitext text/x-wiki 9lf6kahv65t8wk6q3ywjunla7e0vybo 2446343 2446342 2026-08-12T20:11:55Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1178.eqiad.wmnet with OS bookworm 2446343 wikitext text/x-wiki fz2blyp49jqlnzcspxrbqeqfgxtmz9u 2446344 2446343 2026-08-12T20:11:59Z Stashbot 7414 ryankemper@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1178.eqiad.wmnet with OS bookworm 2446344 wikitext text/x-wiki gvgfz4pg7gn7hptfguvhbkujmpue7xd 2446345 2446344 2026-08-12T20:13:31Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1178.eqiad.wmnet with OS bookworm 2446345 wikitext text/x-wiki dtqgiq5erchziriwwnkzo2jgosgzs8h 2446353 2446345 2026-08-12T20:28:04Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1178.eqiad.wmnet with reason: host reimage 2446353 wikitext text/x-wiki 85uzr2ixgz9lv6urzvy6lelltogswk0 2446354 2446353 2026-08-12T20:33:47Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2446354 wikitext text/x-wiki kwvbnsd5tc0ecbg8humv40xndg4bygj 2446355 2446354 2026-08-12T20:34:05Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1324277|Improve Math preference labels for SVG/MathJax/MathML (T433891)]] 2446355 wikitext text/x-wiki c7sb8mjpau82sil31qoom7vu95sfd5r 2446356 2446355 2026-08-12T20:34:20Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1178.eqiad.wmnet with reason: host reimage 2446356 wikitext text/x-wiki re2bj5v0admvi522056djbw8vrwa5jf 2446357 2446356 2026-08-12T20:36:24Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cdobbins@cumin1003" 2446357 wikitext text/x-wiki jo18p8tm8fhfecy31wqhvcnv5r77nf2 2446358 2446357 2026-08-12T20:37:08Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446358 wikitext text/x-wiki mveo68zip86idiv1p5dxgst86slwbzy 2446359 2446358 2026-08-12T20:38:22Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cdobbins@cumin1003" 2446359 wikitext text/x-wiki nm5pxdlp1hr02h7ox3ecjvqa3purwcb 2446360 2446359 2026-08-12T20:38:25Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5022.eqsin.wmnet with OS trixie 2446360 wikitext text/x-wiki mn8ujmcn2tu73x4zjaq1x8uywni56au 2446361 2446360 2026-08-12T20:38:54Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446361 wikitext text/x-wiki sdjx6g0021vuaa056t7e6cpjpy4x7tn 2446362 2446361 2026-08-12T20:39:14Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2446362 wikitext text/x-wiki ftq9631mwu998z5idnx6lt7y7jlwbzk 2446366 2446362 2026-08-12T20:51:41Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1324277|Improve Math preference labels for SVG/MathJax/MathML (T433891)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446366 wikitext text/x-wiki 9t8xm0opszqfts7t4pqgrwfcz219yy6 2446369 2446366 2026-08-12T20:54:02Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2446369 wikitext text/x-wiki egf1xaptwx1td38me6kr9zytmo4455v 2446371 2446369 2026-08-12T20:58:09Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1178.eqiad.wmnet with OS bookworm 2446371 wikitext text/x-wiki q3eozwznzhq7g6k7et7i3rlndqtgbw1 2446374 2446371 2026-08-12T21:05:47Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324277|Improve Math preference labels for SVG/MathJax/MathML (T433891)]] (duration: 31m 42s) 2446374 wikitext text/x-wiki b9jkhs1tcky2bb9rbjzsc5aec8937fb 2446375 2446374 2026-08-12T21:09:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1006 2446375 wikitext text/x-wiki q604n9691rim9eq3a6zgjpahcp1q20n 2446376 2446375 2026-08-12T21:10:38Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1006 2446376 wikitext text/x-wiki em9eyiym6p4mumjx1bnlsgkrwqg82tx 2446377 2446376 2026-08-12T21:10:59Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Set storage compatability to NONE — T433028 - eevans@cumin1003 2446377 wikitext text/x-wiki 0nzm6xlmbeko4eh2avps6xyemhy5677 2446378 2446377 2026-08-12T21:11:03Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2446378 wikitext text/x-wiki aab6wu8uhrr5oga8bsvzgstqsw224uw 2446381 2446378 2026-08-12T21:14:13Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446381 wikitext text/x-wiki sm3althhn681gaifthiawg4c44ilhqt 2446382 2446381 2026-08-12T21:14:36Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446382 wikitext text/x-wiki fnprkr1f0v1hhrf8j2zaezj05hxm61z 2446383 2446382 2026-08-12T21:17:47Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446383 wikitext text/x-wiki 1b2r99q16shiysqdhyuacbarzfip5j2 2446385 2446383 2026-08-12T21:30:10Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446385 wikitext text/x-wiki es2s4la5p3eii8nvpykm0t9i92p9hlk 2446386 2446385 2026-08-12T21:30:41Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446386 wikitext text/x-wiki eoilyx5w1xfsfl69zbtvkzrdv1bdozf 2446387 2446386 2026-08-12T21:31:14Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446387 wikitext text/x-wiki gongsfxos5g70nepm5bwsn3kg827g3w 2446388 2446387 2026-08-12T21:32:27Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2446388 wikitext text/x-wiki ga3y9tzpxi6gudl0ccd949zgf2f41nt 2446389 2446388 2026-08-12T21:36:29Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1324817|Create a maintenance script to backfill Special:AbuseReview (T434688)]], [[gerrit:1324818|Create a maintenance script to backfill Special:AbuseReview (T434688)]] 2446389 wikitext text/x-wiki lgoftvjc23qgcuowurcakwb20shr2oq 2446390 2446389 2026-08-12T21:39:41Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2446390 wikitext text/x-wiki deno9ntvbu52plg13cbu6ncb9amxfvc 2446391 2446390 2026-08-12T21:40:32Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1324817|Create a maintenance script to backfill Special:AbuseReview (T434688)]], [[gerrit:1324818|Create a maintenance script to backfill Special:AbuseReview (T434688)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446391 wikitext text/x-wiki 80hvjq4tvke38jbkrk7w4qu5j6u429s 2446392 2446391 2026-08-12T21:59:28Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2446392 wikitext text/x-wiki 2a43bp8uzpc3j9m7o2d7ykc7sev87p7 2446393 2446392 2026-08-12T22:17:50Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2446393 wikitext text/x-wiki 2iojmucekcemg3unjubxb5yi2q6ocyx 2446394 2446393 2026-08-12T22:21:59Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324817|Create a maintenance script to backfill Special:AbuseReview (T434688)]], [[gerrit:1324818|Create a maintenance script to backfill Special:AbuseReview (T434688)]] (duration: 45m 29s) 2446394 wikitext text/x-wiki l3ztkglqcw6r4k0p5jub0a69lohe3q0 2446395 2446394 2026-08-12T22:35:26Z Stashbot 7414 Dreamy_Jazz: Running `mwscript WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260101000000" --end-timestamp="20260102000000" --sleep=10` 2446395 wikitext text/x-wiki m9wpkemxppxxjqlqkfjc4brhe5wa0na 2446396 2446395 2026-08-12T22:40:56Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=testwiki --start-timestamp="20200101010101" --end-timestamp="20260816010101" --sleep=60` 2446396 wikitext text/x-wiki j5mghj5qkn2l9hp3itb9ter115ve47w 2446397 2446396 2026-08-12T22:45:43Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=testwiki --start-timestamp="20200801010101" --end-timestamp="20260816010101" --sleep=15 --batch-size=5` 2446397 wikitext text/x-wiki shzqw5boch7lzbdpiwwfpxgxstnm2zr 2446398 2446397 2026-08-12T22:49:06Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260801000000" --end-timestamp="20260802000000" --sleep=2 --batch-size=10` 2446398 wikitext text/x-wiki n4lwej4qodv2xhlpbudzxfymrshzk7f 2446399 2446398 2026-08-12T22:59:52Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2446399 wikitext text/x-wiki gs96i44o99tnostz2wsu25es7hxziwg 2446400 2446399 2026-08-12T23:11:27Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1324409|Remove $wmg = $wg hacks in Collection (T119117)]] 2446400 wikitext text/x-wiki 8en44d3gl3o1h8mi35ytgl3e633fgtu 2446401 2446400 2026-08-12T23:13:35Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1324409|Remove $wmg = $wg hacks in Collection (T119117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446401 wikitext text/x-wiki qvdlj88huy0vs62go4hq8dbxlebyaw2 2446402 2446401 2026-08-12T23:14:03Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2446402 wikitext text/x-wiki kt8xjwft6k01i8qa6kwlgjr9xduc3rt 2446403 2446402 2026-08-12T23:18:10Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324409|Remove $wmg = $wg hacks in Collection (T119117)]] (duration: 06m 43s) 2446403 wikitext text/x-wiki au1spbkjnorwd4u2zlmzoh1pc9q7pob 2446408 2446403 2026-08-13T00:16:08Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1178 2446408 wikitext text/x-wiki 7hfs75g40ajik8qprund80xcz4rrf63 2446409 2446408 2026-08-13T00:16:10Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1178 2446409 wikitext text/x-wiki fm7gsp3jcndmudyey5oxxlhg68qw6ee 2446410 2446409 2026-08-13T00:19:33Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1142.eqiad.wmnet with OS bookworm 2446410 wikitext text/x-wiki co80cp2th9nnrwfatumbh2vvgf1uux0 2446411 2446410 2026-08-13T00:34:33Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1142.eqiad.wmnet with reason: host reimage 2446411 wikitext text/x-wiki lr3aq5rahygrohylf9kqeux1obw30us 2446412 2446411 2026-08-13T00:38:41Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1142.eqiad.wmnet with reason: host reimage 2446412 wikitext text/x-wiki m8defij2lwe8upfhun1qpc4f45k7ihx 2446414 2446412 2026-08-13T01:03:03Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1142.eqiad.wmnet with OS bookworm 2446414 wikitext text/x-wiki ai8u4qsqu68lb0pjepkc4k7fxfrgc63 2446416 2446414 2026-08-13T01:46:19Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1219.eqiad.wmnet with OS bookworm 2446416 wikitext text/x-wiki h1rgae2qcaig8ssmf30s94w4tm73qk2 2446428 2446416 2026-08-13T02:00:54Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1219.eqiad.wmnet with reason: host reimage 2446428 wikitext text/x-wiki hxxs1h92ei54urmyq7ks130aqq1n7w1 2446429 2446428 2026-08-13T02:00:57Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2446429 wikitext text/x-wiki tmrhp3af2u0k5v3lpsw1n87x6bfo5xf 2446435 2446429 2026-08-13T02:03:55Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1219.eqiad.wmnet with reason: host reimage 2446435 wikitext text/x-wiki bqng071m3duv9pomc16898482aq309v 2446437 2446435 2026-08-13T02:07:44Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 46s) 2446437 wikitext text/x-wiki 4fb5v2b5oxx2gti10ai845gfxjyyj9o 2446439 2446437 2026-08-13T02:18:37Z Stashbot 7414 ryankemper: T434494 `ryankemper@deploy1003:~$ echo 'https://stats.wikimedia.org/' | mwscript-k8s --attach -- purgeList.php` (default page got cached during yesterday's `an-web1001` reimage) 2446439 wikitext text/x-wiki 8wipvd3ql56squn0r1ri5boinzvlghu 2446440 2446439 2026-08-13T02:23:47Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1219.eqiad.wmnet with OS bookworm 2446440 wikitext text/x-wiki edx783vrwgmoec0i278h9bl2span03z 2446441 2446440 2026-08-13T02:29:35Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1151 2446441 wikitext text/x-wiki qqy24owjqw7bi316fi3v45k4bemvvef 2446442 2446441 2026-08-13T02:29:38Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1151 2446442 wikitext text/x-wiki 8m7m0beqya76z7hdkij6urw676ylt1d 2446444 2446442 2026-08-13T04:06:47Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2446444 wikitext text/x-wiki 3kgw12gvcxsglgyew81ajebfhtwoy1k 2446446 2446444 2026-08-13T05:27:02Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2446446 wikitext text/x-wiki 758qpl57sqr7f568upkeqkwp1y3z0o7 2446447 2446446 2026-08-13T05:40:26Z Stashbot 7414 arnaudb@dns1006: START - running authdns-update 2446447 wikitext text/x-wiki jifhldjbrmhxthx5oop5xqzprsyd7g6 2446448 2446447 2026-08-13T05:42:24Z Stashbot 7414 arnaudb@dns1006: END - running authdns-update 2446448 wikitext text/x-wiki meyy07d9scqqd9oms4j19kqpy9dguwb 2446449 2446448 2026-08-13T07:19:26Z Stashbot 7414 jayme@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on kubestagemaster2005.codfw.wmnet with reason: downtime because of hardware failure and no DRBD 2446449 wikitext text/x-wiki h9mx05cf1fdfb1c92e53g2wcjoa676g 2446450 2446449 2026-08-13T07:24:51Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis testwiki in section s3 2446450 wikitext text/x-wiki 89l7mi7rro7ksdb05z7s4pzqvy0f30b 2446452 2446450 2026-08-13T07:31:24Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis testwiki in section s3 2446452 wikitext text/x-wiki cr4i8axzzgphcwm727fnesp5w1bdrhn 2446454 2446452 2026-08-13T07:38:22Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1273.eqiad.wmnet with reason: Reboot 2446454 wikitext text/x-wiki ti553l6cgv5uxrlit8hdfihybnmj68b 2446456 2446454 2026-08-13T07:56:15Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1273 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P96035 and previous config saved to /var/cache/conftool/dbconfig/20260813-075611-marostegui.json 2446456 wikitext text/x-wiki th6384crwl4l4b7233883how76x6i78 2446458 2446456 2026-08-13T07:56:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1273: Pool in s7 2446458 wikitext text/x-wiki ie2e9ewttdzl12p1kqasn77pqxlhe7a 2446462 2446458 2026-08-13T08:15:02Z Stashbot 7414 Emperor: roll-restart apus frontends in eqiad 2446462 wikitext text/x-wiki 0t97x3dsrxsjp4gun1yfeosqclp7m1i 2446463 2446462 2026-08-13T08:15:30Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reboot-cluster 2446463 wikitext text/x-wiki eg0h0ez65nwxwzdkng43t1u0q0oagk4 2446465 2446463 2026-08-13T08:20:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1161.eqiad.wmnet onto db1275.eqiad.wmnet 2446465 wikitext text/x-wiki gasd3nrq1yu9aps8udgtj0k390rvq63 2446466 2446465 2026-08-13T08:20:20Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1161: Depool db1161.eqiad.wmnet to then clone it to db1275.eqiad.wmnet - marostegui@cumin1003 2446466 wikitext text/x-wiki bwp2aiadx3vnyqxxex0rmw4mwrofnpk 2446467 2446466 2026-08-13T08:20:35Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1161: Depool db1161.eqiad.wmnet to then clone it to db1275.eqiad.wmnet - marostegui@cumin1003 2446467 wikitext text/x-wiki kt1846vrhr9hjyemf51piq9vz9rmxiu 2446470 2446467 2026-08-13T08:22:16Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists2001.wikimedia.org 2446470 wikitext text/x-wiki fr9gge7m4c6bt9we3r9wdzm201j5g34 2446471 2446470 2026-08-13T08:22:51Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1151.eqiad.wmnet with OS bookworm 2446471 wikitext text/x-wiki re0ndd5el21kihmcbpxetast6sytptr 2446472 2446471 2026-08-13T08:23:19Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1151 2446472 wikitext text/x-wiki 5hwyt4ke52hfy1op9tt4ze9lq34fr17 2446473 2446472 2026-08-13T08:23:33Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host aphlict2001.codfw.wmnet 2446473 wikitext text/x-wiki cp8o0hg0th1r9nksy8to8ml122btulz 2446474 2446473 2026-08-13T08:26:04Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org 2446474 wikitext text/x-wiki kkv1s72fvl1i0kkfqckgipu2ifd6gvc 2446475 2446474 2026-08-13T08:27:32Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict2001.codfw.wmnet 2446475 wikitext text/x-wiki gso4djja4e78s3oxsvz6auwi48ctyiq 2446476 2446475 2026-08-13T08:27:54Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host aphlict1002.eqiad.wmnet 2446476 wikitext text/x-wiki p8y0dpc7oi4knnu8x87xgbi9q6n20v2 2446477 2446476 2026-08-13T08:28:23Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org 2446477 wikitext text/x-wiki gjts6m7b1ncvcj2z5u3enmi1wrpgp31 2446478 2446477 2026-08-13T08:28:43Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists2001.wikimedia.org 2446478 wikitext text/x-wiki b2fawfpedf7sh3ccqp8jactn0uwignc 2446479 2446478 2026-08-13T08:29:07Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org 2446479 wikitext text/x-wiki ixxirke5p4vm8ikk1f28gbxenezzjzl 2446480 2446479 2026-08-13T08:29:58Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host planet1003.eqiad.wmnet 2446480 wikitext text/x-wiki py29vlx4hvfmj09q2vr4lg0lvppt6zq 2446481 2446480 2026-08-13T08:30:12Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host planet2003.codfw.wmnet 2446481 wikitext text/x-wiki nioy2ntlcqvju1abzk16j6aecsm4pg2 2446482 2446481 2026-08-13T08:31:52Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict1002.eqiad.wmnet 2446482 wikitext text/x-wiki i4vvasdzgm9n8foh9m5gjay3h4ctv55 2446483 2446482 2026-08-13T08:32:23Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org 2446483 wikitext text/x-wiki png7z61inbmpmpmryg11clb6bz7kd6d 2446484 2446483 2026-08-13T08:32:37Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning 2446484 wikitext text/x-wiki fhh6ndl8vk5ursa1ybzg86httxrnx7d 2446485 2446484 2026-08-13T08:33:04Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org 2446485 wikitext text/x-wiki j3lj5dxsdn4ahfowuz5zvrxu3zjjso8 2446486 2446485 2026-08-13T08:33:56Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet1003.eqiad.wmnet 2446486 wikitext text/x-wiki idwwb5r6tyvb39lncunp2i76sjs39l5 2446487 2446486 2026-08-13T08:34:11Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org 2446487 wikitext text/x-wiki 8lex3k6q2hmciopdn61gt1wm6p06gss 2446488 2446487 2026-08-13T08:34:14Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet2003.codfw.wmnet 2446488 wikitext text/x-wiki m2000ljrcrucvfbpcatq79xwyoji1mj 2446489 2446488 2026-08-13T08:34:22Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org 2446489 wikitext text/x-wiki bx5is92n9zexkuhnjnwpgbg3vwvgf7i 2446490 2446489 2026-08-13T08:34:58Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) 2446490 wikitext text/x-wiki rcg4y1h7kz32q1ko6f3wz2s7236wvlm 2446491 2446490 2026-08-13T08:35:13Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host doc1004.eqiad.wmnet 2446491 wikitext text/x-wiki gvl3juhdjyyfrzzxppkvygey3eyedut 2446492 2446491 2026-08-13T08:35:38Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host etherpad1004.eqiad.wmnet 2446492 wikitext text/x-wiki 74kzf1o8x1kmwhtwwz0b8pavho1o244 2446493 2446492 2026-08-13T08:36:54Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org 2446493 wikitext text/x-wiki i5xag4lhnu0gvmplwqeoizw6jm96beb 2446494 2446493 2026-08-13T08:37:19Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host phab2003.codfw.wmnet 2446494 wikitext text/x-wiki 9y4qau2q760m6gdjkx3w4b5k08h9idn 2446495 2446494 2026-08-13T08:38:01Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host etherpad2002.codfw.wmnet 2446495 wikitext text/x-wiki 1x5o2dk7si8s5d1o32vgcs6y8m2lfn4 2446496 2446495 2026-08-13T08:39:05Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org 2446496 wikitext text/x-wiki 0vpqxdtgvqrixl7ra3b1qrpi5wn43xt 2446497 2446496 2026-08-13T08:39:10Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc1004.eqiad.wmnet 2446497 wikitext text/x-wiki k5tu2vket2p2oibsecnc68m32q7dvl3 2446498 2446497 2026-08-13T08:39:36Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad1004.eqiad.wmnet 2446498 wikitext text/x-wiki nczrpv7kbz4hynowy0ljimoudcsan7i 2446499 2446498 2026-08-13T08:40:07Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gerrit1003.wikimedia.org 2446499 wikitext text/x-wiki 08noy1gmwxmmzi3ufc6dw4shsghcn54 2446500 2446499 2026-08-13T08:40:18Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host doc2003.codfw.wmnet 2446500 wikitext text/x-wiki 0vwyo5vpjm00j1vna537p89hu2w3tch 2446501 2446500 2026-08-13T08:41:07Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org 2446501 wikitext text/x-wiki 8kcmt58ubdl6tp3mstog80079x9e98m 2446502 2446501 2026-08-13T08:41:43Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1273: Pool in s7 2446502 wikitext text/x-wiki 5aykpms6glz31rolggo4so54r7uv919 2446503 2446502 2026-08-13T08:41:58Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host stewards1001.eqiad.wmnet 2446503 wikitext text/x-wiki tfvdpiltruev0b0yvwug73cv5rw1zeu 2446504 2446503 2026-08-13T08:42:01Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad2002.codfw.wmnet 2446504 wikitext text/x-wiki a1amporhv9d42kt9odxigv12slsi09l 2446505 2446504 2026-08-13T08:42:25Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host stewards2001.codfw.wmnet 2446505 wikitext text/x-wiki 5p401fo06dqu4cyulok8s1z692xakah 2446507 2446505 2026-08-13T08:43:06Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab2003.codfw.wmnet 2446507 wikitext text/x-wiki 84xneqs8mf0k5fu64mp9sy5n3fqmncm 2446508 2446507 2026-08-13T08:43:40Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1004.eqiad.wmnet 2446508 wikitext text/x-wiki 02to8dvc2e8aw2mf9rh8eod84lukk3k 2446509 2446508 2026-08-13T08:44:08Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc2003.codfw.wmnet 2446509 wikitext text/x-wiki dsptr7p7ihsl7z9d4qfsra1af5bo6ui 2446510 2446509 2026-08-13T08:44:46Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts2002.codfw.wmnet 2446510 wikitext text/x-wiki 6khq0kgby38iz9ctq7nbkp8vpuodzdm 2446511 2446510 2026-08-13T08:45:20Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reboot-cluster 2446511 wikitext text/x-wiki 4afyu2ldjvwsvsxyzrhl2vtuztr8qma 2446512 2446511 2026-08-13T08:45:24Z Stashbot 7414 Emperor: roll-restart apus frontends in codfw 2446512 wikitext text/x-wiki klyb91sqrtkigvc5mufgvn4zgl5gpj0 2446513 2446512 2026-08-13T08:45:43Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards1001.eqiad.wmnet 2446513 wikitext text/x-wiki 4pikthl0yhck32x5mpxl1a72qsisw7x 2446514 2446513 2026-08-13T08:45:55Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit1003.wikimedia.org 2446514 wikitext text/x-wiki 3aea18vmug3sen6h6a335vo6uf4h2b3 2446515 2446514 2026-08-13T08:46:13Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards2001.codfw.wmnet 2446515 wikitext text/x-wiki blqxnbkg4mxktbj54f4bdh0jxqkmnx1 2446516 2446515 2026-08-13T08:46:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Pool db1161.eqiad.wmnet in after cloning 2446516 wikitext text/x-wiki tnyr3l8vvg7y76x9yqj76vjq74jt7of 2446517 2446516 2026-08-13T08:47:03Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab-runner1002.eqiad.wmnet 2446517 wikitext text/x-wiki 59jyyzcsdeyb7bji6p4p1j8dylltn4p 2446518 2446517 2026-08-13T08:47:14Z Stashbot 7414 hashar@deploy1003: Started scap sync-world: Backport for [[gerrit:1325416|Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default" (T434712)]] 2446518 wikitext text/x-wiki 1lwh9tjo5dsxl18eor35gn92a629fs8 2446519 2446518 2026-08-13T08:47:22Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1004.eqiad.wmnet 2446519 wikitext text/x-wiki cmqido4vlnl94hmz3o2d80hjbwmna6s 2446520 2446519 2026-08-13T08:49:30Z Stashbot 7414 hashar@deploy1003: hashar: Backport for [[gerrit:1325416|Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default" (T434712)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446520 wikitext text/x-wiki 5eekrbe9vww19umima1pmbslrk9posf 2446521 2446520 2026-08-13T08:50:29Z Stashbot 7414 hashar@deploy1003: hashar: Continuing with deployment 2446521 wikitext text/x-wiki rklfdjmx80zgpryxt1cr506q5rgav3x 2446522 2446521 2026-08-13T08:51:13Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts2002.codfw.wmnet 2446522 wikitext text/x-wiki jocw2l5fnw5o4vwt1053wl9tpe02ual 2446523 2446522 2026-08-13T08:51:41Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s6 T434754 2446523 wikitext text/x-wiki 994vsxhplcze13vndhro8rsxhi4obpp 2446524 2446523 2026-08-13T08:51:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2229 with weight 0 T434754', diff saved to https://phabricator.wikimedia.org/P96043 and previous config saved to /var/cache/conftool/dbconfig/20260813-085151-cwilliams.json 2446524 wikitext text/x-wiki 6lru90jhoo0e3wlr2owamlp8fz7fw2q 2446525 2446524 2026-08-13T08:53:12Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1002.eqiad.wmnet 2446525 wikitext text/x-wiki 647gh0a1dxyg02et09snag601cmo75j 2446526 2446525 2026-08-13T08:53:34Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab-runner1003.eqiad.wmnet 2446526 wikitext text/x-wiki q23fd26lhdv4cb7rgvr88r3ce0hs4jm 2446528 2446526 2026-08-13T08:54:36Z Stashbot 7414 hashar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325416|Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default" (T434712)]] (duration: 07m 22s) 2446528 wikitext text/x-wiki izsw58ge6ur4hwm00fkzpb6qb21ehnp 2446529 2446528 2026-08-13T08:57:15Z Stashbot 7414 cezmunsta: Starting s6 codfw failover from db2214 to db2229 - T434754 2446529 wikitext text/x-wiki g3nbk5iggxeekfoblmakchdccypydf0 2446530 2446529 2026-08-13T08:57:56Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2229 to s6 primary T434754', diff saved to https://phabricator.wikimedia.org/P96044 and previous config saved to /var/cache/conftool/dbconfig/20260813-085752-cwilliams.json 2446530 wikitext text/x-wiki 8ozhwu3b0ly0ildclp84j2b87giw4iw 2446531 2446530 2026-08-13T08:59:42Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1003.eqiad.wmnet 2446531 wikitext text/x-wiki 7cg4ro6aftwzh4ewfvt69yem47bv3v0 2446532 2446531 2026-08-13T08:59:46Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab-runner1004.eqiad.wmnet 2446532 wikitext text/x-wiki rxlvs1281v0jdx6emsatlb29u1r8i4v 2446533 2446532 2026-08-13T09:00:05Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2214 T434754', diff saved to https://phabricator.wikimedia.org/P96045 and previous config saved to /var/cache/conftool/dbconfig/20260813-090001-cwilliams.json 2446533 wikitext text/x-wiki k89r4tluxv31jdj3c9tf0aniou8el32 2446535 2446533 2026-08-13T09:01:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2446535 wikitext text/x-wiki i634qtptwtpthk3od36bfocs7tbyc1i 2446537 2446535 2026-08-13T09:03:29Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 7 hosts with reason: reboot 2446537 wikitext text/x-wiki q77qfodumth8zo0u0ei1eab1zdumc7m 2446538 2446537 2026-08-13T09:05:02Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) 2446538 wikitext text/x-wiki 56j1sdevg5r6znqj1b9sf0hlif0qjpi 2446540 2446538 2026-08-13T09:05:56Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1004.eqiad.wmnet 2446540 wikitext text/x-wiki oal63g0x34ngb5s2xpvtphs40a7i3oa 2446541 2446540 2026-08-13T09:06:05Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2002.codfw.wmnet 2446541 wikitext text/x-wiki agaed7ox9bwl7eu2esw6c956tilpsf6 2446543 2446541 2026-08-13T09:07:13Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host apus-be2004.codfw.wmnet 2446543 wikitext text/x-wiki hiknejy0ybu3qce5wjh6yxd7swus6xd 2446544 2446543 2026-08-13T09:09:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.upgrade for 1 hosts 2446544 wikitext text/x-wiki sgzpz3msxk629gnatdg0sc3qh3b4wu9 2446545 2446544 2026-08-13T09:09:17Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2214: Upgrading db2214.codfw.wmnet 2446545 wikitext text/x-wiki 0szi2t4q83yxz7o99smc746q870631t 2446546 2446545 2026-08-13T09:09:21Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2214: Upgrading db2214.codfw.wmnet 2446546 wikitext text/x-wiki 2p5kc5su9zce38lc5sfj5xvji26kerl 2446547 2446546 2026-08-13T09:12:26Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2002.codfw.wmnet 2446547 wikitext text/x-wiki 6fcg3lky6it1mf58grc8r2oxxyz7fda 2446548 2446547 2026-08-13T09:12:30Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2003.codfw.wmnet 2446548 wikitext text/x-wiki d1gpm7zqh3bkvktvs8f0dbdcpte434t 2446549 2446548 2026-08-13T09:13:10Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2004.codfw.wmnet 2446549 wikitext text/x-wiki rnibjvylo2lvwekzvgs7vyy9a02ulto 2446550 2446549 2026-08-13T09:14:13Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host apus-be2005.codfw.wmnet 2446550 wikitext text/x-wiki 06nldog8duh3jyc8zzdj0448kl19fv3 2446551 2446550 2026-08-13T09:16:08Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Upgrade of db2214.codfw.wmnet completed 2446551 wikitext text/x-wiki imd26l1idlsgajvlmi5v9z8o519ipyy 2446552 2446551 2026-08-13T09:18:17Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 7 hosts 2446552 wikitext text/x-wiki qeewzkuiuj3xvdlp3dyqaxsil5m6q0u 2446554 2446552 2026-08-13T09:18:25Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 7 hosts 2446554 wikitext text/x-wiki eyhk1pyzxeujzl6zcy5wqwwhikb9geq 2446555 2446554 2026-08-13T09:18:49Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 7 hosts with reason: reboot 2446555 wikitext text/x-wiki fuulopxbd0axiheawrsuqo6c61jc6ag 2446556 2446555 2026-08-13T09:19:08Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2003.codfw.wmnet 2446556 wikitext text/x-wiki 33rloq15irdb5mwwfwoo7x90ebepith 2446557 2446556 2026-08-13T09:19:13Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab-runner2004.codfw.wmnet 2446557 wikitext text/x-wiki ja7afndg3rc4w48snrlxreewc209o9c 2446558 2446557 2026-08-13T09:19:49Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2005.codfw.wmnet 2446558 wikitext text/x-wiki tj6i0w31m92ezg2mxy4vyxcauaocyc5 2446560 2446558 2026-08-13T09:21:17Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host apus-be2006.codfw.wmnet 2446560 wikitext text/x-wiki nkbw1vxesk7im5vgowgiil725ebvy9u 2446561 2446560 2026-08-13T09:22:51Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.15 refs T430834 2446561 wikitext text/x-wiki gxduxmy71bw0brwmexcfe4kixi53mql 2446562 2446561 2026-08-13T09:25:12Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2004.codfw.wmnet 2446562 wikitext text/x-wiki k8qglucpfmf124kh78dvcg2ygly50f9 2446563 2446562 2026-08-13T09:25:32Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2006.codfw.wmnet 2446563 wikitext text/x-wiki oo08cm8tk3f3d1l62rr9789l7z6rihy 2446565 2446563 2026-08-13T09:27:46Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host moss-be2003.codfw.wmnet 2446565 wikitext text/x-wiki 22kiow8amaj1i03804byl90jt5psqxk 2446566 2446565 2026-08-13T09:27:49Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2446566 wikitext text/x-wiki 0voqmpg4fariiazn03uuh3uqwheuks7 2446568 2446566 2026-08-13T09:29:46Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2446568 wikitext text/x-wiki gwndbogr3k2bz4wo2ri3xw8m4zs4uqj 2446570 2446568 2026-08-13T09:30:29Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1325408|Improve performance of DB query in BackfillAbuseReview.php (T434688)]], [[gerrit:1325407|Improve performance of DB query in BackfillAbuseReview.php (T434688)]] 2446570 wikitext text/x-wiki 8ygaszafcbqwhtaayipdj01h8ck7at5 2446571 2446570 2026-08-13T09:31:52Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Pool db1161.eqiad.wmnet in after cloning 2446571 wikitext text/x-wiki fmr8vybx630er6lfmknc1xrq4x8s4da 2446572 2446571 2026-08-13T09:32:00Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2446572 wikitext text/x-wiki davy97h291xahqv1gkscwpc4fjmvgbs 2446573 2446572 2026-08-13T09:32:07Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1161.eqiad.wmnet onto db1275.eqiad.wmnet 2446573 wikitext text/x-wiki j2ab34d9hqyubtg56qf81qe7nksx6e9 2446574 2446573 2026-08-13T09:32:31Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1325408|Improve performance of DB query in BackfillAbuseReview.php (T434688)]], [[gerrit:1325407|Improve performance of DB query in BackfillAbuseReview.php (T434688)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2446574 wikitext text/x-wiki qo5wbeigh3f1qyfw6ip7x74av6j0jyo 2446575 2446574 2026-08-13T09:33:19Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2446575 wikitext text/x-wiki mutjvltz0vv2fgw5uziekl4cbrp7k9e 2446576 2446575 2026-08-13T09:33:55Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 7 hosts 2446576 wikitext text/x-wiki ls2lre1fpinfjwvnomb3ugbpanzpu9c 2446577 2446576 2026-08-13T09:34:00Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 7 hosts 2446577 wikitext text/x-wiki iqdgvfaza5urjnof2593bo9yimgqkni 2446578 2446577 2026-08-13T09:35:04Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2003.codfw.wmnet 2446578 wikitext text/x-wiki gj9qz3h8t9k4jhw7wrgi592axrrgcdg 2446579 2446578 2026-08-13T09:36:38Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1151 - btullis@cumin1003" 2446579 wikitext text/x-wiki mdpghqovi1v32pc9r2z2tif8nibck6m 2446581 2446579 2026-08-13T09:36:43Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: reboot 2446581 wikitext text/x-wiki c01a7pl9no0z7x9akl2vx0107qzojw9 2446582 2446581 2026-08-13T09:36:46Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1151 - btullis@cumin1003" 2446582 wikitext text/x-wiki p0kzf1f1g876gefsr2d34xmmds55ni9 2446583 2446582 2026-08-13T09:36:48Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446583 wikitext text/x-wiki myxlwtanyziw4hfbl237xpvqqcabaty 2446584 2446583 2026-08-13T09:36:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1151.eqiad.wmnet 13.36.64.10.in-addr.arpa 3.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2446584 wikitext text/x-wiki ks26ajbxfg2ljqmovf6lqsfyp50pzxp 2446585 2446584 2026-08-13T09:36:52Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1151.eqiad.wmnet 13.36.64.10.in-addr.arpa 3.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2446585 wikitext text/x-wiki kal9nze82jwxpyxf0mwutxk8785tp0b 2446586 2446585 2026-08-13T09:36:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1151 2446586 wikitext text/x-wiki ay4wpuikgyo8co87vv71wzu9hf586ki 2446587 2446586 2026-08-13T09:37:26Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325408|Improve performance of DB query in BackfillAbuseReview.php (T434688)]], [[gerrit:1325407|Improve performance of DB query in BackfillAbuseReview.php (T434688)]] (duration: 06m 57s) 2446587 wikitext text/x-wiki l16v5zd1g913g282lizu29g0p0vms22 2446588 2446587 2026-08-13T09:40:08Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1151 2446588 wikitext text/x-wiki e1ks9179q3alaph9b23wdispwyjt7ys 2446589 2446588 2026-08-13T09:40:10Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1151 2446589 wikitext text/x-wiki tl5yav29ww1tlgnqmw7p924po4gguni 2446591 2446589 2026-08-13T09:40:57Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backupmon1001.eqiad.wmnet 2446591 wikitext text/x-wiki 1q7vnwdmex8dq72knwj67s99g431w7k 2446592 2446591 2026-08-13T09:40:59Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backupmon1001.eqiad.wmnet 2446592 wikitext text/x-wiki me1pcujw27imkoe5t7pk83in1cmh518 2446593 2446592 2026-08-13T09:41:14Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=testwiki --start-timestamp="20260311000000" --end-timestamp="20260805000000" --sleep="5" --batch-size="2"` for T434688 2446593 wikitext text/x-wiki o4m9m228liazkq1qocn4ksy4weklsy3 2446595 2446593 2026-08-13T09:42:51Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1165.eqiad.wmnet onto db1279.eqiad.wmnet 2446595 wikitext text/x-wiki b3ul6dui4q02gmtmijxuyjerjxyz8wi 2446596 2446595 2026-08-13T09:42:55Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1165: Depool db1165.eqiad.wmnet to then clone it to db1279.eqiad.wmnet - marostegui@cumin1003 2446596 wikitext text/x-wiki jy8azlznwilhncpz99wxqws98ocx8ti 2446598 2446596 2026-08-13T09:43:30Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1165: Depool db1165.eqiad.wmnet to then clone it to db1279.eqiad.wmnet - marostegui@cumin1003 2446598 wikitext text/x-wiki 04candinkezsvvrii8lxxw4d6zx4oa5 2446599 2446598 2026-08-13T09:45:43Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning 2446599 wikitext text/x-wiki mwwtlsxzsraygq4otvt7pahuz49hjhe 2446600 2446599 2026-08-13T09:45:55Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 8 hosts with reason: reboot 2446600 wikitext text/x-wiki rctben1f39jj892h19hvglvsadmdl8h 2446602 2446600 2026-08-13T09:46:36Z Stashbot 7414 Dreamy_Jazz: Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260801000000" --end-timestamp="20260802000000" --sleep="3" --batch-size="5"` for T434688 2446602 wikitext text/x-wiki 0cog1r5ocya3tmpzmv1w8euq13hkju3 2446603 2446602 2026-08-13T09:55:53Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1151.eqiad.wmnet with reason: host reimage 2446603 wikitext text/x-wiki dk5m1p3fz3nx0wbumtom00xig0vou12 2446605 2446603 2026-08-13T09:59:14Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 8 hosts 2446605 wikitext text/x-wiki ic9412s68usojpfsgqzqdpz1dfxnr8w 2446606 2446605 2026-08-13T09:59:19Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 8 hosts 2446606 wikitext text/x-wiki 5aquuwfc2b86ia1lpv3cwvm7brpr7ki 2446607 2446606 2026-08-13T09:59:22Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1151.eqiad.wmnet with reason: host reimage 2446607 wikitext text/x-wiki puow7d8bahn4b1e79vqjx419tiajb43 2446608 2446607 2026-08-13T09:59:32Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2446608 wikitext text/x-wiki bntvx9xhd2w1wlpx8xj2a1z8q85xy7m 2446609 2446608 2026-08-13T09:59:35Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) 2446609 wikitext text/x-wiki igwlcrr2xe2es1w1llt0rpkz4ry2676 2446611 2446609 2026-08-13T10:00:48Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply 2446611 wikitext text/x-wiki 4je2mtjtr1r8v7qa8sxggjhtwyl60vx 2446613 2446611 2026-08-13T10:00:59Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply 2446613 wikitext text/x-wiki dmxspvua74jpq3rpfoecsl8mh6fmbwh 2446614 2446613 2026-08-13T10:01:11Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply 2446614 wikitext text/x-wiki goqr9fmcq0myxy5qlzmggzftp9vt2r4 2446615 2446614 2026-08-13T10:01:19Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2214: Upgrade of db2214.codfw.wmnet completed 2446615 wikitext text/x-wiki 0rslwaw3ljnaqsixetuu596gvetyh8h 2446616 2446615 2026-08-13T10:01:21Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for 1 hosts 2446616 wikitext text/x-wiki gweadpbkbe2157jh7gzzaxf7hifk65a 2446617 2446616 2026-08-13T10:01:39Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply 2446617 wikitext text/x-wiki 52fq1214j1slm34umq0mcrs6mdqpmz2 2446618 2446617 2026-08-13T10:02:07Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 7 hosts with reason: reboot 2446618 wikitext text/x-wiki 3twtdbsmztdf9zg3a7gicst8zc3hrsl 2446619 2446618 2026-08-13T10:06:53Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply 2446619 wikitext text/x-wiki 6mezdl4kbqwnrx3hl1uwuytqv0cv36s 2446620 2446619 2026-08-13T10:07:14Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply 2446620 wikitext text/x-wiki 9zwauji5kboquuwe1enz6qmlaqd8gjj 2446621 2446620 2026-08-13T10:15:57Z Stashbot 7414 moritzm: installing bind9 security updates (client-side tools/libs only) 2446621 wikitext text/x-wiki 2zpn0ds9vmfeidryaxbf4jmqrgz9ezx 2446622 2446621 2026-08-13T10:24:34Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1151.eqiad.wmnet with OS bookworm 2446622 wikitext text/x-wiki rxyml8j7mbrw3n7ipp84oau8qy2ic0h 2446623 2446622 2026-08-13T10:27:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1165: Pool db1165.eqiad.wmnet in after cloning 2446623 wikitext text/x-wiki 0nm6n5iwbt16bquxiwg2g815i2ny0sx 2446625 2446623 2026-08-13T10:35:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet 2446625 wikitext text/x-wiki 2ow9ibft1oa2yuyjkg7y133l7iixlwc 2446627 2446625 2026-08-13T10:38:31Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1151 from dbctl T434538', diff saved to https://phabricator.wikimedia.org/P96055 and previous config saved to /var/cache/conftool/dbconfig/20260813-103828-marostegui.json 2446627 wikitext text/x-wiki lj2gy7wphf8uozg41cupuerhdi085em 2446628 2446627 2026-08-13T10:39:32Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet 2446628 wikitext text/x-wiki 26x0pxu6204z174z5d2sc70zlg0zb23 2446630 2446628 2026-08-13T10:40:53Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[1216,1225,1239-1240].eqiad.wmnet with reason: reboot 2446630 wikitext text/x-wiki oyc02ajdi7eummz4h77d6h1nsnd69an 2446632 2446630 2026-08-13T10:42:28Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. 2446632 wikitext text/x-wiki gmux3xfgfc2omx521l26rk6raylttvd 2446633 2446632 2026-08-13T10:42:35Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. 2446633 wikitext text/x-wiki 8siag3b9r7idiv39rngq01zzho6929u 2446634 2446633 2026-08-13T10:43:21Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. 2446634 wikitext text/x-wiki b0ou5ikpw88w4o4uty1oobyo4t861xh 2446635 2446634 2026-08-13T10:44:23Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply 2446635 wikitext text/x-wiki mpr4vxczsvx3ekasepxy60ss9csruiz 2446636 2446635 2026-08-13T10:45:38Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet 2446636 wikitext text/x-wiki jsyegz3axkuzrlev1su3q2m9s1gve17 2446637 2446636 2026-08-13T10:45:46Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet 2446637 wikitext text/x-wiki kp9g9pez95api2971qrlo7niviuywyf 2446639 2446637 2026-08-13T10:50:35Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Enabling notifications and pooling 2446639 wikitext text/x-wiki ro2v0i8o40lguonc7g8t6oc21zji80f 2446640 2446639 2026-08-13T10:54:34Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply 2446640 wikitext text/x-wiki 8v02topqyb1xklendf1vcbgd8y95hcz 2446642 2446640 2026-08-13T10:57:19Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply 2446642 wikitext text/x-wiki o7mb3kvgcjkl6o8qoty932oaqcreu6h 2446643 2446642 2026-08-13T11:07:30Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply 2446643 wikitext text/x-wiki bhbri7znmp0kgna0hogee2qjpn89ooq 2446644 2446643 2026-08-13T11:12:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Pool db1165.eqiad.wmnet in after cloning 2446644 wikitext text/x-wiki rbsuy7aqd1krkky1l09u6ecoonpe02j 2446645 2446644 2026-08-13T11:12:53Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1165.eqiad.wmnet onto db1279.eqiad.wmnet 2446645 wikitext text/x-wiki cwqss9u0g05sfx8ew84isfwcmtr7hij 2446646 2446645 2026-08-13T11:17:35Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1213.eqiad.wmnet with OS bookworm 2446646 wikitext text/x-wiki tpvuooxsjbvlje3gyv1ydgtgc0wqx16 2446647 2446646 2026-08-13T11:19:59Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet 2446647 wikitext text/x-wiki 4x9kdhkiu3j59pmb0rziog7lr0yl1m6 2446648 2446647 2026-08-13T11:22:03Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet 2446648 wikitext text/x-wiki 7hgpx0dz6kzevsj5rtacxrr4xmou5eq 2446650 2446648 2026-08-13T11:28:09Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet 2446650 wikitext text/x-wiki 176rw82b5wo72c184nni0hhijbqi94p 2446651 2446650 2026-08-13T11:28:15Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet 2446651 wikitext text/x-wiki klhbktisvoudq5kpn0g5s8560b35y6q 2446652 2446651 2026-08-13T11:32:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1213.eqiad.wmnet with reason: host reimage 2446652 wikitext text/x-wiki j31kvvamgjfllkwuedzobevlfuz1gw9 2446653 2446652 2026-08-13T11:34:36Z Stashbot 7414 moritzm: remove ganeti3005 from esams03 cluster, hardware issues T434646 2446653 wikitext text/x-wiki pn4aog2ndczd07019h6q2d71d8wi5am 2446655 2446653 2026-08-13T11:34:53Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning 2446655 wikitext text/x-wiki m6ej5y8h25c2i1g5oavovw7uibb83j3 2446656 2446655 2026-08-13T11:35:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1167.eqiad.wmnet onto db1281.eqiad.wmnet 2446656 wikitext text/x-wiki 7220c1iijwsj8ms4f642xb53nzc7hmb 2446657 2446656 2026-08-13T11:35:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Depool db1167.eqiad.wmnet to then clone it to db1281.eqiad.wmnet - marostegui@cumin1003 2446657 wikitext text/x-wiki k1685iylekovq53t2h1l0yg7e5qs3u6 2446658 2446657 2026-08-13T11:35:54Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet 2446658 wikitext text/x-wiki 4u2t7kj28jy9b9bohbvirc7ggkalxu1 2446659 2446658 2026-08-13T11:35:57Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446659 wikitext text/x-wiki svjkfgmt5h2xw2iaialmgye5eitdaqb 2446660 2446659 2026-08-13T11:36:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Depool db1167.eqiad.wmnet to then clone it to db1281.eqiad.wmnet - marostegui@cumin1003 2446660 wikitext text/x-wiki tmq4i8wxxomk0ivhahoo114edbh4sop 2446661 2446660 2026-08-13T11:38:22Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1213.eqiad.wmnet with reason: host reimage 2446661 wikitext text/x-wiki 0puadc2lvnj0lwvgapigsyhgjv5g3nw 2446662 2446661 2026-08-13T11:40:26Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446662 wikitext text/x-wiki 6fqs5t0h9zcg4jbg4ce9i2d6o8w1y40 2446663 2446662 2026-08-13T11:40:31Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446663 wikitext text/x-wiki oh7jrbp8bjde5gelkg97z4malc0n16s 2446664 2446663 2026-08-13T11:40:33Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446664 wikitext text/x-wiki a1r28jylzscwlp8j3e8a49vu6jegq4g 2446665 2446664 2026-08-13T11:40:35Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2446665 wikitext text/x-wiki 92hkjhrybq11i49mcvu3pk0uui9y18c 2446666 2446665 2026-08-13T11:40:37Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2446666 wikitext text/x-wiki lg48cagz01un9tk7axwodbgx3vk8j4y 2446667 2446666 2026-08-13T11:40:53Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2446667 wikitext text/x-wiki fqh3pi92g1bwvy0kssrcnqsxq41t7lz 2446668 2446667 2026-08-13T11:41:39Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet 2446668 wikitext text/x-wiki j5pdxo90512k64atralyb0psxdeb0q6 2446669 2446668 2026-08-13T11:44:43Z Stashbot 7414 moritzm: installing Linux 5.10.262 on Bullseye hosts 2446669 wikitext text/x-wiki rjiibpcpmw9yobsb93dtg5hogbhmlcn 2446670 2446669 2026-08-13T11:44:51Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2446670 wikitext text/x-wiki c386wfc0t94za6zyqf788yfwihjony1 2446671 2446670 2026-08-13T11:50:00Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host apus-be1004.eqiad.wmnet 2446671 wikitext text/x-wiki dzs9el7cizmwru1sh4vv39uz4jtr8o7 2446672 2446671 2026-08-13T11:53:42Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet 2446672 wikitext text/x-wiki jfthdekczxlizmujcgk51zx8luza3ql 2446673 2446672 2026-08-13T11:53:50Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446673 wikitext text/x-wiki 5xnqsll5z3oh4u1s80lp2aw5lb0xxai 2446674 2446673 2026-08-13T11:53:54Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" 2446674 wikitext text/x-wiki qre3hj76ei1nega5swsc6bygebhtqju 2446675 2446674 2026-08-13T11:53:56Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2446675 wikitext text/x-wiki nh9to0rxdvt7wbn33eg9jfu2op92a75 2446676 2446675 2026-08-13T11:53:59Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors 2446676 wikitext text/x-wiki pufgcrqcejdnms4tc86r7wr9v88kikl 2446677 2446676 2026-08-13T11:54:01Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors 2446677 wikitext text/x-wiki cq4ku9wz1q1hi0389tlq7w9audnamxu 2446678 2446677 2026-08-13T11:54:09Z Stashbot 7414 cgoubert@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet 2446678 wikitext text/x-wiki 08l4x1xg9h0fu3tkrzrtkakckzlrxwh 2446679 2446678 2026-08-13T11:55:33Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be1004.eqiad.wmnet 2446679 wikitext text/x-wiki kn7rdb9u3cebyfohe2bxesgl09bzgjm 2446680 2446679 2026-08-13T11:56:30Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host apus-be1005.eqiad.wmnet 2446680 wikitext text/x-wiki oi0f534iv902l3b39w2a900n8k6u1ir 2446681 2446680 2026-08-13T11:58:57Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1213.eqiad.wmnet with OS bookworm 2446681 wikitext text/x-wiki 46df7zibibvfxj4hfrw25kq4fpge91e 0 14439 2446251 2404935 2026-08-12T17:06:55Z ADancy (WMF) 17694 /* Places we monitor */ Add SpiderPig log view link 2446251 wikitext text/x-wiki ij3jljkmwlt9faqx65l9f0forzzhmcj 0 17290 2445890 2445167 2026-08-12T11:59:43Z Stashbot 7414 hashar: integration: on Castor: `sudo rm -fR /srv/castor/*/*/*codehealth*/npm` # T427822 2445890 wikitext text/x-wiki htkwfr5kc148x2dsv735exn0lls8rm2 2445913 2445890 2026-08-12T12:31:01Z Stashbot 7414 hashar: integration: on Castor: `sudo rm -fR /srv/castor/*/*/mwext-phpunit-coverage*/npm` # T427922 2445913 wikitext text/x-wiki 6cd7cbfmuxxtio2g93sjmx4ty5i42bv 2446443 2445913 2026-08-13T03:05:12Z Stashbot 7414 TimStarling: created Produnto tables on beta T421436 2446443 wikitext text/x-wiki 0jcmi090lvxgvdeiu84r2itmv97hq6w 498 30942 2446451 2445567 2026-08-13T07:30:00Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049' (T431682) 2446451 wikitext text/x-wiki dtm7zsso26gez88x2gavlg9sjf99vfi 2446453 2446451 2026-08-13T07:38:07Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049' (T431682) 2446453 wikitext text/x-wiki 327f5uk2tvt9az4xxhok3ushsv2gy4b 2446455 2446453 2026-08-13T07:39:52Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051' (T431682) 2446455 wikitext text/x-wiki i37koak5a7e6rcq10el6as5zee6wfcg 2446457 2446455 2026-08-13T07:56:30Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051' (T431682) 2446457 wikitext text/x-wiki b9cbxnhwfhm2qrxap7v48q3vv9ijosm 2446459 2446457 2026-08-13T08:02:03Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054' (T431682) 2446459 wikitext text/x-wiki ghfw0hiyuqrqjospjrazfzx51kcsh2u 2446464 2446459 2026-08-13T08:19:52Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054' (T431682) 2446464 wikitext text/x-wiki nay77hh5s7te3hqx94qo9qj6x6z0h0u 2446469 2446464 2026-08-13T08:22:14Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055' (T431682) 2446469 wikitext text/x-wiki kzp7dlzqmnfbfidq0mzqlkz975khp6j 2446506 2446469 2026-08-13T08:42:47Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055' (T431682) 2446506 wikitext text/x-wiki dhun1of594v0hfv8d5s48d9b1yegbki 2446534 2446506 2026-08-13T09:01:15Z Stashbot 7414 godog: put cloudvirt1064 back into aggregate network-ovs 2446534 wikitext text/x-wiki ooucqu27rr0vgibkf2fovgzsu2l6422 2446536 2446534 2026-08-13T09:01:41Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056' (T431682) 2446536 wikitext text/x-wiki muqzcj0mxtmf44xwdx7a7nrtp6lhigu 2446553 2446536 2026-08-13T09:18:22Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056' (T431682) 2446553 wikitext text/x-wiki 09ovysx5boozd0ysyye4ac1ko5kxziw 2446569 2446553 2026-08-13T09:30:08Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057' (T431682) 2446569 wikitext text/x-wiki 29500x25hx133t3f598eg5yoptg6r8u 2446590 2446569 2026-08-13T09:40:27Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds 2446590 wikitext text/x-wiki aco5nyxt4py2pbtr9h3h91pvoki4bo3 2446601 2446590 2026-08-13T09:46:27Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057' (T431682) 2446601 wikitext text/x-wiki bojyj7fd1j0ic8gkjxghhkykhvf6v8u 2446612 2446601 2026-08-13T10:00:54Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) 2446612 wikitext text/x-wiki 658dfzi3b7ry4rooowsu7bssrfyuhej 498 55577 2446567 2441418 2026-08-13T09:28:36Z Stashbot 7414 godog: roll-restart components to pick up https://github.com/toolforge/paws/pull/521 2446567 wikitext text/x-wiki qhbx04qos8gh5r9lityptby09cc8q7d 0 67018 2446067 2439626 2026-08-12T14:16:14Z NguoiDungKhongDinhDanh 26425 /* Without a browser extension */ - 2446067 wikitext text/x-wiki 7wq2ow5yv6iuo1eu45le1hvpi1duxn0 0 90760 2446445 1886298 2026-08-13T05:17:56Z Effie Mouzeli (WMF) 12880 2446445 wikitext text/x-wiki bi7qgz6b69q5xayj4zlhkoacugsogir 498 117521 2446163 2440817 2026-08-12T15:27:56Z Stashbot 7414 dcaro: restart striker showing ldap errors 2446163 wikitext text/x-wiki khu72fl5jz11l6zcsb9mru459hmppkv 498 443261 2446003 2435824 2026-08-12T13:39:34Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-idp-1 2446003 wikitext text/x-wiki 3ewatdxd6nuz6nfazqkpj46c05fcax1 2446004 2446003 2026-08-12T13:40:20Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-idp-1 2446004 wikitext text/x-wiki ie7g3wl76mulie7wub14jglixxmb31n 0 444622 2446186 2440747 2026-08-12T15:53:30Z FNegri-WMF 32595 /* Run the functional tests in lima-kilo and check they are all green. */ Rename section to match the phab checklist 2446186 wikitext text/x-wiki rbgf9572k6r643afy2e4h7njoof1ur0 2446190 2446186 2026-08-12T15:57:22Z FNegri-WMF 32595 /* Test new k8s version in lima-kilo */ Fix formatting and add functional tests command 2446190 wikitext text/x-wiki juo2e2uzum5fbib8hfgejtg1y2ede0x 2446206 2446190 2026-08-12T16:15:49Z Andrewbogott 33 /* Upgrade control nodes */ 2446206 wikitext text/x-wiki dnn7xipmd46adcqtnqwjptm4jia3ib4 2446214 2446206 2026-08-12T16:27:16Z Andrewbogott 33 /* Upgrade control nodes */ 2446214 wikitext text/x-wiki ihuapqiam3osghztlvbe802jjn0cz8c 0 447941 2446208 2122695 2026-08-12T16:16:21Z FCeratto-WMF 39104 Add switchover helper stub 2446208 wikitext text/x-wiki j2ndmx2guo4eg4joov17qalna312vzo 0 449160 2446243 2445817 2026-08-12T16:57:33Z Dexbot 30554 Bot: Updating the report 2446243 wikitext text/x-wiki gq4ze6v3r4uuhns3lj3ayucepn9eutg 2446407 2446243 2026-08-13T00:01:34Z Dexbot 30554 Bot: Updating the report 2446407 wikitext text/x-wiki lf9f40ykcvpglxzq1rl84a84mq677d6 2446527 2446407 2026-08-13T08:53:50Z Dexbot 30554 Bot: Updating the report 2446527 wikitext text/x-wiki 1b77nn67zl7jndjpac9chj3f2etsijg 0 451075 2445932 2441322 2026-08-12T12:59:48Z DAdedoyin (WMF) 39150 /* Historical */ 2445932 wikitext text/x-wiki mdfmisayv788ihdmeah9uwhkc6ghx3s 2445933 2445932 2026-08-12T13:00:21Z DAdedoyin (WMF) 39150 /* Current */ 2445933 wikitext text/x-wiki fjl4xnlmu78eubmft38m6wytwfkx0sl 0 452278 2446379 2383868 2026-08-12T21:12:07Z BKing (WMF) 29017 /* Servers */ 2446379 wikitext text/x-wiki iyqvyxr1yalf6lvjvqv9pv5g7drkiqq 2446380 2446379 2026-08-12T21:13:36Z BKing (WMF) 29017 2446380 wikitext text/x-wiki rvaw7tuytx7rb26ecde866w2qana4c2 14 453467 2446219 2381029 2026-08-12T16:33:56Z Brian Young1 58539 2446219 wikitext text/x-wiki 9g5e30hhhd14qlcubxg307ujdzxex0u 2446384 2446219 2026-08-12T21:23:37Z TBurmeister (WMF) 27417 Reverted edit by [[Special:Contributions/Brian Young1|Brian Young1]] ([[User talk:Brian Young1|talk]]) to last revision by [[User:BryanDavis|BryanDavis]] 2381029 wikitext text/x-wiki 7lkwjqw6ek6xnikk9ocjwidbj4ca6jd 0 453597 2446220 2444322 2026-08-12T16:35:05Z DCaro (WMF) 18271 /* 2026 */ updated with T433405 2446220 wikitext text/x-wiki czydawz8ga1bgplagctnzyglm8p3bh2 0 453707 2446019 2445144 2026-08-12T13:53:02Z CWilliams-WMF 55316 Added steps to add the host to dbctl 2446019 wikitext text/x-wiki 6iy7bar6vwwbg6wa79d1uwnzmbfws0u 116 453906 2445974 2445523 2026-08-12T13:21:27Z Gitlabaccountapprovalbot 37332 @doublechek was approved. 2445974 wikitext text/x-wiki 2v5voa5zh6n6bvv4zgup5etz36kwsgh 498 454156 2446460 2411002 2026-08-13T08:07:00Z Stashbot 7414 dcaro: restarted webservice with php8.4 pre-built image 2446460 wikitext text/x-wiki p14ktb5bnrywqu8em4ohcu0s39dc5oq 0 459290 2446181 2445162 2026-08-12T15:50:07Z Mforns (WMF) 3929 added 2026-08-12 deployment 2446181 wikitext text/x-wiki lz1hik1row9dzaxjn1mhjr24bpt69tc 0 459326 2446331 2445488 2026-08-12T19:56:57Z Novem Linguae 23682 ce 2446331 wikitext text/x-wiki 1uc7hns8gusrzrjw6uofg2fevg03do4 2446332 2446331 2026-08-12T19:57:19Z Novem Linguae 23682 ce 2446332 wikitext text/x-wiki 2pc3pdahmkghk00kdcfocmyenos5rah 2446333 2446332 2026-08-12T19:58:30Z Novem Linguae 23682 ce 2446333 wikitext text/x-wiki 0y8d1xy01qjh0l7fzq28zti67yrwcd0 0 459699 2446413 2445716 2026-08-13T00:52:25Z Stashbot 7414 eileen: civicrm upgraded from 0d8a3fcd to d5420716 2446413 wikitext text/x-wiki ddbet4wu3grr3jdzy4t4t86oga89lhe 2446438 2446413 2026-08-13T02:10:18Z Stashbot 7414 ejegg: payments-wiki upgraded from fff3c1de to 44194999 2446438 wikitext text/x-wiki ik6k18ierfpzl5swwvqn60btdyc8j4s 0 459994 2446311 2426391 2026-08-12T19:32:17Z APaskulin (WMF) 14880 Fix link 2446311 wikitext text/x-wiki cotr1a72iu1r54ztzoxixbcnzmkdmxu 0 460161 2446081 2445771 2026-08-12T14:30:22Z T Arrow 1475 /* Architecture */ Remove reference to label service rewriting after slack discussion with Wikidata Platform Team which is now not part of the architecture 2446081 wikitext text/x-wiki 8okmlh27ne2ja5q6zvvyflao483p317 0 460194 2446269 2444141 2026-08-12T17:46:38Z APaskulin (WMF) 14880 Typo 2446269 wikitext text/x-wiki a7yvrvyg9fw4vse6l5y4fg9ku2matvx 0 460377 2446253 2444755 2026-08-12T17:15:59Z AWesterinen-WMF 54189 2446253 wikitext text/x-wiki eful7wl2y56gkz247cdhjjjftll4hk7 2446289 2446253 2026-08-12T18:43:56Z AWesterinen-WMF 54189 /* Query Rewrite Pattern */ 2446289 wikitext text/x-wiki 1ksz2ym4a1782uvy2310so0ulk40dgu 0 460538 2445926 2445270 2026-08-12T12:47:50Z Hasan Akgün (WMDE) 33690 Added LocalSettings.d usage and experimental/in progress statements with violations import 2445926 wikitext text/x-wiki fr458idefpcg5xw7o8uzit6wwgrdaj8 2445937 2445926 2026-08-12T13:06:25Z Hasan Akgün (WMDE) 33690 Add in progress badge 2445937 wikitext text/x-wiki gcjpizbmrpri25hot7j4093lj7rk2s2 2446638 2445937 2026-08-13T10:49:15Z Arthur Taylor (WMDE) 37356 /* Import Statements with Violations In progress */ 2446638 wikitext text/x-wiki 1n02j84afxc1dhby8rpwtj2zbbq2jnv 2 460540 2445936 2445665 2026-08-12T13:04:05Z Ottomata 88 added quotes and links to usefulness 2445936 wikitext text/x-wiki 3jdiwkeok9yxnj5m9nnu7xf2eoi6b2c 2445951 2445936 2026-08-12T13:15:28Z Ottomata 88 reformatted usefulness as table 2445951 wikitext text/x-wiki ng9pfvjs7koe4b6fbczu1rie5jrtvvk 2446005 2445951 2026-08-12T13:40:36Z Ottomata 88 /* Usefulness */ added WIP Access column 2446005 wikitext text/x-wiki b48tufjlw0hnbrtmzqvgkhpd4u58egk 2446011 2446005 2026-08-12T13:44:24Z Ottomata 88 reformatting & access options 2446011 wikitext text/x-wiki 4u4tpgububgcflvmmp7kgsto6e8bzna 2446305 2446011 2026-08-12T19:18:32Z Ottomata 88 Added row for External researchers 2446305 wikitext text/x-wiki olydzqvrvx6wiu625suioxjx8c0d53h 0 460545 2446641 2445415 2026-08-13T10:54:47Z JMonton-WMF 47887 Fill the page with descripcion and alerts 2446641 wikitext text/x-wiki bty2ngazdty2z7i55sdhzcq5awosxwq 3 460546 2445965 2026-08-12T13:18:24Z StrikerBot 8475 Welcome to Toolforge! 2445965 wikitext text/x-wiki 1m6obph4dzi26dctqqnrl3qe51e5zff 0 460547 2446203 2026-08-12T16:12:18Z CDanis (WMF) 13326 Redirected page to [[Linked Artifacts Cache]] 2446203 wikitext text/x-wiki fyeo1lvrkoq0yrm7eog4nm2vsjs8nme 0 460548 2446580 2026-08-13T09:36:42Z JMonton-WMF 47887 pageview-trending-relative documentation page - placeholder 2446580 wikitext text/x-wiki ebomoh3j41t8ama2n7fwrkxv2p7rjmd 2446654 2446580 2026-08-13T11:34:39Z JMonton-WMF 47887 Documentation for Pageview Trending Relative + alerts 2446654 wikitext text/x-wiki 4l0y0nt2h1elmyog48rtiz2vbfq064v