Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.4 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 1516 2419725 2408819 2026-05-26T18:07:55Z Stashbot 7414 AntiComposite: restart all bots -- disconnected 2419725 wikitext text/x-wiki muo5paq0igh80drf3ce0e6sqaa56egp 498 3086 2420137 2417919 2026-05-27T09:11:52Z Stashbot 7414 godog: delete obsolete limitranges - T427352 2420137 wikitext text/x-wiki mh4dalldden8nbwoisle5unoko4ymda 0 3214 2420155 2401094 2026-05-27T09:35:14Z Effie Mouzeli (WMF) 12880 /* Servers */ 2420155 wikitext text/x-wiki kmd9l1d9txgf82c7fqwo5eq0qn85tdm 2420163 2420155 2026-05-27T09:43:20Z BJensen-WMF 48942 /* Servers */ Add a link to deployment instructions. 2420163 wikitext text/x-wiki 3ninlg47agys7dnjmrjkmv2nu0ql2on 0 4108 2419298 2419138 2026-05-26T12:06:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1293706]] to Tuesday, May 26 UTC afternoon backport window 2419298 wikitext text/x-wiki 0kw54uktt9882y4a34brnzhwg9lco5j 2419506 2419298 2026-05-26T15:30:54Z Aude 39 /* {{Deployment_day|date=2026-05-26}} */ move mine to later deployment window 2419506 wikitext text/x-wiki b9kc5h3aatcoojiqrrgrnlmy31h9fkn 2419637 2419506 2026-05-26T17:06:37Z WBrown (WMF) 37696 /* {{Deployment_day|date=2026-05-26}} */ already done before the window 2419637 wikitext text/x-wiki gndrcrzemlnj46n371bilowjpvb3x6x 2419781 2419637 2026-05-26T19:15:20Z Aude 39 /* Week of May 25 */ move to mine tomorrow 2419781 wikitext text/x-wiki rcsmkjz60sfn1xpqfpjp0kko0lx8jmj 2419782 2419781 2026-05-26T19:16:11Z ScheduleDeploymentBot 37566 Add [[gerrit:1293794]] to Tuesday, May 26 UTC late backport window 2419782 wikitext text/x-wiki 5nppdqxuur3xwmz5sgeirgv7kpot59e 2419926 2419782 2026-05-26T23:00:38Z ScheduleDeploymentBot 37566 Add [[gerrit:1285412]] to Thursday, May 28 UTC late backport window 2419926 wikitext text/x-wiki hy5f8ahrqxgw2spso4y0x2nl43p5ed9 2420007 2419926 2026-05-27T06:02:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1270986]] to Wednesday, May 27 UTC morning backport window 2420007 wikitext text/x-wiki 9hqyeyfr5n8eucqk1vhc6lalgurusxl 2420013 2420007 2026-05-27T06:19:20Z Robertsky 30671 Undid revision [[Special:Diff/2420007|2420007]] by [[Special:Contributions/ScheduleDeploymentBot|ScheduleDeploymentBot]] ([[User talk:ScheduleDeploymentBot|talk]]) 2420013 wikitext text/x-wiki hy5f8ahrqxgw2spso4y0x2nl43p5ed9 2420096 2420013 2026-05-27T08:38:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1294217]] to Wednesday, May 27 UTC afternoon backport window 2420096 wikitext text/x-wiki d9d8w4pgmdju43qberlymne4r136smh 2420108 2420096 2026-05-27T08:51:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1290781]] to Wednesday, May 27 UTC afternoon backport window 2420108 wikitext text/x-wiki dpzc5tethwaaz12gu1gh7c5nz40qelt 2420153 2420108 2026-05-27T09:30:08Z ABran-WMF 37196 /* Friday, May 29 */ 2420153 wikitext text/x-wiki 59c8npp4bonk7bpvwxyg337mcb3ib7g 2420216 2420153 2026-05-27T11:02:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1294247]] to Wednesday, May 27 UTC afternoon backport window 2420216 wikitext text/x-wiki 0k9ka79lsc40fm2qg0fg47d7huwcit8 498 6726 2420065 2417649 2026-05-27T07:52:57Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2420065 wikitext text/x-wiki q09pv48uba9eijizqla2d32rnq8d3qa 2420066 2420065 2026-05-27T07:54:11Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2420066 wikitext text/x-wiki oy6onssjvx8v8ee3b7lwl7wh8s7wf4s 0 7919 2419295 2419294 2026-05-26T11:59:35Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293167|hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055|hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] (duration: 15m 26s) 2419295 wikitext text/x-wiki fee2dhw0opsanvu50j2bqfw1rp2w0ly 2419296 2419295 2026-05-26T12:01:44Z Stashbot 7414 XioNoX: start ssw1-a1-codfw network maintenance (no impact expected as the spines are redundant) 2419296 wikitext text/x-wiki k7beyocwrsgjd0lul42apec0zo0gdv5 2419297 2419296 2026-05-26T12:03:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92987 and previous config saved to /var/cache/conftool/dbconfig/20260526-120258-fceratto.json 2419297 wikitext text/x-wiki ov5wbo34pddphs3lnou2g4gpf570sk3 2419299 2419297 2026-05-26T12:08:23Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Migration of db2181.codfw.wmnet completed 2419299 wikitext text/x-wiki n349tllun4xbhilx70yhi5cyml36klv 2419300 2419299 2026-05-26T12:08:26Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2419300 wikitext text/x-wiki g8p78ytudydmpw9e3pe9xckvpt2bxhv 2419301 2419300 2026-05-26T12:08:45Z Stashbot 7414 fabfur: downtime, disable puppet and stop pybal for rack maintenance (T426199) 2419301 wikitext text/x-wiki 6qn8mi70ksiqdp46md98ffoi4z8t60f 2419303 2419301 2026-05-26T12:09:27Z Stashbot 7414 fabfur@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Planned downtime for rack maintenance 2419303 wikitext text/x-wiki tf05bu9j4yenexuuw72yqu110ur9swu 2419304 2419303 2026-05-26T12:13:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T426633)', diff saved to https://phabricator.wikimedia.org/P92990 and previous config saved to /var/cache/conftool/dbconfig/20260526-121306-fceratto.json 2419304 wikitext text/x-wiki s23lfeuige0goh0j6vi0gevvnbdseoz 2419305 2419304 2026-05-26T12:13:30Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance 2419305 wikitext text/x-wiki 4we92393dyt15eyisiq0062yv0p8hva 2419306 2419305 2026-05-26T12:13:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2225 (T426633)', diff saved to https://phabricator.wikimedia.org/P92991 and previous config saved to /var/cache/conftool/dbconfig/20260526-121336-fceratto.json 2419306 wikitext text/x-wiki 475fxe2g7cq2qnj3odqdz3pu5ux1w3v 2419307 2419306 2026-05-26T12:17:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mirror1001.wikimedia.org 2419307 wikitext text/x-wiki lebwrp0cozfdze1talf7ss9vm3mykib 2419308 2419307 2026-05-26T12:19:53Z Stashbot 7414 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2419308 wikitext text/x-wiki 2t4l5i57xn0dzan8i1xl97f4p46d2ye 2419309 2419308 2026-05-26T12:20:15Z Stashbot 7414 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2419309 wikitext text/x-wiki qs0bwfqtu189z900g8pqn4e0kq0qd0o 2419310 2419309 2026-05-26T12:20:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T426633)', diff saved to https://phabricator.wikimedia.org/P92993 and previous config saved to /var/cache/conftool/dbconfig/20260526-122044-fceratto.json 2419310 wikitext text/x-wiki dnwg4ug7bd6d8sg23fjdqndpgb1y8l9 2419311 2419310 2026-05-26T12:22:18Z Stashbot 7414 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2419311 wikitext text/x-wiki 9oo7hjnpyok9rylmetjjuw191lmk2x8 2419312 2419311 2026-05-26T12:23:01Z Stashbot 7414 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2419312 wikitext text/x-wiki mkpk92kjmvf9pksmt8ffcm0tnkp9nub 2419313 2419312 2026-05-26T12:23:13Z Stashbot 7414 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply 2419313 wikitext text/x-wiki nqew02ugeudubh3srls9l9qbi80ydvy 2419314 2419313 2026-05-26T12:23:45Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mirror1001.wikimedia.org 2419314 wikitext text/x-wiki pbdh41oapf0y17tzregoefi0z8u9qmy 2419315 2419314 2026-05-26T12:24:03Z Stashbot 7414 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply 2419315 wikitext text/x-wiki pirw8c47gtubssn6zhtd645jojtwrdu 2419316 2419315 2026-05-26T12:24:44Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a1-codfw,ssw1-a1-codfw IPv6,ssw1-a1-codfw.mgmt with reason: Switch maintenance 2419316 wikitext text/x-wiki 7mpo5k9rifzfy1mwl3i1jl3ybq09alm 2419317 2419316 2026-05-26T12:26:11Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* 2419317 wikitext text/x-wiki 7t3ssvcmxcbbezklfldrhsa01zg5avz 2419318 2419317 2026-05-26T12:26:38Z Stashbot 7414 fabfur: depooled cp204 for network activity (T426199) 2419318 wikitext text/x-wiki dxyyet9seu57h78jwq456b7lpu3uyi4 2419319 2419318 2026-05-26T12:30:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92995 and previous config saved to /var/cache/conftool/dbconfig/20260526-123052-fceratto.json 2419319 wikitext text/x-wiki 2x9ga6lugdo6klha7b22p9njbi5dbol 2419320 2419319 2026-05-26T12:33:59Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org 2419320 wikitext text/x-wiki jf8b62sv6xsrhtt4dj4299y5i054ptc 2419321 2419320 2026-05-26T12:35:29Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1214: Migration of db1214.eqiad.wmnet completed 2419321 wikitext text/x-wiki 9dae2z0klhkhqmxkc7c8j6rrwm9ow1g 2419322 2419321 2026-05-26T12:35:32Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2419322 wikitext text/x-wiki dabvipx2he9tfv8imrpfj4go7qp6jq8 2419323 2419322 2026-05-26T12:37:48Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org 2419323 wikitext text/x-wiki jmn31p85wsl6596hiioqpy1c92v9g6y 2419324 2419323 2026-05-26T12:41:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92997 and previous config saved to /var/cache/conftool/dbconfig/20260526-124059-fceratto.json 2419324 wikitext text/x-wiki mn1wnzduqx57gmr58lznvoco8haogb7 2419325 2419324 2026-05-26T12:51:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T426633)', diff saved to https://phabricator.wikimedia.org/P92998 and previous config saved to /var/cache/conftool/dbconfig/20260526-125105-fceratto.json 2419325 wikitext text/x-wiki 6vbawgbwhq6v7zkn0p9hsfnbqfvpa4r 2419326 2419325 2026-05-26T12:51:18Z Stashbot 7414 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2419326 wikitext text/x-wiki 9sz0mgbeewqgo5xvvbqnykv8yeuwl7z 2419327 2419326 2026-05-26T12:51:29Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance 2419327 wikitext text/x-wiki oq9gygjh9qukuv4oh957e7th2h7z9j8 2419328 2419327 2026-05-26T12:51:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2226 (T426633)', diff saved to https://phabricator.wikimedia.org/P92999 and previous config saved to /var/cache/conftool/dbconfig/20260526-125135-fceratto.json 2419328 wikitext text/x-wiki c9037i6az5z727gu6fxy8ptird8yahg 2419329 2419328 2026-05-26T12:58:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T426633)', diff saved to https://phabricator.wikimedia.org/P93000 and previous config saved to /var/cache/conftool/dbconfig/20260526-125834-fceratto.json 2419329 wikitext text/x-wiki alqs75kqa30wtmqi07u3cpttdeacw46 2419330 2419329 2026-05-26T13:00:44Z Stashbot 7414 topranks: deactivate CR BGP to doh2002 to test backup path via doh2001 2419330 wikitext text/x-wiki 731nc0rb02gm0ng4n882r1rpdjl65mh 2419331 2419330 2026-05-26T13:03:16Z Stashbot 7414 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2419331 wikitext text/x-wiki 16doppem19d0o3gb9f1megkb4yuyvqr 2419332 2419331 2026-05-26T13:04:09Z Stashbot 7414 kart_: Update Recommendation API to 2026-05-26-074931-production 2419332 wikitext text/x-wiki pmm915bbdj56mriw3bz3rjmuy8v9tx6 2419333 2419332 2026-05-26T13:04:39Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419333 wikitext text/x-wiki pr5k739cldcxvxnh0lrgewlvu88ruwx 2419334 2419333 2026-05-26T13:04:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2167: Upgrading db2167.codfw.wmnet 2419334 wikitext text/x-wiki o0o96jf8nuq6v71767n0700hxiogo54 2419335 2419334 2026-05-26T13:05:10Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1293177|Instrumentation: log new articles namespace and source (T422146)]] 2419335 wikitext text/x-wiki jzbmkgbq5r4q3j2n801qs3ol2vzvujr 2419336 2419335 2026-05-26T13:05:21Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2167: Upgrading db2167.codfw.wmnet 2419336 wikitext text/x-wiki 8cl8t6vlyelzxv0cy72l9wv35aop5bt 2419337 2419336 2026-05-26T13:07:02Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1293177|Instrumentation: log new articles namespace and source (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419337 wikitext text/x-wiki kwhsuj65hv2w8ul6f1zmiqtm1nz90ie 2419338 2419337 2026-05-26T13:07:14Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2167.codfw.wmnet with OS trixie 2419338 wikitext text/x-wiki fgalrasrg8nizob9hegswtle5q4iutg 2419340 2419338 2026-05-26T13:08:02Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2419340 wikitext text/x-wiki dkdl0tgpvfwy6vfzc90ujeygzcun5dg 2419341 2419340 2026-05-26T13:08:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93003 and previous config saved to /var/cache/conftool/dbconfig/20260526-130842-fceratto.json 2419341 wikitext text/x-wiki ps8iwyv2h3tc1b73uz09dofx8a521np 2419342 2419341 2026-05-26T13:12:19Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293177|Instrumentation: log new articles namespace and source (T422146)]] (duration: 07m 09s) 2419342 wikitext text/x-wiki ercbfsnn65btqllkpzewzpyl5mb20rw 2419343 2419342 2026-05-26T13:14:12Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293706|Disable the `no` language code for translation (T424613)]] 2419343 wikitext text/x-wiki py8g28j4xnzdu9z0l1c7d2166uc9a37 2419345 2419343 2026-05-26T13:16:05Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1293706|Disable the `no` language code for translation (T424613)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419345 wikitext text/x-wiki 22mc6i3rc6uv8t4c5sn6gv8a7iscqae 2419346 2419345 2026-05-26T13:18:34Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Continuing with deployment 2419346 wikitext text/x-wiki 80eoednzla5x2wf2msf10fum8hx4ioe 2419347 2419346 2026-05-26T13:18:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93004 and previous config saved to /var/cache/conftool/dbconfig/20260526-131850-fceratto.json 2419347 wikitext text/x-wiki 6poi5fr2dz6h5uv4097f8q83ufuemke 2419348 2419347 2026-05-26T13:20:47Z Stashbot 7414 ladsgroup@dns1004: START - running authdns-update 2419348 wikitext text/x-wiki rnfod9vtd8neiwnv68zinuztmuc5sff 2419349 2419348 2026-05-26T13:22:27Z Stashbot 7414 ladsgroup@dns1004: END - running authdns-update 2419349 wikitext text/x-wiki lmuhwrylx51z9w788dqqv87sv70pt8e 2419350 2419349 2026-05-26T13:22:43Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293706|Disable the `no` language code for translation (T424613)]] (duration: 08m 30s) 2419350 wikitext text/x-wiki 6m6bgnue0siu9cxez0l0h0ll0kax6i7 2419352 2419350 2026-05-26T13:25:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2167.codfw.wmnet with reason: host reimage 2419352 wikitext text/x-wiki hz9vt5uhjrsrd38mxthe4ly0iwkqsh8 2419353 2419352 2026-05-26T13:25:43Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1293662|Enable IRS Direct Reporting on testwiki (T425025)]] 2419353 wikitext text/x-wiki hyl5cxf7ihewsuq0kvavbiy61hu6mkp 2419354 2419353 2026-05-26T13:27:38Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1293662|Enable IRS Direct Reporting on testwiki (T425025)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419354 wikitext text/x-wiki sj179n7uxxlcer7vjoblspgy1r2193g 2419355 2419354 2026-05-26T13:28:02Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a2-codfw,lsw1-a2-codfw IPv6,lsw1-a2-codfw.mgmt with reason: Switch maintenance 2419355 wikitext text/x-wiki gplqgtv5gedrtmn5vqzyilroxoqppwc 2419356 2419355 2026-05-26T13:29:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T426633)', diff saved to https://phabricator.wikimedia.org/P93005 and previous config saved to /var/cache/conftool/dbconfig/20260526-132857-fceratto.json 2419356 wikitext text/x-wiki 2ponlcoi8nol0r54ku16nttwbd1o2vj 2419357 2419356 2026-05-26T13:29:20Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Switch maintenance 2419357 wikitext text/x-wiki mpyyb7yz7n8ptd4ahwxmxgct1m05r7g 2419358 2419357 2026-05-26T13:29:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance 2419358 wikitext text/x-wiki l9ynyppggwgr64dwhkscrq25g2sgqch 2419359 2419358 2026-05-26T13:29:26Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: host reimage 2419359 wikitext text/x-wiki 9ryc2tkmjkndk4h2gqqgfdwu294sbep 2419360 2419359 2026-05-26T13:29:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2238 (T426633)', diff saved to https://phabricator.wikimedia.org/P93006 and previous config saved to /var/cache/conftool/dbconfig/20260526-132927-fceratto.json 2419360 wikitext text/x-wiki tjb0pzsxwnxin3ogckuonho1jy21yyf 2419361 2419360 2026-05-26T13:29:51Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet 2419361 wikitext text/x-wiki frlfu5yhg8cepi0tull44jbnjrqsqyy 2419362 2419361 2026-05-26T13:30:58Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2419362 wikitext text/x-wiki bvqadfrrq0rfi8a1evq363tfqu3t967 2419363 2419362 2026-05-26T13:31:36Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet 2419363 wikitext text/x-wiki 924jv0d2qsmiazn33z71zfkgs353cz4 2419364 2419363 2026-05-26T13:34:07Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2196: switch maintenance 2419364 wikitext text/x-wiki hzca1v75us4goj644uh5a6r31zxyc35 2419365 2419364 2026-05-26T13:34:28Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: switch maintenance 2419365 wikitext text/x-wiki n9gkz7it4kecqhi6dsy0ni6przm7jp3 2419366 2419365 2026-05-26T13:34:54Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2221: switch maintenance 2419366 wikitext text/x-wiki 79d7gxnzxajw0g1rqs1kenh37m01mqn 2419367 2419366 2026-05-26T13:35:11Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293662|Enable IRS Direct Reporting on testwiki (T425025)]] (duration: 09m 28s) 2419367 wikitext text/x-wiki msepvzy7fpdv8tya97ir5w9ho5gk20g 2419368 2419367 2026-05-26T13:35:15Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: switch maintenance 2419368 wikitext text/x-wiki l9ce1owhems1r8yie7a30kxbpe287ec 2419369 2419368 2026-05-26T13:35:22Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2222: switch maintenance 2419369 wikitext text/x-wiki q0kyqg4919xtk4bmjjyogn46020be6g 2419370 2419369 2026-05-26T13:35:52Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: switch maintenance 2419370 wikitext text/x-wiki izxt8jbwub4gw1hc5rqggtfqfxg6csu 2419371 2419370 2026-05-26T13:36:01Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2223: switch maintenance 2419371 wikitext text/x-wiki lu2u4fmffslvldd4cig4jn6hpa4etkt 2419372 2419371 2026-05-26T13:36:21Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2223: switch maintenance 2419372 wikitext text/x-wiki 7uae48tchbr8hqsm6s2hk17z4gq9511 2419373 2419372 2026-05-26T13:36:43Z Stashbot 7414 XioNoX: reboot lsw1-a2-codfw for software upgrade - T426199 2419373 wikitext text/x-wiki pdxfsrg2g9tqskjea9pmc5umdc7l7aj 2419374 2419373 2026-05-26T13:36:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T426633)', diff saved to https://phabricator.wikimedia.org/P93011 and previous config saved to /var/cache/conftool/dbconfig/20260526-133656-fceratto.json 2419374 wikitext text/x-wiki gs2iwvjbynunq7gx4seku67l4jhfsih 2419378 2419374 2026-05-26T13:46:26Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2167.codfw.wmnet with OS trixie 2419378 wikitext text/x-wiki 07srv1kp2cwnidbjj7c5oetlfv0zly2 2419379 2419378 2026-05-26T13:47:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93012 and previous config saved to /var/cache/conftool/dbconfig/20260526-134703-fceratto.json 2419379 wikitext text/x-wiki i3o7dunfpufnghqbtfmax47rzflt69w 2419380 2419379 2026-05-26T13:48:24Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2419380 wikitext text/x-wiki jz3mbb700fytzo2osk3w2nem1e1q7y2 2419381 2419380 2026-05-26T13:49:44Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.decommission 2419381 wikitext text/x-wiki mkdgwe6qs1qkircgfkwcktyajmp8nw2 2419382 2419381 2026-05-26T13:50:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1013.eqiad.wmnet 2419382 wikitext text/x-wiki 0mgr7qq2dtcvbffsywsruv51p55mv4i 2419383 2419382 2026-05-26T13:53:25Z Stashbot 7414 Amir1: drop flaggedrevs tables on cawikinews (T423577) 2419383 wikitext text/x-wiki 7s3jq9vrdslixmmeamlre6gmrlcjgcn 2419384 2419383 2026-05-26T13:55:02Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2167: Migration of db2167.codfw.wmnet completed 2419384 wikitext text/x-wiki 0zkchapk7g9h7fudyiz6wjj2eb0hezd 2419385 2419384 2026-05-26T13:56:03Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1054.eqiad.wmnet with OS trixie 2419385 wikitext text/x-wiki oxs54e84rarb33pglp9b2z7lhcni7s9 2419386 2419385 2026-05-26T13:57:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93014 and previous config saved to /var/cache/conftool/dbconfig/20260526-135711-fceratto.json 2419386 wikitext text/x-wiki p0l4pfj905hknlil4v65gzntfzj213w 2419388 2419386 2026-05-26T14:00:52Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2419388 wikitext text/x-wiki lulig728exiu7tc335pqj0adbg3egwg 2419389 2419388 2026-05-26T14:04:38Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2419389 wikitext text/x-wiki cd14jhhw67v1lc1hubusq3tfo4rfsu9 2419390 2419389 2026-05-26T14:04:53Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2419390 wikitext text/x-wiki 3dldrc3ho4w1a6xmt4cu9dsll9wz39t 2419391 2419390 2026-05-26T14:04:56Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419391 wikitext text/x-wiki 4jbb3wl6pdilfwc2993lvmvaflt14q3 2419392 2419391 2026-05-26T14:04:58Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1013.eqiad.wmnet 2419392 wikitext text/x-wiki 4pdqhagp4h6np1stqssb3221kncf4ln 2419393 2419392 2026-05-26T14:05:02Z Stashbot 7414 marostegui@cumin1003: Removing pc1013 from zarcillo T427190 2419393 wikitext text/x-wiki l4n0w2ajgsbsiywmhslqwwq2u9p3wga 2419394 2419393 2026-05-26T14:05:07Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) 2419394 wikitext text/x-wiki b9zbv728np1vo75j0lmzyejjixk17he 2419395 2419394 2026-05-26T14:07:04Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1293710|Site info should output thumblimits as array (T427066)]] 2419395 wikitext text/x-wiki d1pq5ulmggdbzi6y3aa6jnhv9hsun0q 2419397 2419395 2026-05-26T14:07:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T426633)', diff saved to https://phabricator.wikimedia.org/P93016 and previous config saved to /var/cache/conftool/dbconfig/20260526-140718-fceratto.json 2419397 wikitext text/x-wiki 4g787qr35qe5usjxjcdbj8xoe6177yq 2419398 2419397 2026-05-26T14:07:43Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance 2419398 wikitext text/x-wiki qzh740g4pekj15i7vznxye5gsqii59v 2419399 2419398 2026-05-26T14:07:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2156 (T426633)', diff saved to https://phabricator.wikimedia.org/P93017 and previous config saved to /var/cache/conftool/dbconfig/20260526-140748-fceratto.json 2419399 wikitext text/x-wiki in6681erv1bjf46ncgc0uh7gsteb3bn 2419400 2419399 2026-05-26T14:08:20Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet 2419400 wikitext text/x-wiki e6m2dw8g4gakdpf5d37r0zoxuzx386o 2419401 2419400 2026-05-26T14:08:24Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet 2419401 wikitext text/x-wiki tbans03l4fro3bi1zszky61jgpmt60r 2419402 2419401 2026-05-26T14:08:58Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1293710|Site info should output thumblimits as array (T427066)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419402 wikitext text/x-wiki 280xbvakh58p7vjevvmj0fdg0v7fxnn 2419403 2419402 2026-05-26T14:09:26Z Stashbot 7414 fabfur: restoring lvs2011 as primary (T426199) 2419403 wikitext text/x-wiki 7p5hwcqpvzi64o3dn64apapnfz2ns2u 2419404 2419403 2026-05-26T14:09:35Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2419404 wikitext text/x-wiki 8fvsg2owrjjkg2zs7o88ojdzed32edj 2419406 2419404 2026-05-26T14:10:31Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for lvs2011.codfw.wmnet 2419406 wikitext text/x-wiki 25ai5ubvdwqzdpp7pd0zqx8nxljzdbb 2419407 2419406 2026-05-26T14:10:36Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2011.codfw.wmnet 2419407 wikitext text/x-wiki t42l3z7gnbw7cvsex9byszo5xuzug9p 2419408 2419407 2026-05-26T14:10:38Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage 2419408 wikitext text/x-wiki hlycjpro7h1zud3haat7cfijgxbqtym 2419409 2419408 2026-05-26T14:12:29Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2419409 wikitext text/x-wiki 8qdv1mm0tyrlnus5xuiqo1wsvdl94n9 2419410 2419409 2026-05-26T14:13:44Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293710|Site info should output thumblimits as array (T427066)]] (duration: 06m 40s) 2419410 wikitext text/x-wiki kb2albi8n5yv6kgrrxvuxh8ktvkw3ks 2419411 2419410 2026-05-26T14:14:23Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* 2419411 wikitext text/x-wiki 07xxce4r2fogr2pwxod4c27chmqruii 2419412 2419411 2026-05-26T14:14:38Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage 2419412 wikitext text/x-wiki natoev8fj4u3npsh14k5mim204jify3 2419413 2419412 2026-05-26T14:14:47Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2223: switch maintenance 2419413 wikitext text/x-wiki 0eoo8v6yrh8yxuulu1el2mkwfozt5cv 2419414 2419413 2026-05-26T14:14:52Z Stashbot 7414 fabfur: repooled cp2043 (T426199) 2419414 wikitext text/x-wiki hrxzpuo34zrkuo1y0o7cnn1a1su0iyp 2419415 2419414 2026-05-26T14:16:06Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2419415 wikitext text/x-wiki bj5w0tvat816v7q21seuip73jp81ka2 2419416 2419415 2026-05-26T14:16:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T426633)', diff saved to https://phabricator.wikimedia.org/P93021 and previous config saved to /var/cache/conftool/dbconfig/20260526-141628-fceratto.json 2419416 wikitext text/x-wiki cdscinr7ar3388gp3xy3uvervegmg62 2419417 2419416 2026-05-26T14:18:38Z Stashbot 7414 jynus: restarting mediabackups@codfw after maintenance on a codfw backup media storage server T426199 2419417 wikitext text/x-wiki 9fqhewp852zkcl2bxhtawe5v053k2kw 2419418 2419417 2026-05-26T14:19:10Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backup2015.codfw.wmnet,db2197.codfw.wmnet 2419418 wikitext text/x-wiki 6c9t3gzp56xsl731wdm4lk1xcyp206r 2419419 2419418 2026-05-26T14:19:13Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backup2015.codfw.wmnet,db2197.codfw.wmnet 2419419 wikitext text/x-wiki odvgd5zs6qua1c3vkspo7631nhnvof9 2419420 2419419 2026-05-26T14:19:46Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet 2419420 wikitext text/x-wiki 8cqoew9gf7vkzfyyrb95sz1umjxv2qa 2419422 2419420 2026-05-26T14:24:51Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1014: Rack maintenance completed 2419422 wikitext text/x-wiki l1scckpwa4cv2lyoytbmp79yrt16912 2419423 2419422 2026-05-26T14:24:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2419423 wikitext text/x-wiki qob1l733liwoig8ol4idbnu1pm2f4h4 2419424 2419423 2026-05-26T14:24:57Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) 2419424 wikitext text/x-wiki 129aft8qwu5efemo3w3tywoin39zv1w 2419425 2419424 2026-05-26T14:24:59Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1014: Rack maintenance completed 2419425 wikitext text/x-wiki 6dx5nvf1fcpqd0j9xjb85pfqcuv7fp7 2419426 2419425 2026-05-26T14:25:47Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2419426 wikitext text/x-wiki i6k110b6dy4ijtuwg7qz8iqkshpn7wb 2419427 2419426 2026-05-26T14:26:02Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2419427 wikitext text/x-wiki b84g6jkt57yaiyinghs4mvqtujiangz 2419428 2419427 2026-05-26T14:26:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93023 and previous config saved to /var/cache/conftool/dbconfig/20260526-142636-fceratto.json 2419428 wikitext text/x-wiki 0evsukniylvb9y4g4m3co89adzys84c 2419430 2419428 2026-05-26T14:31:25Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1054.eqiad.wmnet with OS trixie 2419430 wikitext text/x-wiki 7rohfeejzisg2c6mff4wnprwfvzcacv 2419432 2419430 2026-05-26T14:36:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93026 and previous config saved to /var/cache/conftool/dbconfig/20260526-143643-fceratto.json 2419432 wikitext text/x-wiki r35bsykku7n9ixmvev0t1peof8hcdgm 2419433 2419432 2026-05-26T14:40:32Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Migration of db2167.codfw.wmnet completed 2419433 wikitext text/x-wiki 0ijgw7oznyk7p7f4vbjilp9mpv54zbe 2419434 2419433 2026-05-26T14:40:34Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2419434 wikitext text/x-wiki 1a6y2b4kz3775nsp4eaolew5erioz84 2419435 2419434 2026-05-26T14:43:33Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw 2419435 wikitext text/x-wiki 26u4dxtsjn7r2gfexh6vttxxq6nl0dh 2419436 2419435 2026-05-26T14:45:24Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=codfw 2419436 wikitext text/x-wiki kcr83pvjrr264texng0f9w1iheb05co 2419437 2419436 2026-05-26T14:45:35Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=codfw 2419437 wikitext text/x-wiki skm26rhpifxsyvibtkrj2e29a7x7xq2 2419440 2419437 2026-05-26T14:46:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T426633)', diff saved to https://phabricator.wikimedia.org/P93029 and previous config saved to /var/cache/conftool/dbconfig/20260526-144651-fceratto.json 2419440 wikitext text/x-wiki gx6nonkz62tg0enco9rv51md2oa1oab 2419441 2419440 2026-05-26T14:47:15Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance 2419441 wikitext text/x-wiki fttpo9ave6jswooa6k6cg4j67eelo5k 2419442 2419441 2026-05-26T14:47:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2177 (T426633)', diff saved to https://phabricator.wikimedia.org/P93030 and previous config saved to /var/cache/conftool/dbconfig/20260526-144718-fceratto.json 2419442 wikitext text/x-wiki f0wyz3vpwnm5nxfw4g9otfs06o5x4wd 2419443 2419442 2026-05-26T14:47:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A 2419443 wikitext text/x-wiki gcr1lip4y5xent2wij268q3na5p2417 2419444 2419443 2026-05-26T14:49:24Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2029.codfw.wmnet to cluster codfw and group A 2419444 wikitext text/x-wiki 44le7mhmqmhtjucb5q4on8zmxmtlx70 2419445 2419444 2026-05-26T14:49:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A 2419445 wikitext text/x-wiki cuczl42vo8ibtnew6ljepki8hn9rfve 2419446 2419445 2026-05-26T14:49:49Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2419446 wikitext text/x-wiki rnndastkc9okjzt5tk698tnaocm6o1e 2419447 2419446 2026-05-26T14:49:52Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2221: Rack maintenance completed 2419447 wikitext text/x-wiki bt4b6zi1x9riz2ldsa2q05vifhhq7sm 2419448 2419447 2026-05-26T14:49:55Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2419448 wikitext text/x-wiki 43k35ecvppicd6n39v6cm2qh5481jae 2419449 2419448 2026-05-26T14:51:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2222: Rack maintenance completed 2419449 wikitext text/x-wiki 4in8xju6k2dal72ye0yqaig8g37owi2 2419450 2419449 2026-05-26T14:51:12Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2030.codfw.wmnet to cluster codfw and group A 2419450 wikitext text/x-wiki f8ox0ohsuuetbxc365s22n7un0j5s6n 2419451 2419450 2026-05-26T14:52:05Z Stashbot 7414 moritzm: remove ganeti1025 from eqiad Ganeti cluster T424680 2419451 wikitext text/x-wiki dczstchoypjn8m9ofk2ksbmrwur3jv5 2419452 2419451 2026-05-26T14:52:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet 2419452 wikitext text/x-wiki 24mthooi31p8icrhbwq6xj5pc1m8u0p 2419453 2419452 2026-05-26T14:54:21Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet 2419453 wikitext text/x-wiki t1pv2hpl7dz1j5hdoovv3kjpuv2o2be 2419454 2419453 2026-05-26T14:55:16Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet 2419454 wikitext text/x-wiki 2cb6669gguct1kzr41my9rkoll88n9h 2419455 2419454 2026-05-26T14:55:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T426633)', diff saved to https://phabricator.wikimedia.org/P93033 and previous config saved to /var/cache/conftool/dbconfig/20260526-145538-fceratto.json 2419455 wikitext text/x-wiki nj3ijgfxie5rx8u94yk1aeaft5ec043 2419456 2419455 2026-05-26T14:55:45Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2222.codfw.wmnet 2419456 wikitext text/x-wiki fixo00gayull2idoqkh1e2jw2a75aws 2419457 2419456 2026-05-26T14:55:48Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2222.codfw.wmnet 2419457 wikitext text/x-wiki 10h4ki549s1q064wzg9gnkaxd0ytggh 2419458 2419457 2026-05-26T14:55:51Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2221.codfw.wmnet 2419458 wikitext text/x-wiki oqovbo80hq9jhr7ptzxz5n74al0jthf 2419459 2419458 2026-05-26T14:55:54Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2221.codfw.wmnet 2419459 wikitext text/x-wiki fd0as7bgorn1arifp5ndjodo80ph36i 2419460 2419459 2026-05-26T14:56:53Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Rack maintenance completed 2419460 wikitext text/x-wiki cbeia59l9ssj8oo3vifcu1y6alq5y3m 2419461 2419460 2026-05-26T15:00:18Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: switch maintenance 2419461 wikitext text/x-wiki hbjpnvg02667ojelmd4odhsijl8u80r 2419462 2419461 2026-05-26T15:01:53Z Stashbot 7414 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy 2419462 wikitext text/x-wiki 2c6vzlm2if4nitvjfn2ae6u6u9gtgls 2419463 2419462 2026-05-26T15:01:56Z Stashbot 7414 bjensen: uploading prometheus-memcached-exporter_0.16.0-1_amd64 on apt1002 2419463 wikitext text/x-wiki g3b12u5264hqvg3ys75vo4kvm0u4lf5 2419464 2419463 2026-05-26T15:02:21Z Stashbot 7414 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator deploy 2419464 wikitext text/x-wiki qiqzsk3wmgiz2mvnjlc66nd12fn8o2h 2419465 2419464 2026-05-26T15:02:32Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2002 for T427286 2419465 wikitext text/x-wiki clczmslxdwenwah1u635qke07cpa9ko 2419467 2419465 2026-05-26T15:03:17Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2002 for T427286 (duration: 00m 45s) 2419467 wikitext text/x-wiki hxuojvuid21m283zqt5g7oysz3gfzl2 2419468 2419467 2026-05-26T15:03:41Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab1004 for T427286 2419468 wikitext text/x-wiki 3b2jzatmkxwqbeu0tty361vgibrf5se 2419470 2419468 2026-05-26T15:04:23Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab1004 for T427286 (duration: 00m 39s) 2419470 wikitext text/x-wiki bu097jxiqmt6bns4nls1p76r637xmee 2419471 2419470 2026-05-26T15:05:01Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Rack maintenance completed 2419471 wikitext text/x-wiki hsja7y2lf700vhbmvnhlc13dugscahh 2419473 2419471 2026-05-26T15:05:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93037 and previous config saved to /var/cache/conftool/dbconfig/20260526-150546-fceratto.json 2419473 wikitext text/x-wiki 2avtfnpa4nromp8914ro5gchre8lhle 2419474 2419473 2026-05-26T15:06:26Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Rack maintenance completed 2419474 wikitext text/x-wiki 0x7fz35grhq0onbsakpg4o0hy2odjpz 2419475 2419474 2026-05-26T15:07:57Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw 2419475 wikitext text/x-wiki tuis3nkb0zu8167ia6lnzrsnitjn5gc 2419477 2419475 2026-05-26T15:10:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2196.codfw.wmnet 2419477 wikitext text/x-wiki cf0c4427r59bhc8ghcmm6vk3ampb120 2419478 2419477 2026-05-26T15:10:57Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2196.codfw.wmnet 2419478 wikitext text/x-wiki 19xsoozbhid30tuqdd7uet7ea4tylpx 2419479 2419478 2026-05-26T15:12:09Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Rack maintenance completed 2419479 wikitext text/x-wiki kkgrycloqh89lyyuojo5sqhjkl5pq9t 2419481 2419479 2026-05-26T15:15:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93040 and previous config saved to /var/cache/conftool/dbconfig/20260526-151552-fceratto.json 2419481 wikitext text/x-wiki nm7k2w5hu5gv2vuchsppkyw1n3m0wyb 2419482 2419481 2026-05-26T15:22:11Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419482 wikitext text/x-wiki n2odrz2j08ju967jm6u54w0d511jfvy 2419483 2419482 2026-05-26T15:22:40Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419483 wikitext text/x-wiki sjz70r11ljm0iofwkblv9l6lvixlxap 2419484 2419483 2026-05-26T15:22:43Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419484 wikitext text/x-wiki t34b2mf48cubqzu2bds8j95wx2biu7r 2419485 2419484 2026-05-26T15:23:10Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419485 wikitext text/x-wiki me04uu1z9vzza6n6wdyudbb0xdjlrvr 2419486 2419485 2026-05-26T15:24:08Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419486 wikitext text/x-wiki reys2y6hbucd72actgyxvwup0f0pzmz 2419487 2419486 2026-05-26T15:24:10Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419487 wikitext text/x-wiki pgmde22lea2qim0nc2hpoqu4334t71q 2419488 2419487 2026-05-26T15:24:13Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419488 wikitext text/x-wiki 4w2ez8fwjk8crrjg7tws7dqp7tlvyt6 2419489 2419488 2026-05-26T15:24:15Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419489 wikitext text/x-wiki n5yesbvjh68i9s3vw2nzwm2ervvdtg7 2419491 2419489 2026-05-26T15:26:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T426633)', diff saved to https://phabricator.wikimedia.org/P93041 and previous config saved to /var/cache/conftool/dbconfig/20260526-152559-fceratto.json 2419491 wikitext text/x-wiki 9tapkx5lt1uw8zvrzbpbu5umsj1qz6f 2419492 2419491 2026-05-26T15:26:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance 2419492 wikitext text/x-wiki d1m14pyj00v9kedr7klwdkdhh2npj2a 2419493 2419492 2026-05-26T15:26:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2190 (T426633)', diff saved to https://phabricator.wikimedia.org/P93042 and previous config saved to /var/cache/conftool/dbconfig/20260526-152629-fceratto.json 2419493 wikitext text/x-wiki s16tve3254fria7pdvlzlwkqj5s0pwf 2419494 2419493 2026-05-26T15:28:06Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419494 wikitext text/x-wiki j59j4o6n73vd9gz3ktqugtq1ahaanmn 2419495 2419494 2026-05-26T15:28:09Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419495 wikitext text/x-wiki cgv70cbs9435k4lzh5gmkpnuosj68zw 2419496 2419495 2026-05-26T15:28:12Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419496 wikitext text/x-wiki 207bji1xnytyk94uzg5n0k1vw5wtcdk 2419497 2419496 2026-05-26T15:28:15Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419497 wikitext text/x-wiki i43r4wwdbssi87szsjxmfd4v0qinym8 2419498 2419497 2026-05-26T15:29:12Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419498 wikitext text/x-wiki qpsy3cmws2enoj6qjmt33zqvi98nee9 2419499 2419498 2026-05-26T15:29:15Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419499 wikitext text/x-wiki 0jfxh4bzv9i4y994mbje0i3ly4xvnbv 2419500 2419499 2026-05-26T15:29:17Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419500 wikitext text/x-wiki 56r07mgxjjumcyfpso5jppldw62ghob 2419501 2419500 2026-05-26T15:29:19Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419501 wikitext text/x-wiki 7pqvhr1pvf5pyqgejlcavsm9t3nhzrs 2419502 2419501 2026-05-26T15:30:15Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419502 wikitext text/x-wiki jpn2sayk3yvt8ngw2rne6pvznnkttah 2419503 2419502 2026-05-26T15:30:17Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419503 wikitext text/x-wiki 4yx33d9b0tucdlp7anp5vsu08ejrda7 2419504 2419503 2026-05-26T15:30:20Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419504 wikitext text/x-wiki h64887h5ijkaxcy8v0c9k34q7p1u8d1 2419505 2419504 2026-05-26T15:30:22Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419505 wikitext text/x-wiki 9copuj5z1dnsjs1qe0p04srbyza1r0b 2419508 2419505 2026-05-26T15:34:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T426633)', diff saved to https://phabricator.wikimedia.org/P93043 and previous config saved to /var/cache/conftool/dbconfig/20260526-153357-fceratto.json 2419508 wikitext text/x-wiki rotpa1m80feyea2jt79gogzezmpbt2h 2419515 2419508 2026-05-26T15:44:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93044 and previous config saved to /var/cache/conftool/dbconfig/20260526-154405-fceratto.json 2419515 wikitext text/x-wiki n8ztdo5lvis5tgfnw1wjc915320by65 2419516 2419515 2026-05-26T15:46:30Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad 2419516 wikitext text/x-wiki 4lrrow0ldz3u86hoha88krmqe73qpem 2419518 2419516 2026-05-26T15:54:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93045 and previous config saved to /var/cache/conftool/dbconfig/20260526-155413-fceratto.json 2419518 wikitext text/x-wiki 8f6r4omm2wx099jsbhoiybfkjt5nck5 2419519 2419518 2026-05-26T15:55:25Z Stashbot 7414 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - T423727 2419519 wikitext text/x-wiki gor22tgpt27d60s4s2nylp81n80r9u6 2419520 2419519 2026-05-26T15:55:46Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419520 wikitext text/x-wiki bsjd21itip2p3gc2dzlvim80pay4qkt 2419521 2419520 2026-05-26T15:55:50Z Stashbot 7414 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - T423727 (duration: 00m 22s) 2419521 wikitext text/x-wiki tbr5qv3uv2mf3pipj87ap0prba3izs3 2419522 2419521 2026-05-26T15:55:52Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419522 wikitext text/x-wiki lwq0qw09453amgctgtk67fs6ouc7kbq 2419523 2419522 2026-05-26T15:55:55Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419523 wikitext text/x-wiki a0m2tqsot4mik2jjto18te602bddfpp 2419524 2419523 2026-05-26T15:55:58Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419524 wikitext text/x-wiki aey0ys6pli7r4rev53xcg1uu7o3yg5y 2419526 2419524 2026-05-26T15:57:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419526 wikitext text/x-wiki ow2sfy39sczrjgmn0rd5ycc7jeb97cd 2419527 2419526 2026-05-26T15:57:24Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419527 wikitext text/x-wiki b0sczpz3ys4mi5kulzqnthtrzfbzypo 2419528 2419527 2026-05-26T15:57:27Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419528 wikitext text/x-wiki 05poqd1gd5yn46eqlcs64rt1lhma4kf 2419529 2419528 2026-05-26T15:57:29Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419529 wikitext text/x-wiki s4uss2v18fccikj1hf4grv0h1l40jzt 2419534 2419529 2026-05-26T16:00:14Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419534 wikitext text/x-wiki jxydjx3byc7l26faw8krddiv9c00c9i 2419535 2419534 2026-05-26T16:00:17Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419535 wikitext text/x-wiki slfsmtlyshzuucdem459f7ek0s69lh2 2419536 2419535 2026-05-26T16:00:19Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419536 wikitext text/x-wiki 2v7mecsdpdb3im2r8691wgdgbxuxrr8 2419537 2419536 2026-05-26T16:00:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419537 wikitext text/x-wiki 62fv44swc0nyk66whl2pz1x9j7u6p6e 2419540 2419537 2026-05-26T16:02:53Z Stashbot 7414 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - T423727 2419540 wikitext text/x-wiki 64t3lonegm1uy3mclx2cmi5zoniixyx 2419542 2419540 2026-05-26T16:03:21Z Stashbot 7414 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - T423727 (duration: 00m 28s) 2419542 wikitext text/x-wiki 11mr5xwkmo7qw8wb2uaseoxh57ncpla 2419543 2419542 2026-05-26T16:03:53Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419543 wikitext text/x-wiki fhas2ohxu7msgg19m73rmpboq1q9t71 2419544 2419543 2026-05-26T16:03:55Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419544 wikitext text/x-wiki 758x4c9paqm1gtl9w0earjrdmd107ii 2419545 2419544 2026-05-26T16:03:58Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419545 wikitext text/x-wiki kpk51ywgz8wjxw8ubxmyjteuxccvkao 2419546 2419545 2026-05-26T16:04:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419546 wikitext text/x-wiki nxazcd88nt613djonavaencllm11x3l 2419547 2419546 2026-05-26T16:04:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T426633)', diff saved to https://phabricator.wikimedia.org/P93046 and previous config saved to /var/cache/conftool/dbconfig/20260526-160420-fceratto.json 2419547 wikitext text/x-wiki 0uh31tfuvemkd5psiutl94pvz4jogqe 2419548 2419547 2026-05-26T16:04:45Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance 2419548 wikitext text/x-wiki ois68clcpbsyd770nk4g7n09r70tcwq 2419549 2419548 2026-05-26T16:04:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2194 (T426633)', diff saved to https://phabricator.wikimedia.org/P93047 and previous config saved to /var/cache/conftool/dbconfig/20260526-160450-fceratto.json 2419549 wikitext text/x-wiki b1s4t8tha24cjlatrb7ha5z8hzghw04 2419551 2419549 2026-05-26T16:06:18Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419551 wikitext text/x-wiki cin45k0x7g3savevnmytr3k9vjq437n 2419552 2419551 2026-05-26T16:06:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419552 wikitext text/x-wiki 2cnmtr6aa3mdgdv1qyw9c8ddwaivp5m 2419553 2419552 2026-05-26T16:06:23Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419553 wikitext text/x-wiki nd9jg90lqdx88tenl3jrt74kp1hc6s7 2419554 2419553 2026-05-26T16:06:26Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419554 wikitext text/x-wiki 4p6k7zgexu2g0cphxii5h697rz5kfvw 2419556 2419554 2026-05-26T16:07:44Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad 2419556 wikitext text/x-wiki ans5w69y6wa19a08v0qecjxegbpedra 2419558 2419556 2026-05-26T16:10:57Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419558 wikitext text/x-wiki gx2zvipe2mwnu3p6y1kehwu03mkp4h4 2419559 2419558 2026-05-26T16:11:00Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419559 wikitext text/x-wiki boggt3et04o9ubmfpdqgkpx2itlwshe 2419560 2419559 2026-05-26T16:11:02Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419560 wikitext text/x-wiki riucqs3zy9ja7my23sbvkab2pq9jso9 2419561 2419560 2026-05-26T16:11:05Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419561 wikitext text/x-wiki o4kqu8lmcmp236951xu343k6ygjciw6 2419562 2419561 2026-05-26T16:13:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T426633)', diff saved to https://phabricator.wikimedia.org/P93049 and previous config saved to /var/cache/conftool/dbconfig/20260526-161328-fceratto.json 2419562 wikitext text/x-wiki k09yvxwtc6hefwe3dhbetggwxd5j8hu 2419563 2419562 2026-05-26T16:13:41Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet 2419563 wikitext text/x-wiki 7q09vat3o49hs4pjfn01x8kufd8s840 2419564 2419563 2026-05-26T16:23:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93050 and previous config saved to /var/cache/conftool/dbconfig/20260526-162336-fceratto.json 2419564 wikitext text/x-wiki tcm8yd9lg5dd5ypmj0bln49ylnafij8 2419565 2419564 2026-05-26T16:30:49Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419565 wikitext text/x-wiki 2kjw7tmh733bnuc9s49hovs5a3qfg6b 2419566 2419565 2026-05-26T16:30:51Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419566 wikitext text/x-wiki 1x1wtqhkbe9ryinpeobvgzo34j3yc6j 2419569 2419566 2026-05-26T16:31:04Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419569 wikitext text/x-wiki 4kpp0v76k2qvt7c72sz2ctdgc6fkoyp 2419570 2419569 2026-05-26T16:31:07Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419570 wikitext text/x-wiki ogxr1i7wihcv3d4gn38uk4k3bsk65se 2419571 2419570 2026-05-26T16:33:43Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw and A:cp 2419571 wikitext text/x-wiki 6zzuydpj0ozggk4yfpcjbsqqru6gvay 2419572 2419571 2026-05-26T16:33:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93051 and previous config saved to /var/cache/conftool/dbconfig/20260526-163344-fceratto.json 2419572 wikitext text/x-wiki 5c1z7bev0kdlqgv500icsde9viil8sr 2419573 2419572 2026-05-26T16:34:00Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw and A:cp 2419573 wikitext text/x-wiki repsx9i3mgzgzxwwhk8ucboa1ubu5a8 2419574 2419573 2026-05-26T16:34:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419574 wikitext text/x-wiki kvh3iow3jr1613zyhcs2e9hbhd69o2u 2419575 2419574 2026-05-26T16:34:23Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419575 wikitext text/x-wiki 73t95iyd9e5xrmhrluhpb7ji9tskpvm 2419576 2419575 2026-05-26T16:34:25Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419576 wikitext text/x-wiki pzlgu9hcrl9od9bwmk6h4gwg36yfyiv 2419577 2419576 2026-05-26T16:34:28Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419577 wikitext text/x-wiki gct0q1pogtm35jb48oc3cu87bucxoga 2419578 2419577 2026-05-26T16:34:58Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2419578 wikitext text/x-wiki azvz96qzczm44j2awwayjosm97ixsp6 2419579 2419578 2026-05-26T16:35:20Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2419579 wikitext text/x-wiki hltgh5jj14m56amjoghvogjqt8nunpu 2419580 2419579 2026-05-26T16:36:01Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419580 wikitext text/x-wiki fgubh1jy329nvrkiegvg8bsg4e4zs1h 2419581 2419580 2026-05-26T16:36:04Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419581 wikitext text/x-wiki sao50aezuf8woyz1ki1etwfatfdi2rs 2419582 2419581 2026-05-26T16:36:06Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419582 wikitext text/x-wiki plajo1eshu9csqtbymusaw0fyr3cw4x 2419583 2419582 2026-05-26T16:36:08Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419583 wikitext text/x-wiki 72pz7837or1qvhf9fdcfp0xxnxp3hk7 2419584 2419583 2026-05-26T16:36:44Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2419584 wikitext text/x-wiki 228g6oovr0a2u7crq8ozxn6ppzhyspd 2419585 2419584 2026-05-26T16:37:01Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2419585 wikitext text/x-wiki jkobqn2w2zrogi0y4fpe9k3yeg1b6ca 2419586 2419585 2026-05-26T16:37:16Z Stashbot 7414 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2419586 wikitext text/x-wiki tmzisdjsngb5nvwiloy48oqnb4xhirs 2419587 2419586 2026-05-26T16:37:32Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419587 wikitext text/x-wiki 83uu44h3lgmdbk7pg9i7tf6ez3yyg0a 2419588 2419587 2026-05-26T16:37:35Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419588 wikitext text/x-wiki o2nypp35clemhzkirsx5n4tj3scp87r 2419589 2419588 2026-05-26T16:37:37Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419589 wikitext text/x-wiki ky97edwklv19afgxffnj1yx0qr9f495 2419590 2419589 2026-05-26T16:37:40Z Stashbot 7414 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2419590 wikitext text/x-wiki dvstxus5cway82fobimk9gd5my71559 2419591 2419590 2026-05-26T16:37:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419591 wikitext text/x-wiki ib8w6h92vq9n07gowuaxjow3p9hdy77 2419592 2419591 2026-05-26T16:39:13Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419592 wikitext text/x-wiki 8xs43cunxfm925qe0k4lb9kd0gy6cfh 2419593 2419592 2026-05-26T16:39:16Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419593 wikitext text/x-wiki bvr1f3ruccdtf857qeqtdy33myt99xc 2419594 2419593 2026-05-26T16:39:18Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419594 wikitext text/x-wiki bkxoarb31o88j0tvfziaaew9d9260cl 2419595 2419594 2026-05-26T16:39:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419595 wikitext text/x-wiki tua85btdlbrzcw60xdnogumhunfh2rs 2419596 2419595 2026-05-26T16:40:16Z Stashbot 7414 brett: reboot lvs 101[345].eqiad.wmnet 2419596 wikitext text/x-wiki nazrqngjl9w5a5zt9mj7hrww9brgkiu 2419597 2419596 2026-05-26T16:40:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419597 wikitext text/x-wiki fdfqtgq9vbupo4127rsfar7s79hhljl 2419598 2419597 2026-05-26T16:40:23Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419598 wikitext text/x-wiki 5b93bwv35kna37xjnplzr8hq3jwos2s 2419599 2419598 2026-05-26T16:40:26Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2419599 wikitext text/x-wiki 5k80sx3z4ls5h5vy5h2lq7xtxi66n7c 2419600 2419599 2026-05-26T16:40:29Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419600 wikitext text/x-wiki 7qxc9axft7glkhugahi8mk6pach49wm 2419601 2419600 2026-05-26T16:40:31Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419601 wikitext text/x-wiki 92bq8o6w47gwp278q7thmu16xgwfhez 2419602 2419601 2026-05-26T16:41:22Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet 2419602 wikitext text/x-wiki 0bz5486stxajyu3jcdkuoxzisgv4xd7 2419603 2419602 2026-05-26T16:42:43Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet 2419603 wikitext text/x-wiki qhlqqh86kxhlrbdi0xwe7pcgwdqotse 2419604 2419603 2026-05-26T16:43:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T426633)', diff saved to https://phabricator.wikimedia.org/P93052 and previous config saved to /var/cache/conftool/dbconfig/20260526-164352-fceratto.json 2419604 wikitext text/x-wiki 8bq0wfctklid4hxhxfi9nmc8x49mfe9 2419605 2419604 2026-05-26T16:44:14Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" 2419605 wikitext text/x-wiki f0jovup7u3c4lm16fwr47ipyrimmxqr 2419606 2419605 2026-05-26T16:44:16Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance 2419606 wikitext text/x-wiki fial5xoicx0muypfzicp44kppi99jfc 2419607 2419606 2026-05-26T16:44:19Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" 2419607 wikitext text/x-wiki ij9ts1gv04p4tnnyhuk00e9x0fg9nua 2419608 2419607 2026-05-26T16:44:22Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419608 wikitext text/x-wiki 0u11nycbyffv4ryyc4xsmz6khdttj67 2419609 2419608 2026-05-26T16:44:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 (T426633)', diff saved to https://phabricator.wikimedia.org/P93053 and previous config saved to /var/cache/conftool/dbconfig/20260526-164421-fceratto.json 2419609 wikitext text/x-wiki 3r767nqxos1xx614ybdthc9p4sd4zle 2419610 2419609 2026-05-26T16:44:58Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419610 wikitext text/x-wiki szmf4oklonoe603gqrz2515hwqr5jv8 2419611 2419610 2026-05-26T16:45:01Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419611 wikitext text/x-wiki mhn9smevua2h2cuprk9chwjc90jv6mw 2419612 2419611 2026-05-26T16:45:03Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419612 wikitext text/x-wiki 85hbnjf9yad579dk1eryrnbzvvddpgp 2419613 2419612 2026-05-26T16:45:05Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419613 wikitext text/x-wiki 0jub19mf3uyy5h50nj1nrawnzkgtyna 2419614 2419613 2026-05-26T16:45:44Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419614 wikitext text/x-wiki 679ztqd5vojmvtdeutyrz8g344rc3lg 2419615 2419614 2026-05-26T16:45:47Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419615 wikitext text/x-wiki 833lbgii429fxf1h671l23eqk61ucfk 2419616 2419615 2026-05-26T16:50:49Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419616 wikitext text/x-wiki fsaah3ully510eekoy07482yb5xb9g0 2419617 2419616 2026-05-26T16:50:51Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419617 wikitext text/x-wiki iat4ki8srvbuyjzgxui63428dsbohhl 2419618 2419617 2026-05-26T16:50:54Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419618 wikitext text/x-wiki ltcbgi7n6fjmj0ydeizs2g3y9atlf05 2419619 2419618 2026-05-26T16:50:56Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419619 wikitext text/x-wiki 45ezd7914ir6lvfwk7mpfe6brh95hzz 2419620 2419619 2026-05-26T16:52:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T426633)', diff saved to https://phabricator.wikimedia.org/P93054 and previous config saved to /var/cache/conftool/dbconfig/20260526-165240-fceratto.json 2419620 wikitext text/x-wiki dnoprwatq3qnsnjcq5i5el54r03q9sk 2419621 2419620 2026-05-26T16:52:45Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419621 wikitext text/x-wiki hpb0nehpmjc3lvdlbqlghgypapbu08h 2419622 2419621 2026-05-26T16:55:06Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419622 wikitext text/x-wiki 9lub59nsjr0oj6m90zqyd8seuhqam5x 2419623 2419622 2026-05-26T16:57:52Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419623 wikitext text/x-wiki 18jl1kaa3pjznbqsr5m2ku0kf9lqo7g 2419624 2419623 2026-05-26T17:00:05Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419624 wikitext text/x-wiki gmzi9y33s877w1fbs7n8ys6sest30tx 2419625 2419624 2026-05-26T17:00:08Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419625 wikitext text/x-wiki mem4n71zvxw3xcaguzpl0yt8qduoexz 2419626 2419625 2026-05-26T17:00:10Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419626 wikitext text/x-wiki 8vy7drosvk9diy1z174th9l40s3yzuy 2419627 2419626 2026-05-26T17:00:13Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419627 wikitext text/x-wiki ccvpf25tx4zkncbqxcjgmvh9ljblpqx 2419628 2419627 2026-05-26T17:02:46Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419628 wikitext text/x-wiki 346xfc7040drq18c6x23yhgnqzr4kur 2419629 2419628 2026-05-26T17:02:48Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419629 wikitext text/x-wiki h4rme8319wcnb79o172qwljzve4nj60 2419630 2419629 2026-05-26T17:02:51Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419630 wikitext text/x-wiki drv6parco7tztj2ral9zy72rbynywir 2419631 2419630 2026-05-26T17:02:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93055 and previous config saved to /var/cache/conftool/dbconfig/20260526-170247-fceratto.json 2419631 wikitext text/x-wiki nl98kcgceju2r7m8trbqsq3t7kb80t3 2419632 2419631 2026-05-26T17:02:56Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419632 wikitext text/x-wiki tdzuntz7u9ikpues4c1apzehfqbxf76 2419633 2419632 2026-05-26T17:05:16Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419633 wikitext text/x-wiki f684k61ofdc658oyzdycrtje7d8d45m 2419634 2419633 2026-05-26T17:05:18Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419634 wikitext text/x-wiki attxfjwk0eoyfhcd0q1vgo9olts40uh 2419635 2419634 2026-05-26T17:05:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419635 wikitext text/x-wiki s3n4sc2djbh7yby7el0ex2v0q69dehh 2419636 2419635 2026-05-26T17:05:23Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419636 wikitext text/x-wiki ds1f891t450rdxdz5rzpeahxqhwgdjs 2419638 2419636 2026-05-26T17:07:51Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419638 wikitext text/x-wiki izuzf63kdw39qy76vtz3519gct9cgsd 2419639 2419638 2026-05-26T17:09:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419639 wikitext text/x-wiki rnigz1y743ks9em18o0n20fxd6st9vs 2419640 2419639 2026-05-26T17:09:44Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419640 wikitext text/x-wiki j2xdidog5pgyykbmj7r273zk0oxpitk 2419641 2419640 2026-05-26T17:09:47Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419641 wikitext text/x-wiki fvwfnvai851f6jq0604xz3pghaainnm 2419642 2419641 2026-05-26T17:09:49Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419642 wikitext text/x-wiki pvwi9h62s3h4xz4m6lfqgeyv6kg72ew 2419643 2419642 2026-05-26T17:11:14Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419643 wikitext text/x-wiki 3hd9i7cbnzz43cme5hua4lxarchgmee 2419644 2419643 2026-05-26T17:11:17Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419644 wikitext text/x-wiki ms9utk5c02t0o0sdzlq3pl5xgkqzzj9 2419645 2419644 2026-05-26T17:11:20Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419645 wikitext text/x-wiki 6prindrb8s9n7oskgy5u6pa5ykdw1ug 2419646 2419645 2026-05-26T17:11:23Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419646 wikitext text/x-wiki n83qtr7ilwps61a83o823h3ap573hi8 2419647 2419646 2026-05-26T17:12:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93056 and previous config saved to /var/cache/conftool/dbconfig/20260526-171255-fceratto.json 2419647 wikitext text/x-wiki t87fo4kvtw2dhzle5h0iyn8284brtc5 2419648 2419647 2026-05-26T17:13:08Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419648 wikitext text/x-wiki gtazergs1zp9a0n3jqpt1hyow3mh6gb 2419649 2419648 2026-05-26T17:13:11Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419649 wikitext text/x-wiki 7mcwlb45n17kc9rf1gzhrpzv7vxxpyk 2419650 2419649 2026-05-26T17:13:13Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419650 wikitext text/x-wiki o2jnm3eg1f8gkoczjlmipauj0uudosd 2419651 2419650 2026-05-26T17:13:16Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419651 wikitext text/x-wiki kwrss7l55ls7v4g9iypeehsfdd335ev 2419652 2419651 2026-05-26T17:13:30Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419652 wikitext text/x-wiki e18pft0u45utfdlozfikattdimdcctw 2419653 2419652 2026-05-26T17:14:33Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419653 wikitext text/x-wiki 62cijwr6yhtfkg8wlr8s8jwj0hcdfwr 2419654 2419653 2026-05-26T17:14:35Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419654 wikitext text/x-wiki dqxtg18x49bv9gvumv83ygkdidolmli 2419655 2419654 2026-05-26T17:14:38Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419655 wikitext text/x-wiki 64j95bbc1a3eu1gq27bc3c0f6waok09 2419656 2419655 2026-05-26T17:14:40Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419656 wikitext text/x-wiki 3tejjk95b1d3rpvge6ibqwexaprbwjx 2419657 2419656 2026-05-26T17:15:16Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: T426585 - bking@cumin2002 2419657 wikitext text/x-wiki 8vl0bvdcz8ejnmhduxz1agic8r18k65 2419658 2419657 2026-05-26T17:16:44Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419658 wikitext text/x-wiki 6b7kew7z1kozmv6onvzan6ys21t3an5 2419659 2419658 2026-05-26T17:16:47Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419659 wikitext text/x-wiki 1f9fbf5i86krvhdjbbrstppl9es65nu 2419660 2419659 2026-05-26T17:16:50Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419660 wikitext text/x-wiki r83x4717bnjqz4vttji49ia7ipl5ql1 2419661 2419660 2026-05-26T17:16:52Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419661 wikitext text/x-wiki i4gur5s0p6k9fuszths7l1j3guvq49a 2419662 2419661 2026-05-26T17:17:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419662 wikitext text/x-wiki inr3k3mmc43s4swx2zyj3m93wr1dx9h 2419663 2419662 2026-05-26T17:17:03Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419663 wikitext text/x-wiki o9yfrt4eq3yfcupcgxo2bezdyt5xmsv 2419664 2419663 2026-05-26T17:17:06Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419664 wikitext text/x-wiki nrcjje03vlpz90z8hdpa4hftnhuh830 2419665 2419664 2026-05-26T17:17:08Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419665 wikitext text/x-wiki 61gcpxei5snwg8x7j6czgrw5is0wtwo 2419666 2419665 2026-05-26T17:17:23Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419666 wikitext text/x-wiki azfjrxmnorwcavaqghe84u6grh2zrfb 2419667 2419666 2026-05-26T17:17:26Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419667 wikitext text/x-wiki 99yfugw9kmt0wbjv8a3t22wjv2czw3f 2419668 2419667 2026-05-26T17:17:28Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419668 wikitext text/x-wiki j7ws2fmdvwoqb2jhjl0y6dzj9y808qs 2419669 2419668 2026-05-26T17:17:31Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419669 wikitext text/x-wiki nlh0vpa74pk8zq5c2h2nw4row3bvjau 2419670 2419669 2026-05-26T17:18:51Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419670 wikitext text/x-wiki 44clr3bxz0k75pv62n3dk2ew0ntk7uo 2419671 2419670 2026-05-26T17:20:19Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet 2419671 wikitext text/x-wiki o9rkmvm2zgxq2nw9asvqbpakoekid4k 2419672 2419671 2026-05-26T17:21:00Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2419672 wikitext text/x-wiki ons939d4yth2vpinzv6p3q0ghpzo5zc 2419673 2419672 2026-05-26T17:21:44Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet 2419673 wikitext text/x-wiki 6jp3iobe54kjcbhp5pav5zx4mm60od4 2419674 2419673 2026-05-26T17:23:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T426633)', diff saved to https://phabricator.wikimedia.org/P93057 and previous config saved to /var/cache/conftool/dbconfig/20260526-172303-fceratto.json 2419674 wikitext text/x-wiki oauugpg4r78mxix09inyoil1swnuunj 2419675 2419674 2026-05-26T17:23:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance 2419675 wikitext text/x-wiki fqm58shhd5t60m1pnib5v7p7jih5ghd 2419676 2419675 2026-05-26T17:23:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2227 (T426633)', diff saved to https://phabricator.wikimedia.org/P93058 and previous config saved to /var/cache/conftool/dbconfig/20260526-172332-fceratto.json 2419676 wikitext text/x-wiki 2xzsuumz2fnruqtdbztaiwphi18h8j9 2419677 2419676 2026-05-26T17:24:52Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" 2419677 wikitext text/x-wiki rrh31g8g5xc4p85x9oullq26dzoczk1 2419678 2419677 2026-05-26T17:24:56Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419678 wikitext text/x-wiki jwyauwaynf4p46qsx2fnx0k9c2pv1mn 2419679 2419678 2026-05-26T17:24:58Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" 2419679 wikitext text/x-wiki gxhv7k6hfbq9x9rzaz1hjs76hh14ghz 2419680 2419679 2026-05-26T17:25:01Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419680 wikitext text/x-wiki ek2al4z2fdzye9lmec1xnj2foylefno 2419681 2419680 2026-05-26T17:25:03Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419681 wikitext text/x-wiki trogw8z76xrg7clyui1jc8tsiskqu1d 2419682 2419681 2026-05-26T17:25:05Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419682 wikitext text/x-wiki q7h9lubcwawa758kairmmdljxxnazy1 2419683 2419682 2026-05-26T17:25:08Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419683 wikitext text/x-wiki 3d6s50hzisw25txxhyjv9u8sxsemb99 2419684 2419683 2026-05-26T17:26:09Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419684 wikitext text/x-wiki 97rtevmv6dwee43bezyjidc4n5kxz2p 2419685 2419684 2026-05-26T17:27:16Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2419685 wikitext text/x-wiki ip1gxf0tilaea8xjobmkf2kfyqnfupa 2419687 2419685 2026-05-26T17:31:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T426633)', diff saved to https://phabricator.wikimedia.org/P93059 and previous config saved to /var/cache/conftool/dbconfig/20260526-173109-fceratto.json 2419687 wikitext text/x-wiki 4gaysfl903h24gufn52aeep0ypss989 2419692 2419687 2026-05-26T17:32:29Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419692 wikitext text/x-wiki bz6b0t18r4jy5ex315zixacdvirjc6s 2419693 2419692 2026-05-26T17:32:32Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419693 wikitext text/x-wiki mva333smrhtny0lkd5sx1yjxsoq8zg0 2419694 2419693 2026-05-26T17:32:35Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419694 wikitext text/x-wiki lpq1v8w0vf4jqszusnpq3m29hdfo314 2419695 2419694 2026-05-26T17:32:38Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419695 wikitext text/x-wiki ahz2stj50umkbltzspff6zetf582djb 2419696 2419695 2026-05-26T17:33:22Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419696 wikitext text/x-wiki c7pecjgbzgdi3s3a0gfrykmt38o3u1b 2419697 2419696 2026-05-26T17:33:25Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419697 wikitext text/x-wiki de2uq2qhx8ojt677ygckptfruzi3i2k 2419698 2419697 2026-05-26T17:33:28Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419698 wikitext text/x-wiki 8toz7bbpp63owxkojdcjzw7x18pr7f9 2419699 2419698 2026-05-26T17:33:30Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419699 wikitext text/x-wiki lr1ms1ac56mk9gox1hqw8hl2c018doe 2419700 2419699 2026-05-26T17:34:49Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293779|Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] 2419700 wikitext text/x-wiki dfsp3ytd6vglwr0tdq44l5z6lu064j4 2419701 2419700 2026-05-26T17:36:31Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419701 wikitext text/x-wiki nlm9o3ejz6ezr8e1v8tkthuv529m4gq 2419702 2419701 2026-05-26T17:36:50Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1293779|Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419702 wikitext text/x-wiki 9l49kp3qvvlej4lj3smgbq570fescm3 2419703 2419702 2026-05-26T17:36:56Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419703 wikitext text/x-wiki mf78h1sfn3ylwhkrrcxupj7xdxjeqjb 2419704 2419703 2026-05-26T17:36:59Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419704 wikitext text/x-wiki 4w7ocduxdnekpfe44s2v4tk26f6yz7k 2419705 2419704 2026-05-26T17:37:24Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419705 wikitext text/x-wiki a65iq7zo51yfwkmdh89k0je06iu03u5 2419706 2419705 2026-05-26T17:37:58Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2419706 wikitext text/x-wiki lfai5gfofyax5urab642arkuraykfs6 2419707 2419706 2026-05-26T17:39:31Z Stashbot 7414 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be2089.codfw.wmnet 2419707 wikitext text/x-wiki qxw53t4slg618tpa0d8rouqh5afv6eo 2419708 2419707 2026-05-26T17:41:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93060 and previous config saved to /var/cache/conftool/dbconfig/20260526-174117-fceratto.json 2419708 wikitext text/x-wiki byhtxqleatmevm9cl5bvr55i1th6po7 2419709 2419708 2026-05-26T17:42:14Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293779|Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] (duration: 07m 25s) 2419709 wikitext text/x-wiki iw0mr0wlk232xiqa7g19orpooou0bk5 2419712 2419709 2026-05-26T17:51:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93062 and previous config saved to /var/cache/conftool/dbconfig/20260526-175124-fceratto.json 2419712 wikitext text/x-wiki 2ao0s9hwxayqnyflczw3f7t44q5gh1h 2419713 2419712 2026-05-26T17:54:17Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419713 wikitext text/x-wiki rn7qoye52bxm6ag83w3eozozz66lhwv 2419714 2419713 2026-05-26T17:54:20Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419714 wikitext text/x-wiki p7bvi1gdbfqdme0nexs8nleeg1nyr5c 2419715 2419714 2026-05-26T17:54:23Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419715 wikitext text/x-wiki thli7t2krijwrvc40rm39i5mz6tsqrt 2419716 2419715 2026-05-26T17:54:25Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419716 wikitext text/x-wiki bd2v6ojxv3q2ma6y7dh908ogn3yalvy 2419718 2419716 2026-05-26T17:59:05Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet 2419718 wikitext text/x-wiki 5hyisjb29ba1ic78cljtlsm7ja81l1g 2419719 2419718 2026-05-26T18:00:45Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet 2419719 wikitext text/x-wiki pampdi4678x5n97gydstxg9m27cns1z 2419720 2419719 2026-05-26T18:01:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T426633)', diff saved to https://phabricator.wikimedia.org/P93063 and previous config saved to /var/cache/conftool/dbconfig/20260526-180132-fceratto.json 2419720 wikitext text/x-wiki 4hasigz4vmzt5l9gicmz4cg6gn6c2vp 2419721 2419720 2026-05-26T18:02:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance 2419721 wikitext text/x-wiki 01cbpyao20gpatpx4fwa5nw089r8exg 2419722 2419721 2026-05-26T18:02:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2159 (T426633)', diff saved to https://phabricator.wikimedia.org/P93064 and previous config saved to /var/cache/conftool/dbconfig/20260526-180205-fceratto.json 2419722 wikitext text/x-wiki hmamp0orxmkcp6p6jyoyui30j885l5n 2419726 2419722 2026-05-26T18:09:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T426633)', diff saved to https://phabricator.wikimedia.org/P93065 and previous config saved to /var/cache/conftool/dbconfig/20260526-180915-fceratto.json 2419726 wikitext text/x-wiki 6spfg645k6ndj2re06w9se174hg15cn 2419730 2419726 2026-05-26T18:15:12Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419730 wikitext text/x-wiki 0p3sxmml8h7mzssrfeoeflb7phqslb4 2419731 2419730 2026-05-26T18:15:16Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419731 wikitext text/x-wiki f09azqyt41r4zeurwdkglx2c8wvymx9 2419732 2419731 2026-05-26T18:15:19Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419732 wikitext text/x-wiki 9wf1vbtfu5vyx24dnbisqrqf44c68lo 2419733 2419732 2026-05-26T18:15:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419733 wikitext text/x-wiki iyzdy935tagd9rrc55ko1inro235y9s 2419737 2419733 2026-05-26T18:19:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93066 and previous config saved to /var/cache/conftool/dbconfig/20260526-181923-fceratto.json 2419737 wikitext text/x-wiki am4mtjoytf8cm1u4cb8wq8ee8bx5z7y 2419739 2419737 2026-05-26T18:21:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419739 wikitext text/x-wiki k81ifw8ar2fqzusohrxmbljmhxr81ml 2419740 2419739 2026-05-26T18:21:02Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419740 wikitext text/x-wiki kauvw9kjbbgr39ockj8n1udoqsqcjj1 2419741 2419740 2026-05-26T18:21:05Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419741 wikitext text/x-wiki j9y96xdop1p7aazaa7cuud96ztkysys 2419742 2419741 2026-05-26T18:21:08Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419742 wikitext text/x-wiki 5x1jfyl75mhuneasmucw9hqsrxgrjsx 2419747 2419742 2026-05-26T18:24:36Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.dns.netbox 2419747 wikitext text/x-wiki 07abyohbqxp47opihox7qooaa75rwnj 2419752 2419747 2026-05-26T18:29:08Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" 2419752 wikitext text/x-wiki 1vbgcgms6ue59qfdcl95bxwhlyvp9qj 2419753 2419752 2026-05-26T18:29:14Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" 2419753 wikitext text/x-wiki mged7yvkw8v018aew5ig7ka3ql25u0k 2419754 2419753 2026-05-26T18:29:17Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419754 wikitext text/x-wiki tj60uzaw4geh1kxm1zpntvlayrsbm61 2419755 2419754 2026-05-26T18:29:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93068 and previous config saved to /var/cache/conftool/dbconfig/20260526-182931-fceratto.json 2419755 wikitext text/x-wiki ptxka6s05b93g101sqaklpm6p03tktm 2419756 2419755 2026-05-26T18:30:32Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: T426585 - bking@cumin2002 2419756 wikitext text/x-wiki h66hqy384vwj4m4ctqllbq6ygh7t4ss 2419760 2419756 2026-05-26T18:37:49Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet 2419760 wikitext text/x-wiki bd72ya5enetm8yj1pla7i1ou1vbkd3f 2419761 2419760 2026-05-26T18:39:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T426633)', diff saved to https://phabricator.wikimedia.org/P93069 and previous config saved to /var/cache/conftool/dbconfig/20260526-183939-fceratto.json 2419761 wikitext text/x-wiki 92bwxcjgur6w5v95fr5z6dio6uoyp6l 2419762 2419761 2026-05-26T18:40:04Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance 2419762 wikitext text/x-wiki sf1g961lwknsnx7re9n4cfhsg0wsucg 2419763 2419762 2026-05-26T18:40:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2168 (T426633)', diff saved to https://phabricator.wikimedia.org/P93070 and previous config saved to /var/cache/conftool/dbconfig/20260526-184009-fceratto.json 2419763 wikitext text/x-wiki 8adu9f05cg2lm2ebg1p30zr7kcvuqc5 2419764 2419763 2026-05-26T18:41:32Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet 2419764 wikitext text/x-wiki t4nwfix89tri7oy0ej7koorfggn9qk4 2419765 2419764 2026-05-26T18:41:35Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb2014.codfw.wmnet with OS trixie 2419765 wikitext text/x-wiki pbr8rj4wyq6vewbqdmudbgqgvq2vx4o 2419766 2419765 2026-05-26T18:43:33Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie 2419766 wikitext text/x-wiki d47gv0ibb0ez1qdskl25w30s6t6g248 2419767 2419766 2026-05-26T18:44:06Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie 2419767 wikitext text/x-wiki csooqyvjfvx1ymtsk8x42g7hmrbkst9 2419769 2419767 2026-05-26T18:47:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T426633)', diff saved to https://phabricator.wikimedia.org/P93071 and previous config saved to /var/cache/conftool/dbconfig/20260526-184724-fceratto.json 2419769 wikitext text/x-wiki d01jzefpv6r9dvu7kqi7q6ts30lto9d 2419772 2419769 2026-05-26T18:56:21Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage 2419772 wikitext text/x-wiki jtz8t8cv3mfmfbflgc8gpxci4fz4157 2419773 2419772 2026-05-26T18:56:35Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage 2419773 wikitext text/x-wiki hb5iw6h2sr1zhf1ejowm7zv97ta3vmx 2419774 2419773 2026-05-26T18:57:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93072 and previous config saved to /var/cache/conftool/dbconfig/20260526-185732-fceratto.json 2419774 wikitext text/x-wiki q8t4hugf7br6rwz60y3mw7y1try0ou3 2419775 2419774 2026-05-26T18:59:09Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet 2419775 wikitext text/x-wiki jjq0ma7rrbqx49frl82r1ta73iglyxg 2419776 2419775 2026-05-26T19:03:32Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage 2419776 wikitext text/x-wiki b5y4sst3abkiwa3ambuccgntit34o0v 2419777 2419776 2026-05-26T19:07:39Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage 2419777 wikitext text/x-wiki f4ca5us0pdal7w1zylzqldxeyvl1iq5 2419778 2419777 2026-05-26T19:07:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93073 and previous config saved to /var/cache/conftool/dbconfig/20260526-190740-fceratto.json 2419778 wikitext text/x-wiki 7elw4frdbcttm1u06xen1wsoe0wohkf 2419783 2419778 2026-05-26T19:16:36Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet 2419783 wikitext text/x-wiki e49e7s64sf6p0o18nglsiq2m2kkiyqp 2419784 2419783 2026-05-26T19:17:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T426633)', diff saved to https://phabricator.wikimedia.org/P93074 and previous config saved to /var/cache/conftool/dbconfig/20260526-191748-fceratto.json 2419784 wikitext text/x-wiki 1hlu340ci4k5pqg8w2q3om2ogy2un8y 2419785 2419784 2026-05-26T19:18:12Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance 2419785 wikitext text/x-wiki i0yvyxmd8o3gsv7xyukf9myb5ns02ta 2419786 2419785 2026-05-26T19:18:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2182 (T426633)', diff saved to https://phabricator.wikimedia.org/P93075 and previous config saved to /var/cache/conftool/dbconfig/20260526-191818-fceratto.json 2419786 wikitext text/x-wiki 2dy3m28lzgavu43sq59svkhzq46j61y 2419787 2419786 2026-05-26T19:19:21Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie 2419787 wikitext text/x-wiki 86z1pz837m1pv59vd2uwpzclsb68907 2419788 2419787 2026-05-26T19:19:50Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2419788 wikitext text/x-wiki mskfb5pcob0k7d7mlyv2wm6nsm97t27 2419789 2419788 2026-05-26T19:20:33Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet 2419789 wikitext text/x-wiki 22670nffhrq3kfy0bb38fiuwl756wfx 2419790 2419789 2026-05-26T19:21:25Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.dns.netbox 2419790 wikitext text/x-wiki dlr0ft71f7xsknm97f9ayljxrx8xv16 2419791 2419790 2026-05-26T19:24:58Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2419791 wikitext text/x-wiki m4u7nhr2k4q8iajh2qhw28vqmdny6dz 2419792 2419791 2026-05-26T19:25:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T426633)', diff saved to https://phabricator.wikimedia.org/P93076 and previous config saved to /var/cache/conftool/dbconfig/20260526-192533-fceratto.json 2419792 wikitext text/x-wiki imj9cdtkbjuzsawkvznnme3xw9ux53a 2419793 2419792 2026-05-26T19:30:10Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" 2419793 wikitext text/x-wiki tcmcg2xctgu7si97ugo6rfqcgdgprzy 2419794 2419793 2026-05-26T19:35:40Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" 2419794 wikitext text/x-wiki qddmq61txa2wome8h8scn4il9ztrekg 2419795 2419794 2026-05-26T19:35:43Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419795 wikitext text/x-wiki jyabsezg0hq4gl2bcdd5diggj5xsz8b 2419796 2419795 2026-05-26T19:35:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93077 and previous config saved to /var/cache/conftool/dbconfig/20260526-193541-fceratto.json 2419796 wikitext text/x-wiki k4qhhtc8f8h81k2e3on47un4n30a197 2419797 2419796 2026-05-26T19:38:19Z Stashbot 7414 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie 2419797 wikitext text/x-wiki 5b002qx82hzf1zswfwck8ospetcbrmx 2419798 2419797 2026-05-26T19:39:22Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie 2419798 wikitext text/x-wiki 2dbgtjow6pyss289rrqrxbd19rwrehi 2419799 2419798 2026-05-26T19:40:46Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2419799 wikitext text/x-wiki cl7ney7v6hwztk0qn5wgwatp4ct9erm 2419800 2419799 2026-05-26T19:40:49Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2013.codfw.wmnet with OS trixie 2419800 wikitext text/x-wiki 3856hrfbq99vw4xlhkzgvgqkxtgswp0 2419801 2419800 2026-05-26T19:40:53Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2419801 wikitext text/x-wiki 3rbdln1krdg5bz6w6rbu3ejcfpxbnqf 2419802 2419801 2026-05-26T19:40:55Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2014.codfw.wmnet with OS trixie 2419802 wikitext text/x-wiki htl2c99yb9mfwdc8lq3t2fgvyzmyj9p 2419803 2419802 2026-05-26T19:43:02Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 2419803 wikitext text/x-wiki nx372s0nqcwbglv2ff0gm4jplio2v5x 2419804 2419803 2026-05-26T19:43:05Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 2419804 wikitext text/x-wiki czgqzxp1v9ot55fpl77lbrzbedn82un 2419805 2419804 2026-05-26T19:43:11Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 2419805 wikitext text/x-wiki jjwkcu6usub0o158exhnm0lr7rbgsao 2419806 2419805 2026-05-26T19:43:14Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 2419806 wikitext text/x-wiki 362cvchkmpubssn99wzjaibk22bjytq 2419807 2419806 2026-05-26T19:43:50Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2419807 wikitext text/x-wiki if0s8kb3ajqktnjk5m1lywlzmwzzblh 2419808 2419807 2026-05-26T19:44:14Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2419808 wikitext text/x-wiki 2fc3yju72ajiupcfizpx07icnrjevum 2419809 2419808 2026-05-26T19:45:25Z Stashbot 7414 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie 2419809 wikitext text/x-wiki 71ibya0a9wp7uwj7occykrrvsru1tgm 2419810 2419809 2026-05-26T19:45:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93078 and previous config saved to /var/cache/conftool/dbconfig/20260526-194549-fceratto.json 2419810 wikitext text/x-wiki nfgi3wttj7ayvqb6erclarzshz9q7fz 2419811 2419810 2026-05-26T19:51:18Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2419811 wikitext text/x-wiki c9v7s2n6pwr5m0b57g5djud5taw2nhj 2419812 2419811 2026-05-26T19:51:31Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2419812 wikitext text/x-wiki ixv4pap7mbqgz4i20ao2weh940dq1dr 2419813 2419812 2026-05-26T19:55:24Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet 2419813 wikitext text/x-wiki 2xkvyiotmj7m934964keevdsxc6w0es 2419814 2419813 2026-05-26T19:55:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T426633)', diff saved to https://phabricator.wikimedia.org/P93079 and previous config saved to /var/cache/conftool/dbconfig/20260526-195557-fceratto.json 2419814 wikitext text/x-wiki kgtkad9ncnfoos2onnwshxec8g6hkke 2419815 2419814 2026-05-26T19:56:26Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance 2419815 wikitext text/x-wiki jex008r75wi3zmylg4j5wovqpn7asln 2419816 2419815 2026-05-26T19:56:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2208 (T426633)', diff saved to https://phabricator.wikimedia.org/P93080 and previous config saved to /var/cache/conftool/dbconfig/20260526-195632-fceratto.json 2419816 wikitext text/x-wiki g2xwirc0tzq4tizi1l07i7glqlimggt 2419817 2419816 2026-05-26T19:57:37Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2028.codfw.wmnet with OS trixie 2419817 wikitext text/x-wiki 32f1pgfgvv6apsxw6s4vjm9dvw0bavq 2419818 2419817 2026-05-26T19:58:42Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2029.codfw.wmnet with OS trixie 2419818 wikitext text/x-wiki opf2pxnwqqz3grcnn0md7n1rfqvdrhv 2419819 2419818 2026-05-26T19:59:20Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet 2419819 wikitext text/x-wiki 95lx7l2040lc5fzz1qdtyyocz5ad43r 2419820 2419819 2026-05-26T20:03:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T426633)', diff saved to https://phabricator.wikimedia.org/P93081 and previous config saved to /var/cache/conftool/dbconfig/20260526-200333-fceratto.json 2419820 wikitext text/x-wiki l7f2ovrm2148f51eqaejvgmxn5fr136 2419821 2419820 2026-05-26T20:09:24Z Stashbot 7414 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1293161|Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794|Re-enable ReadingLists survey on beta cluster (T426781)]] 2419821 wikitext text/x-wiki q116vvtlmnb7xuanwbxmgqf3ajet5pw 2419822 2419821 2026-05-26T20:11:19Z Stashbot 7414 alexsanford@deploy1003: alexsanford, aude: Backport for [[gerrit:1293161|Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794|Re-enable ReadingLists survey on beta cluster (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419822 wikitext text/x-wiki 92n3g6u7x2lp2qdw9rktmbpw82ldbfh 2419823 2419822 2026-05-26T20:13:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93082 and previous config saved to /var/cache/conftool/dbconfig/20260526-201341-fceratto.json 2419823 wikitext text/x-wiki a0j5nv4b0wu2b4l3dfyjzhkhxvawpz1 2419824 2419823 2026-05-26T20:14:24Z Stashbot 7414 alexsanford@deploy1003: alexsanford, aude: Continuing with deployment 2419824 wikitext text/x-wiki dq9mz4jmfgjr3or471835y07h8cy1xa 2419825 2419824 2026-05-26T20:18:38Z Stashbot 7414 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293161|Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794|Re-enable ReadingLists survey on beta cluster (T426781)]] (duration: 09m 14s) 2419825 wikitext text/x-wiki lz9zgt4122b58volsb5eaealj4tc1be 2419826 2419825 2026-05-26T20:23:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93083 and previous config saved to /var/cache/conftool/dbconfig/20260526-202349-fceratto.json 2419826 wikitext text/x-wiki iojp21aj0bkjg2v6aflomqpe5higgjq 2419827 2419826 2026-05-26T20:32:00Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2419827 wikitext text/x-wiki qezs4tua5kk2gnhbt60lckwn148e83s 2419828 2419827 2026-05-26T20:32:26Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2419828 wikitext text/x-wiki 1r0afu01y8vgaaaw4uce3k6dbd0637i 2419829 2419828 2026-05-26T20:32:28Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2419829 wikitext text/x-wiki pngbjjp26u71i69zs8wr1e4dgqkehtt 2419830 2419829 2026-05-26T20:32:53Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2419830 wikitext text/x-wiki bkakvr33zql4mtrytg4i5ltfts0mec8 2419831 2419830 2026-05-26T20:34:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T426633)', diff saved to https://phabricator.wikimedia.org/P93084 and previous config saved to /var/cache/conftool/dbconfig/20260526-203357-fceratto.json 2419831 wikitext text/x-wiki 3reahp48azpsdfv09mulg1x9zen85gc 2419832 2419831 2026-05-26T20:34:23Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet 2419832 wikitext text/x-wiki f1eu7lzzhnsgp3xryljkkmlucwaswtr 2419833 2419832 2026-05-26T20:34:26Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance 2419833 wikitext text/x-wiki 4t6isu855649hxa3ooeyxy6j4ga8qeh 2419834 2419833 2026-05-26T20:34:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2221 (T426633)', diff saved to https://phabricator.wikimedia.org/P93085 and previous config saved to /var/cache/conftool/dbconfig/20260526-203430-fceratto.json 2419834 wikitext text/x-wiki chmw1cpd4t0uhffod9snddliuhxa3nj 2419835 2419834 2026-05-26T20:38:11Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet 2419835 wikitext text/x-wiki g2njpr4vhh2w17fl2mkskxpsbp5zfzd 2419836 2419835 2026-05-26T20:41:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T426633)', diff saved to https://phabricator.wikimedia.org/P93086 and previous config saved to /var/cache/conftool/dbconfig/20260526-204143-fceratto.json 2419836 wikitext text/x-wiki s7b06muk1yujbkpkzyt8z9919f254f2 2419837 2419836 2026-05-26T20:45:33Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.dns.netbox 2419837 wikitext text/x-wiki hafzlvc19pdnd3ee31y04vmo9xcle6t 2419838 2419837 2026-05-26T20:50:13Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" 2419838 wikitext text/x-wiki 5f75dxzia8pr3if3dg0vbwatqb468fe 2419839 2419838 2026-05-26T20:50:19Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" 2419839 wikitext text/x-wiki 6mvu92cr8gkyqv1rwd0bzcvzjw7to9o 2419840 2419839 2026-05-26T20:50:22Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419840 wikitext text/x-wiki hj1ntcg32icrv7mfuwrggc42bdwba4g 2419841 2419840 2026-05-26T20:51:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93087 and previous config saved to /var/cache/conftool/dbconfig/20260526-205152-fceratto.json 2419841 wikitext text/x-wiki 8rs7ujjz3sncznzvds6vtie0fseranh 2419842 2419841 2026-05-26T20:55:40Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab2003.codfw.wmnet with reason: WIP 2419842 wikitext text/x-wiki h7l11347sg0xi1dq7wiuff39sa5p85h 2419843 2419842 2026-05-26T21:02:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93088 and previous config saved to /var/cache/conftool/dbconfig/20260526-210159-fceratto.json 2419843 wikitext text/x-wiki mylu14sr3ehebfiv6cbkc631mrhn4nj 2419844 2419843 2026-05-26T21:06:33Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie 2419844 wikitext text/x-wiki 02yz6w93v2vqxwlpuk8g8x4u7ea6x4p 2419845 2419844 2026-05-26T21:12:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T426633)', diff saved to https://phabricator.wikimedia.org/P93089 and previous config saved to /var/cache/conftool/dbconfig/20260526-211207-fceratto.json 2419845 wikitext text/x-wiki lyfgwvm3u9bacg9xo9z7t6296g7qn1x 2419846 2419845 2026-05-26T21:12:32Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance 2419846 wikitext text/x-wiki 6jmjnmir62yvvdka6wpm9msp9q6gq1d 2419847 2419846 2026-05-26T21:12:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2222 (T426633)', diff saved to https://phabricator.wikimedia.org/P93090 and previous config saved to /var/cache/conftool/dbconfig/20260526-211238-fceratto.json 2419847 wikitext text/x-wiki l71ybd0u5j5gebzpiik1vba8dzzpyoz 2419848 2419847 2026-05-26T21:14:51Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet 2419848 wikitext text/x-wiki 7xg3pcq170wcaq3ci52tcqyyocre6jc 2419849 2419848 2026-05-26T21:14:54Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw and A:cp 2419849 wikitext text/x-wiki bff751bdhl2np4q9iiqr5ss8pknqnmj 2419850 2419849 2026-05-26T21:15:32Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet 2419850 wikitext text/x-wiki 69owscoz0htqw3to086cheoxbpmngot 2419851 2419850 2026-05-26T21:15:35Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw and A:cp 2419851 wikitext text/x-wiki pizwlq5ykfpifydj3zwtdjennoiazpm 2419852 2419851 2026-05-26T21:19:07Z Stashbot 7414 jhathaway: dmarc ingress test run mx-in1001 2419852 wikitext text/x-wiki 9a2lr7wbl3bppqjij7x0w6puw0r8rpe 2419853 2419852 2026-05-26T21:19:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T426633)', diff saved to https://phabricator.wikimedia.org/P93091 and previous config saved to /var/cache/conftool/dbconfig/20260526-211948-fceratto.json 2419853 wikitext text/x-wiki gwgidt83ux18s8wwb68xig9hbk50kxh 2419854 2419853 2026-05-26T21:27:48Z Stashbot 7414 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` in tmux session - T421688 2419854 wikitext text/x-wiki k5ovo766opleuxc3ps7srvg72n9k4fl 2419855 2419854 2026-05-26T21:29:07Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1008.eqiad.wmnet with OS trixie 2419855 wikitext text/x-wiki kzbpoi3vmbou239ltf8u2fr2xp0388b 2419856 2419855 2026-05-26T21:29:38Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1008 2419856 wikitext text/x-wiki chfh5g45vj9kuxi40heuo7m1d96uyi7 2419857 2419856 2026-05-26T21:29:50Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.netbox 2419857 wikitext text/x-wiki d4m2u28mp3ctmy4ou03vgvy9cew9j7m 2419858 2419857 2026-05-26T21:29:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93092 and previous config saved to /var/cache/conftool/dbconfig/20260526-212955-fceratto.json 2419858 wikitext text/x-wiki 6lt8w7rj9gxestzood6v3n68xdlw2ye 2419859 2419858 2026-05-26T21:30:41Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1009.eqiad.wmnet with OS trixie 2419859 wikitext text/x-wiki epnja9mwlljy1jce1ivgf0ak9h9a9db 2419860 2419859 2026-05-26T21:31:13Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1009 2419860 wikitext text/x-wiki 53lfmni8p7uqvx54ll4t4t02cg2wylk 2419861 2419860 2026-05-26T21:31:47Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1010.eqiad.wmnet with OS trixie 2419861 wikitext text/x-wiki 1ohpj27kdwgmr41u37aemop0r8xjaai 2419862 2419861 2026-05-26T21:32:11Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1010 2419862 wikitext text/x-wiki m08iivsug8jgr5fkl43r2he60brr9fi 2419863 2419862 2026-05-26T21:32:13Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1010 2419863 wikitext text/x-wiki o035793yr79ibrg75aqg7jb2vcrnylz 2419864 2419863 2026-05-26T21:35:14Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.netbox 2419864 wikitext text/x-wiki 2y3hh2jeqd85m5e5pduo83kee0b0ioh 2419865 2419864 2026-05-26T21:36:32Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" 2419865 wikitext text/x-wiki 9qqg8oksys5tut2txulxaec7a3ipga1 2419866 2419865 2026-05-26T21:36:38Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" 2419866 wikitext text/x-wiki h9z91yfc4f5n37xazv12hr1k4v6zatj 2419867 2419866 2026-05-26T21:36:41Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419867 wikitext text/x-wiki 7283favoxkjg94gii34xse18xvo8zvo 2419868 2419867 2026-05-26T21:36:43Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2419868 wikitext text/x-wiki 3a974mredipvt5csf9tbg8tdg42gao7 2419869 2419868 2026-05-26T21:36:46Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2419869 wikitext text/x-wiki c0r24hs2gg70h900n2qicp8ez8gpizi 2419870 2419869 2026-05-26T21:36:48Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1008 2419870 wikitext text/x-wiki qpk5svoyirouhc9favf5ak0920cvp69 2419871 2419870 2026-05-26T21:40:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93093 and previous config saved to /var/cache/conftool/dbconfig/20260526-214003-fceratto.json 2419871 wikitext text/x-wiki kosr8fifmvo2a52pm8hi04boazrlv2b 2419872 2419871 2026-05-26T21:41:00Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1008 2419872 wikitext text/x-wiki 68fci5wdyg2wc9jmg3oi6c0ok0cno0t 2419873 2419872 2026-05-26T21:41:03Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1008 2419873 wikitext text/x-wiki foxhgpt0devm95mhd3atcmr45q59xi2 2419874 2419873 2026-05-26T21:42:55Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" 2419874 wikitext text/x-wiki lqgj1buv2g5netuv1wydhvfeteegip8 2419875 2419874 2026-05-26T21:42:58Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage 2419875 wikitext text/x-wiki 7ln4wye2ct6jqkculb7l4yomlmf6y4a 2419876 2419875 2026-05-26T21:43:00Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" 2419876 wikitext text/x-wiki 4udxn170vsiz5fpanpjyxy442lmrm27 2419877 2419876 2026-05-26T21:43:03Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419877 wikitext text/x-wiki cfue5ppxv6hio51zeoa7j5zl0q5vatk 2419878 2419877 2026-05-26T21:43:05Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2419878 wikitext text/x-wiki 3e7ncugog4n3hqx1x878k9282fgajou 2419879 2419878 2026-05-26T21:43:08Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2419879 wikitext text/x-wiki m2upbdmvimn0xst5prhve39fw56vm3o 2419880 2419879 2026-05-26T21:43:11Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1009 2419880 wikitext text/x-wiki 6asvsvse03adhcc7kp2g5gvuwijc4i4 2419881 2419880 2026-05-26T21:44:26Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1009 2419881 wikitext text/x-wiki 18fpozoft4cyyg19yt43uakfu8odjlw 2419882 2419881 2026-05-26T21:44:29Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1009 2419882 wikitext text/x-wiki gu6g7omsek05tzzm69dakhxci88tpts 2419885 2419882 2026-05-26T21:47:29Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet 2419885 wikitext text/x-wiki bdxqjr8wuhfbw3vxkbeo9vgtmbwalv0 2419886 2419885 2026-05-26T21:49:23Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage 2419886 wikitext text/x-wiki lhhqh0axydvh9zj27eznld3774oldst 2419887 2419886 2026-05-26T21:50:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T426633)', diff saved to https://phabricator.wikimedia.org/P93094 and previous config saved to /var/cache/conftool/dbconfig/20260526-215011-fceratto.json 2419887 wikitext text/x-wiki 82dwz3hrboku3ox0abisjomr94o3eg8 2419888 2419887 2026-05-26T21:50:38Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance 2419888 wikitext text/x-wiki 38l3rv6mys5vfz00u54lzd1gkucsvij 2419889 2419888 2026-05-26T21:50:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2163 (T426633)', diff saved to https://phabricator.wikimedia.org/P93095 and previous config saved to /var/cache/conftool/dbconfig/20260526-215043-fceratto.json 2419889 wikitext text/x-wiki m8aqillguscgsf7tgvmv8cvmjdi61pw 2419890 2419889 2026-05-26T21:51:41Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage 2419890 wikitext text/x-wiki sww9zgznv40bwmnq16jvi5l7e80vpoh 2419892 2419890 2026-05-26T21:54:58Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage 2419892 wikitext text/x-wiki bz541budp5oe6sbaw8tfyxkk2tzlpu3 2419893 2419892 2026-05-26T21:55:15Z Stashbot 7414 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1293701|MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] 2419893 wikitext text/x-wiki 18r88kf7if36y22s112ptwgnlqbqqml 2419894 2419893 2026-05-26T21:56:09Z Stashbot 7414 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp6015.drmrs.wmnet 2419894 wikitext text/x-wiki oytmvx5z7hxkkfenpx1ux1soeo1ivbv 2419895 2419894 2026-05-26T21:56:21Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1010.eqiad.wmnet with OS trixie 2419895 wikitext text/x-wiki gqr6f3j5fp5zsscdhsldm4ef83ca4wj 2419896 2419895 2026-05-26T21:56:30Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet 2419896 wikitext text/x-wiki 2m7ponlzitgm1lcmjllzkbqfosf2mcp 2419897 2419896 2026-05-26T21:57:12Z Stashbot 7414 egardner@deploy1003: egardner, mfossati: Backport for [[gerrit:1293701|MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419897 wikitext text/x-wiki 9mmmpgy1lmpy0x3on40oj70bssa95kj 2419898 2419897 2026-05-26T21:58:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T426633)', diff saved to https://phabricator.wikimedia.org/P93096 and previous config saved to /var/cache/conftool/dbconfig/20260526-215803-fceratto.json 2419898 wikitext text/x-wiki s64j8kdv8pnd14uh2se5bfysj1tv3dh 2419899 2419898 2026-05-26T21:59:28Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage 2419899 wikitext text/x-wiki 579s83doj60wdjjxib4ek2ti3cxwme3 2419900 2419899 2026-05-26T22:00:25Z Stashbot 7414 egardner@deploy1003: egardner, mfossati: Continuing with deployment 2419900 wikitext text/x-wiki dlnwxqykj5ts232vcdm412kyaua8oxv 2419901 2419900 2026-05-26T22:03:01Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage 2419901 wikitext text/x-wiki 7uxad13frlsv5c66j001eakj809oii2 2419902 2419901 2026-05-26T22:04:46Z Stashbot 7414 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293701|MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] (duration: 09m 30s) 2419902 wikitext text/x-wiki 4qdeg9v3723lzrrunu9f88cyhmsb0r9 2419903 2419902 2026-05-26T22:08:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93097 and previous config saved to /var/cache/conftool/dbconfig/20260526-220811-fceratto.json 2419903 wikitext text/x-wiki sjn1kpsxe0ljcjqfq9pawjllfyux3dw 2419904 2419903 2026-05-26T22:08:24Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1008.eqiad.wmnet with OS trixie 2419904 wikitext text/x-wiki 368vlw78ypoqjihv77kpsvm2200bon3 2419905 2419904 2026-05-26T22:10:03Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1009.eqiad.wmnet with OS trixie 2419905 wikitext text/x-wiki fzp20bmaqteejhg73qrljwj611wfkty 2419906 2419905 2026-05-26T22:18:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93098 and previous config saved to /var/cache/conftool/dbconfig/20260526-221819-fceratto.json 2419906 wikitext text/x-wiki cdpeovwv97r6v3yly18zmiyy1vy7hsk 2419907 2419906 2026-05-26T22:23:50Z Stashbot 7414 robh@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp6015.drmrs.wmnet 2419907 wikitext text/x-wiki dxyr9emzonh5hgcadjbe4ksucrt3rw3 2419908 2419907 2026-05-26T22:28:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T426633)', diff saved to https://phabricator.wikimedia.org/P93099 and previous config saved to /var/cache/conftool/dbconfig/20260526-222828-fceratto.json 2419908 wikitext text/x-wiki hc85faty3v9go95gwfhx5v72u3hywtr 2419909 2419908 2026-05-26T22:28:42Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance 2419909 wikitext text/x-wiki 18qczjz5c4lp890gm4wfmyvlwtwpj4m 2419910 2419909 2026-05-26T22:28:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2164 (T426633)', diff saved to https://phabricator.wikimedia.org/P93100 and previous config saved to /var/cache/conftool/dbconfig/20260526-222848-fceratto.json 2419910 wikitext text/x-wiki mliywy8hthh4xqoovtqh3l5zizvsg9u 2419911 2419910 2026-05-26T22:35:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T426633)', diff saved to https://phabricator.wikimedia.org/P93101 and previous config saved to /var/cache/conftool/dbconfig/20260526-223556-fceratto.json 2419911 wikitext text/x-wiki brpdt8eo9ekd72094m2sqn8n5h9txiy 2419918 2419911 2026-05-26T22:46:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93103 and previous config saved to /var/cache/conftool/dbconfig/20260526-224604-fceratto.json 2419918 wikitext text/x-wiki o332dxil754ecs3ni7h8whtixs5273t 2419924 2419918 2026-05-26T22:56:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93104 and previous config saved to /var/cache/conftool/dbconfig/20260526-225612-fceratto.json 2419924 wikitext text/x-wiki pn0ofwfv7bk6304qg95mloolgxeixir 2419927 2419924 2026-05-26T23:06:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T426633)', diff saved to https://phabricator.wikimedia.org/P93105 and previous config saved to /var/cache/conftool/dbconfig/20260526-230620-fceratto.json 2419927 wikitext text/x-wiki 60rlqu63v14xt9ca4at43gxsbreu7c9 2419928 2419927 2026-05-26T23:06:44Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance 2419928 wikitext text/x-wiki 4muvdleyfe9ncn19t7hsy0hmefpop50 2419929 2419928 2026-05-26T23:06:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 (T426633)', diff saved to https://phabricator.wikimedia.org/P93106 and previous config saved to /var/cache/conftool/dbconfig/20260526-230650-fceratto.json 2419929 wikitext text/x-wiki 8488o6a8dzzn0vtg4clbnv58w3082bp 2419930 2419929 2026-05-26T23:07:48Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5026.* 2419930 wikitext text/x-wiki p081pegovkn55q7w0fukg5eijbks8tb 2419931 2419930 2026-05-26T23:14:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T426633)', diff saved to https://phabricator.wikimedia.org/P93107 and previous config saved to /var/cache/conftool/dbconfig/20260526-231358-fceratto.json 2419931 wikitext text/x-wiki 40r8w2g8zoh8tdlne1cdj3h5vpkq59o 2419932 2419931 2026-05-26T23:24:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93108 and previous config saved to /var/cache/conftool/dbconfig/20260526-232406-fceratto.json 2419932 wikitext text/x-wiki o4j4jkxl9y4hussbll1nmkdexxl5nez 2419933 2419932 2026-05-26T23:27:52Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5026.* 2419933 wikitext text/x-wiki orrafsvt9btcua6szp0vdgczwl454gq 2419934 2419933 2026-05-26T23:34:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93109 and previous config saved to /var/cache/conftool/dbconfig/20260526-233414-fceratto.json 2419934 wikitext text/x-wiki h09h3oce4x9w419cxqn8q5qtmw3m8w1 2419935 2419934 2026-05-26T23:44:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T426633)', diff saved to https://phabricator.wikimedia.org/P93110 and previous config saved to /var/cache/conftool/dbconfig/20260526-234421-fceratto.json 2419935 wikitext text/x-wiki p5ec45pzecqeaf6wcw2e2f5r9a8quwh 2419936 2419935 2026-05-26T23:44:45Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance 2419936 wikitext text/x-wiki qlmf3vtiqfmjbnxd5b4a156nvdug5h1 2419937 2419936 2026-05-26T23:44:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2166 (T426633)', diff saved to https://phabricator.wikimedia.org/P93111 and previous config saved to /var/cache/conftool/dbconfig/20260526-234451-fceratto.json 2419937 wikitext text/x-wiki mv95sa1tjbihl4mz2czrv7u00jwu72d 2419938 2419937 2026-05-26T23:52:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T426633)', diff saved to https://phabricator.wikimedia.org/P93112 and previous config saved to /var/cache/conftool/dbconfig/20260526-235201-fceratto.json 2419938 wikitext text/x-wiki efqlnryp40akool89xigfyq6j69py0c 2419939 2419938 2026-05-27T00:02:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93113 and previous config saved to /var/cache/conftool/dbconfig/20260527-000209-fceratto.json 2419939 wikitext text/x-wiki s4xn7dthwtvufeuul0gyfsh6xus1pve 2419941 2419939 2026-05-27T00:12:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93114 and previous config saved to /var/cache/conftool/dbconfig/20260527-001220-fceratto.json 2419941 wikitext text/x-wiki qqppnxh2myulsfnllup47rxahush2cc 2419942 2419941 2026-05-27T00:22:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T426633)', diff saved to https://phabricator.wikimedia.org/P93115 and previous config saved to /var/cache/conftool/dbconfig/20260527-002228-fceratto.json 2419942 wikitext text/x-wiki rp28k42qzdvvfm1s685by2skoup1fi9 2419943 2419942 2026-05-27T00:23:04Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance 2419943 wikitext text/x-wiki 06qula4bodiz6mk5l6hd3zspujt5wep 2419944 2419943 2026-05-27T00:23:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 (T426633)', diff saved to https://phabricator.wikimedia.org/P93116 and previous config saved to /var/cache/conftool/dbconfig/20260527-002309-fceratto.json 2419944 wikitext text/x-wiki fo5drobu0strh5urge0je8ukindwkqn 2419946 2419944 2026-05-27T00:31:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T426633)', diff saved to https://phabricator.wikimedia.org/P93117 and previous config saved to /var/cache/conftool/dbconfig/20260527-003141-fceratto.json 2419946 wikitext text/x-wiki 6uon8cbn62aq6km2bzyoib9zifyyk8p 2419947 2419946 2026-05-27T00:41:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93118 and previous config saved to /var/cache/conftool/dbconfig/20260527-004149-fceratto.json 2419947 wikitext text/x-wiki 3e5ym3gr1z4hjevt1vwp7zt9gyim9qn 2419948 2419947 2026-05-27T00:51:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93119 and previous config saved to /var/cache/conftool/dbconfig/20260527-005157-fceratto.json 2419948 wikitext text/x-wiki dmp1msxvhg1vgrl94ikb1xohqq6zjsj 2419949 2419948 2026-05-27T01:02:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T426633)', diff saved to https://phabricator.wikimedia.org/P93120 and previous config saved to /var/cache/conftool/dbconfig/20260527-010205-fceratto.json 2419949 wikitext text/x-wiki 9nrts6hz0svsn0bbd223v4but6inbdu 2419950 2419949 2026-05-27T01:02:29Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance 2419950 wikitext text/x-wiki qf76x1v3vg2pzufclscxiy99vxybzrp 2419951 2419950 2026-05-27T01:02:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 (T426633)', diff saved to https://phabricator.wikimedia.org/P93121 and previous config saved to /var/cache/conftool/dbconfig/20260527-010234-fceratto.json 2419951 wikitext text/x-wiki jx6o7g0l7lik05h6qa3wt4rvumbe7xp 2419952 2419951 2026-05-27T01:11:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T426633)', diff saved to https://phabricator.wikimedia.org/P93122 and previous config saved to /var/cache/conftool/dbconfig/20260527-011111-fceratto.json 2419952 wikitext text/x-wiki i9lohqjtp1vmqniftnhs5gxuyryrwxr 2419953 2419952 2026-05-27T01:21:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93123 and previous config saved to /var/cache/conftool/dbconfig/20260527-012119-fceratto.json 2419953 wikitext text/x-wiki guq4thmyornmrdu25fm3xbc73jous4d 2419954 2419953 2026-05-27T01:31:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93124 and previous config saved to /var/cache/conftool/dbconfig/20260527-013126-fceratto.json 2419954 wikitext text/x-wiki i8musjz3ah3k389ij44ncfpguesjerw 2419955 2419954 2026-05-27T01:41:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T426633)', diff saved to https://phabricator.wikimedia.org/P93125 and previous config saved to /var/cache/conftool/dbconfig/20260527-014134-fceratto.json 2419955 wikitext text/x-wiki e634xf2wm83shur4l2ywbrk4dq99yaq 2419956 2419955 2026-05-27T01:41:58Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance 2419956 wikitext text/x-wiki 7pmk1a4cg0jmo69qbey6x6aw2evoqzx 2419957 2419956 2026-05-27T01:42:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2179 (T426633)', diff saved to https://phabricator.wikimedia.org/P93126 and previous config saved to /var/cache/conftool/dbconfig/20260527-014204-fceratto.json 2419957 wikitext text/x-wiki 3v2ufxdjhp0tx78h4adw88zlqd2yfzj 2419958 2419957 2026-05-27T01:50:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T426633)', diff saved to https://phabricator.wikimedia.org/P93127 and previous config saved to /var/cache/conftool/dbconfig/20260527-015037-fceratto.json 2419958 wikitext text/x-wiki l0xifdjyau74c9l0sexh1rj7utdy5ff 2419959 2419958 2026-05-27T02:00:47Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2419959 wikitext text/x-wiki kgax6czfv0z87p8ukk9vgmnwujyto0a 2419960 2419959 2026-05-27T02:00:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93128 and previous config saved to /var/cache/conftool/dbconfig/20260527-020045-fceratto.json 2419960 wikitext text/x-wiki nnrojy1bwllhzfc1u9ut6cuhglh4oh7 2419961 2419960 2026-05-27T02:07:16Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) 2419961 wikitext text/x-wiki q9jxz0x5bmy51j8rmn4jhyzd847gn88 2419962 2419961 2026-05-27T02:10:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93129 and previous config saved to /var/cache/conftool/dbconfig/20260527-021053-fceratto.json 2419962 wikitext text/x-wiki 1xb5frn7m07obz56twf17xrvu6sy2lx 2419963 2419962 2026-05-27T02:21:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T426633)', diff saved to https://phabricator.wikimedia.org/P93130 and previous config saved to /var/cache/conftool/dbconfig/20260527-022100-fceratto.json 2419963 wikitext text/x-wiki 4s29b9e9jj6lp60hblj28avxo0wfb0g 2419964 2419963 2026-05-27T02:21:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance 2419964 wikitext text/x-wiki ga5o2my1jg1h2lg85xbqirqpuxayeus 2419965 2419964 2026-05-27T02:21:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 (T426633)', diff saved to https://phabricator.wikimedia.org/P93131 and previous config saved to /var/cache/conftool/dbconfig/20260527-022133-fceratto.json 2419965 wikitext text/x-wiki ri44de3x2zehl8samettu7jj09pe8n1 2419966 2419965 2026-05-27T02:29:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T426633)', diff saved to https://phabricator.wikimedia.org/P93132 and previous config saved to /var/cache/conftool/dbconfig/20260527-022953-fceratto.json 2419966 wikitext text/x-wiki nkiih9pkjh3lg8xtc4v12drc71pmy5v 2419967 2419966 2026-05-27T02:40:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93133 and previous config saved to /var/cache/conftool/dbconfig/20260527-024000-fceratto.json 2419967 wikitext text/x-wiki by9b1hl1jt4di8jrhhriq653kdxnwnt 2419968 2419967 2026-05-27T02:50:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93134 and previous config saved to /var/cache/conftool/dbconfig/20260527-025008-fceratto.json 2419968 wikitext text/x-wiki j8fpfiz3yx66xk2nayqolrajhoup90v 2419969 2419968 2026-05-27T03:00:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T426633)', diff saved to https://phabricator.wikimedia.org/P93135 and previous config saved to /var/cache/conftool/dbconfig/20260527-030016-fceratto.json 2419969 wikitext text/x-wiki 8vc3gzf3w4bafyneons3f6jg32kzg2r 2419970 2419969 2026-05-27T03:00:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance 2419970 wikitext text/x-wiki 1izsv7vwwh7u2q91fziprf0t0elpds1 2419971 2419970 2026-05-27T03:00:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2210 (T426633)', diff saved to https://phabricator.wikimedia.org/P93136 and previous config saved to /var/cache/conftool/dbconfig/20260527-030045-fceratto.json 2419971 wikitext text/x-wiki qin54fwmphtx03rnrzd3iratlf8xc29 2419972 2419971 2026-05-27T03:09:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T426633)', diff saved to https://phabricator.wikimedia.org/P93137 and previous config saved to /var/cache/conftool/dbconfig/20260527-030915-fceratto.json 2419972 wikitext text/x-wiki 6cylcwlo878iyhxn9aw4u4hj5xzp36r 2419973 2419972 2026-05-27T03:19:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93138 and previous config saved to /var/cache/conftool/dbconfig/20260527-031923-fceratto.json 2419973 wikitext text/x-wiki 3m4zgr4mn2lo06oun8ztldoy43fdq91 2419974 2419973 2026-05-27T03:29:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93139 and previous config saved to /var/cache/conftool/dbconfig/20260527-032931-fceratto.json 2419974 wikitext text/x-wiki m3i6oiby7ho19258lbtlifmqw3hhv86 2419975 2419974 2026-05-27T03:39:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T426633)', diff saved to https://phabricator.wikimedia.org/P93140 and previous config saved to /var/cache/conftool/dbconfig/20260527-033938-fceratto.json 2419975 wikitext text/x-wiki sl9px0ahsczgjzi5699bn1wi37c57cr 2419976 2419975 2026-05-27T03:40:02Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance 2419976 wikitext text/x-wiki lbj2pp1lsqa56yrpt6kizeod86k7hi1 2419977 2419976 2026-05-27T03:40:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 (T426633)', diff saved to https://phabricator.wikimedia.org/P93141 and previous config saved to /var/cache/conftool/dbconfig/20260527-034008-fceratto.json 2419977 wikitext text/x-wiki igkvv9ll227187fq162bb4gj71q6zwv 2419978 2419977 2026-05-27T03:48:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T426633)', diff saved to https://phabricator.wikimedia.org/P93142 and previous config saved to /var/cache/conftool/dbconfig/20260527-034828-fceratto.json 2419978 wikitext text/x-wiki qmcrie5babjohqmyohm2f7gxzm2gzlf 2419979 2419978 2026-05-27T03:58:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93143 and previous config saved to /var/cache/conftool/dbconfig/20260527-035836-fceratto.json 2419979 wikitext text/x-wiki 5ydmj13wnw4g1k5bcyicetjx2re2qdb 2419980 2419979 2026-05-27T04:08:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93144 and previous config saved to /var/cache/conftool/dbconfig/20260527-040844-fceratto.json 2419980 wikitext text/x-wiki m9gx5tj5fl38kr7i438udsqrxkhw2di 2419981 2419980 2026-05-27T04:18:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T426633)', diff saved to https://phabricator.wikimedia.org/P93145 and previous config saved to /var/cache/conftool/dbconfig/20260527-041852-fceratto.json 2419981 wikitext text/x-wiki k49v23iffomg085c2p4e7vdjgbbfaar 2419982 2419981 2026-05-27T04:19:16Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance 2419982 wikitext text/x-wiki ovq8u1unxb0hrkpwntij6taeyygj55o 2419983 2419982 2026-05-27T04:19:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 (T426633)', diff saved to https://phabricator.wikimedia.org/P93146 and previous config saved to /var/cache/conftool/dbconfig/20260527-041921-fceratto.json 2419983 wikitext text/x-wiki nzcu3nbvxk9adk2rqlyzbugtzv2kuhu 2419984 2419983 2026-05-27T04:27:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T426633)', diff saved to https://phabricator.wikimedia.org/P93147 and previous config saved to /var/cache/conftool/dbconfig/20260527-042737-fceratto.json 2419984 wikitext text/x-wiki rn2305rf7npbchq5lvnazkkpsoja6sn 2419985 2419984 2026-05-27T04:37:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93148 and previous config saved to /var/cache/conftool/dbconfig/20260527-043744-fceratto.json 2419985 wikitext text/x-wiki 1vixyqp8lo0j59vqgwk6xavl3veh1fh 2419986 2419985 2026-05-27T04:47:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93149 and previous config saved to /var/cache/conftool/dbconfig/20260527-044751-fceratto.json 2419986 wikitext text/x-wiki m8tup994is9nkxhyvovvzq97m73m0ui 2419987 2419986 2026-05-27T04:58:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T426633)', diff saved to https://phabricator.wikimedia.org/P93150 and previous config saved to /var/cache/conftool/dbconfig/20260527-045759-fceratto.json 2419987 wikitext text/x-wiki fuotuygizeihw636i1rg9wmudp2692h 2419988 2419987 2026-05-27T04:58:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance 2419988 wikitext text/x-wiki a8q2p5i1bo6565h7naoygbhy1i3e3qk 2419989 2419988 2026-05-27T04:58:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 (T426633)', diff saved to https://phabricator.wikimedia.org/P93151 and previous config saved to /var/cache/conftool/dbconfig/20260527-045827-fceratto.json 2419989 wikitext text/x-wiki 2lh0gc6n993z0910i0m94agvvdatf8a 2419990 2419989 2026-05-27T05:06:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T426633)', diff saved to https://phabricator.wikimedia.org/P93152 and previous config saved to /var/cache/conftool/dbconfig/20260527-050645-fceratto.json 2419990 wikitext text/x-wiki bezwwovqvz21ez56g1eeo7pgcv0o0o9 2419991 2419990 2026-05-27T05:16:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93153 and previous config saved to /var/cache/conftool/dbconfig/20260527-051653-fceratto.json 2419991 wikitext text/x-wiki ffhhglnhr74lazmjgrhie6jgt7mv6fj 2419992 2419991 2026-05-27T05:26:27Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1014 from dbctl T427270', diff saved to https://phabricator.wikimedia.org/P93154 and previous config saved to /var/cache/conftool/dbconfig/20260527-052624-marostegui.json 2419992 wikitext text/x-wiki t9q01xb0ostxj9r7jid5xonksx1a96d 2419993 2419992 2026-05-27T05:27:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93155 and previous config saved to /var/cache/conftool/dbconfig/20260527-052700-fceratto.json 2419993 wikitext text/x-wiki roa86rntnjyridgvcryyzn2f91p7egg 2419994 2419993 2026-05-27T05:37:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T426633)', diff saved to https://phabricator.wikimedia.org/P93156 and previous config saved to /var/cache/conftool/dbconfig/20260527-053708-fceratto.json 2419994 wikitext text/x-wiki 0pr5tvlwci8r6rtic9euk117117ajiy 2419995 2419994 2026-05-27T05:37:22Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance 2419995 wikitext text/x-wiki 856bmbelpwinm0p5dswddy2wfs31mk2 2419996 2419995 2026-05-27T05:37:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 (T426633)', diff saved to https://phabricator.wikimedia.org/P93157 and previous config saved to /var/cache/conftool/dbconfig/20260527-053727-fceratto.json 2419996 wikitext text/x-wiki 89nhnvuo7t17zipffrsywtq5ihmdce7 2419997 2419996 2026-05-27T05:38:10Z Stashbot 7414 moritzm: remove ganeti1026 from eqiad Ganeti cluster T424680 2419997 wikitext text/x-wiki 4oavtal4mdjfhhh0xzgfcurrbk4hakk 2419998 2419997 2026-05-27T05:40:06Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419998 wikitext text/x-wiki t9t1fv8bg7hgj52kdk9lndj4k6oykm1 2419999 2419998 2026-05-27T05:40:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2055: Upgrading es2055.codfw.wmnet 2419999 wikitext text/x-wiki m7kde39jguscn58j3tw1hkk5vojr5gs 2420000 2419999 2026-05-27T05:40:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2055: Upgrading es2055.codfw.wmnet 2420000 wikitext text/x-wiki gshh0c108g07q5ynypmqdtxyx1xlu0o 2420001 2420000 2026-05-27T05:41:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2055.codfw.wmnet with OS trixie 2420001 wikitext text/x-wiki flklm7lrsssdsb63wks9eye1a4oi3xw 2420002 2420001 2026-05-27T05:45:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 (T426633)', diff saved to https://phabricator.wikimedia.org/P93159 and previous config saved to /var/cache/conftool/dbconfig/20260527-054550-fceratto.json 2420002 wikitext text/x-wiki goxjqe14ppjqfs6z4dc9oftm19epw6f 2420005 2420002 2026-05-27T05:56:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93160 and previous config saved to /var/cache/conftool/dbconfig/20260527-055558-fceratto.json 2420005 wikitext text/x-wiki tjbj8fjc4obqbcf7u584cj0ittj7ur6 2420006 2420005 2026-05-27T05:56:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2055.codfw.wmnet with reason: host reimage 2420006 wikitext text/x-wiki 5xu2u368jv8asfnwxg3g786m1i6sput 2420008 2420006 2026-05-27T06:04:18Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2055.codfw.wmnet with reason: host reimage 2420008 wikitext text/x-wiki nbktz7wybz3ysmrf2zpx4cswrtl7uz6 2420009 2420008 2026-05-27T06:06:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93161 and previous config saved to /var/cache/conftool/dbconfig/20260527-060606-fceratto.json 2420009 wikitext text/x-wiki q5fgyb5d3jqlqgmvp70uuekxnl8nyl6 2420010 2420009 2026-05-27T06:16:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 (T426633)', diff saved to https://phabricator.wikimedia.org/P93162 and previous config saved to /var/cache/conftool/dbconfig/20260527-061613-fceratto.json 2420010 wikitext text/x-wiki szhaki29ng9pkj5jdp3n37ujnovcvwd 2420011 2420010 2026-05-27T06:16:38Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance 2420011 wikitext text/x-wiki idfejuh0qvumnqsc1dfkmt36ims8o2k 2420012 2420011 2026-05-27T06:16:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 (T426633)', diff saved to https://phabricator.wikimedia.org/P93163 and previous config saved to /var/cache/conftool/dbconfig/20260527-061643-fceratto.json 2420012 wikitext text/x-wiki 37b2v52jnwobxhaxrkqjr0q6gu8ffi1 2420014 2420012 2026-05-27T06:21:11Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2055.codfw.wmnet with OS trixie 2420014 wikitext text/x-wiki bj0mcl99qgev4kl9g913jxd9egiyn3s 2420015 2420014 2026-05-27T06:21:58Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2420015 wikitext text/x-wiki fuvjbkyjdfx6qguobbudd79wtyyrjs8 2420016 2420015 2026-05-27T06:22:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2055: repool after maintenance 2420016 wikitext text/x-wiki c0kdq33irljzw5a7c0d1avnj537zytp 2420017 2420016 2026-05-27T06:25:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 (T426633)', diff saved to https://phabricator.wikimedia.org/P93165 and previous config saved to /var/cache/conftool/dbconfig/20260527-062503-fceratto.json 2420017 wikitext text/x-wiki ck3o149pcyb7135wjoejmbhdfnj9324 2420018 2420017 2026-05-27T06:35:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93166 and previous config saved to /var/cache/conftool/dbconfig/20260527-063511-fceratto.json 2420018 wikitext text/x-wiki lzaj6uf2solr7712s56cgkrd1txngb1 2420019 2420018 2026-05-27T06:45:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93168 and previous config saved to /var/cache/conftool/dbconfig/20260527-064519-fceratto.json 2420019 wikitext text/x-wiki efjditav79zbzjdtga5mdw20c9ape5b 2420020 2420019 2026-05-27T06:54:44Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1025.eqiad.wmnet 2420020 wikitext text/x-wiki s09chm2dsqvh8vlgq0q7a6dbujkrmdo 2420021 2420020 2026-05-27T06:55:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 (T426633)', diff saved to https://phabricator.wikimedia.org/P93170 and previous config saved to /var/cache/conftool/dbconfig/20260527-065526-fceratto.json 2420021 wikitext text/x-wiki ojamwd4h30to8dn2g8keioyo47itgim 2420022 2420021 2026-05-27T06:55:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance 2420022 wikitext text/x-wiki 4cxexax0asypkgt803vjaoio2isqhcf 2420023 2420022 2026-05-27T06:55:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 (T426633)', diff saved to https://phabricator.wikimedia.org/P93171 and previous config saved to /var/cache/conftool/dbconfig/20260527-065545-fceratto.json 2420023 wikitext text/x-wiki jlx1ic4k2qs9vo50znu92ty94gr7c9n 2420024 2420023 2026-05-27T07:02:34Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1003.wikimedia.org 2420024 wikitext text/x-wiki j6qo1ij1u1wrrwxsyg9i64cxsrhg75z 2420025 2420024 2026-05-27T07:04:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 (T426633)', diff saved to https://phabricator.wikimedia.org/P93172 and previous config saved to /var/cache/conftool/dbconfig/20260527-070410-fceratto.json 2420025 wikitext text/x-wiki qez0eveqajutur1cst4dtmrmh32sw89 2420026 2420025 2026-05-27T07:06:03Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: Maintenance on db1190 2420026 wikitext text/x-wiki sxzheiejvgmifyb4mbx2supzzodlxbm 2420027 2420026 2026-05-27T07:06:18Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2420027 wikitext text/x-wiki obt2r285v92zljfpk6oz8dqnig4pg06 2420028 2420027 2026-05-27T07:07:03Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1003.wikimedia.org 2420028 wikitext text/x-wiki b7jbda182q6mciyxsrpk97iq0p88x4i 2420029 2420028 2026-05-27T07:07:27Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org 2420029 wikitext text/x-wiki pihahy74znnblyyggujxdm5h3haqyq0 2420030 2420029 2026-05-27T07:07:59Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2055: repool after maintenance 2420030 wikitext text/x-wiki 34pp6a05kcgw8s6cvbjmnbd277ut8yl 2420031 2420030 2026-05-27T07:11:57Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org 2420031 wikitext text/x-wiki mu3d2pwy6vn20itdcvsajktbjz920z2 2420032 2420031 2026-05-27T07:13:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2420032 wikitext text/x-wiki 04varhkdewz3wis1gj9lzxn90dwv1ef 2420033 2420032 2026-05-27T07:13:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.decommission 2420033 wikitext text/x-wiki pcvx4xm0feohd8uewpq6m4wwv4wm58r 2420034 2420033 2026-05-27T07:13:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1014.eqiad.wmnet 2420034 wikitext text/x-wiki 916j2cmjruo15ywy6gvevig8uc3wujb 2420035 2420034 2026-05-27T07:14:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93174 and previous config saved to /var/cache/conftool/dbconfig/20260527-071418-fceratto.json 2420035 wikitext text/x-wiki 7wvqvta8nilly6bzi41bsrliud6lrzt 2420036 2420035 2026-05-27T07:14:27Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2420036 wikitext text/x-wiki 5ju4uo21alr6g77oruof84smsxd4z2m 2420037 2420036 2026-05-27T07:14:30Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2420037 wikitext text/x-wiki mite2gn0k1mqa13yefo3gh871wma5i4 2420038 2420037 2026-05-27T07:14:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1025.eqiad.wmnet 2420038 wikitext text/x-wiki qmd54kj2za66hubg2q6sjude218ws60 2420039 2420038 2026-05-27T07:15:21Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1026.eqiad.wmnet 2420039 wikitext text/x-wiki k4vgw8u4lrtgd9m1pv595fvh1bjll13 2420040 2420039 2026-05-27T07:18:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2420040 wikitext text/x-wiki a3j3xusy4rb7qu7h32eeej7vjqasv7w 2420041 2420040 2026-05-27T07:23:11Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2420041 wikitext text/x-wiki khdujs7n7d42ul0k22xzfxpxjdhb7u5 2420042 2420041 2026-05-27T07:23:25Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2420042 wikitext text/x-wiki huki732xg79bghx3rrtb7y46zaldotg 2420043 2420042 2026-05-27T07:23:28Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2420043 wikitext text/x-wiki or922d2snbyol5x9j3ne4sze4kh7l7q 2420044 2420043 2026-05-27T07:23:31Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1014.eqiad.wmnet 2420044 wikitext text/x-wiki tw6i8bqcq6xmklvt8a8o22mk046aiy7 2420045 2420044 2026-05-27T07:23:34Z Stashbot 7414 marostegui@cumin1003: Removing pc1014 from zarcillo T427190 2420045 wikitext text/x-wiki jcz7biya5m4auchcxzx5iemkww58ndb 2420046 2420045 2026-05-27T07:23:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) 2420046 wikitext text/x-wiki s1q42xq30bnxf57ivcpivwyoxicne08 2420047 2420046 2026-05-27T07:24:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93175 and previous config saved to /var/cache/conftool/dbconfig/20260527-072426-fceratto.json 2420047 wikitext text/x-wiki h36k491ewqprrsscr11cw87ehdql6ub 2420048 2420047 2026-05-27T07:28:56Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2420048 wikitext text/x-wiki 834dwnmie7rvwgg20cnowz6stst204b 2420049 2420048 2026-05-27T07:33:39Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1294125|Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126|Add script to demote ineligible members of restricted global groups (T425395)]] 2420049 wikitext text/x-wiki or2xq4sd6baz93ebc1g2qqnxqq88l1z 2420050 2420049 2026-05-27T07:34:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 (T426633)', diff saved to https://phabricator.wikimedia.org/P93176 and previous config saved to /var/cache/conftool/dbconfig/20260527-073434-fceratto.json 2420050 wikitext text/x-wiki 2ivppaemdrm09pam33lifkqi6tnv993 2420051 2420050 2026-05-27T07:34:58Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance 2420051 wikitext text/x-wiki 3j056rx86uhk1eqfzlhuq401ew2cwpz 2420052 2420051 2026-05-27T07:35:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2248 (T426633)', diff saved to https://phabricator.wikimedia.org/P93177 and previous config saved to /var/cache/conftool/dbconfig/20260527-073504-fceratto.json 2420052 wikitext text/x-wiki 5heu79bq0shdavdl1xvpz40xo4aegl4 2420053 2420052 2026-05-27T07:35:38Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1294125|Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126|Add script to demote ineligible members of restricted global groups (T425395)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420053 wikitext text/x-wiki epfq25v2m1ny0uorgv4t702uv2z8gun 2420054 2420053 2026-05-27T07:36:03Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2420054 wikitext text/x-wiki f02mu3geyesh096nifiwkffak2vz0ku 2420055 2420054 2026-05-27T07:40:22Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294125|Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126|Add script to demote ineligible members of restricted global groups (T425395)]] (duration: 06m 42s) 2420055 wikitext text/x-wiki cxogbznn7ujx22drrmngxt3143lqmpq 2420056 2420055 2026-05-27T07:40:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 (T426633)', diff saved to https://phabricator.wikimedia.org/P93178 and previous config saved to /var/cache/conftool/dbconfig/20260527-074031-fceratto.json 2420056 wikitext text/x-wiki 6dffsbvygrz2ngbzlirkf3pnxtm8wk2 2420057 2420056 2026-05-27T07:41:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420057 wikitext text/x-wiki 1b3onwvaxng1okqdxqeyc07fvnzja0s 2420058 2420057 2026-05-27T07:42:03Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2051: Upgrading es2051.codfw.wmnet 2420058 wikitext text/x-wiki a4755b3s3r2rndvtouwqaz200769l5c 2420059 2420058 2026-05-27T07:42:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: Upgrading es2051.codfw.wmnet 2420059 wikitext text/x-wiki 70d1k3am1he1b9fz8imtvyr7muacczq 2420060 2420059 2026-05-27T07:43:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2420060 wikitext text/x-wiki 2mj7shx8e0s528qkqli1f4oyxf3lkvu 2420061 2420060 2026-05-27T07:43:42Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2420061 wikitext text/x-wiki s72dst0bgfvafjokpkv64x9v93k1nj6 2420062 2420061 2026-05-27T07:43:45Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2420062 wikitext text/x-wiki mo94uz94rpcejh6xeg8vb9isl9aitq8 2420063 2420062 2026-05-27T07:43:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1026.eqiad.wmnet 2420063 wikitext text/x-wiki nsjn2n6ui27imtfnz3wht345a4nukx2 2420064 2420063 2026-05-27T07:50:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93180 and previous config saved to /var/cache/conftool/dbconfig/20260527-075039-fceratto.json 2420064 wikitext text/x-wiki 1tlik3s5nc8rnrn2rb5xnso4xoz7ihp 2420067 2420064 2026-05-27T07:59:11Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2051.codfw.wmnet with OS trixie 2420067 wikitext text/x-wiki o4oa9hrd3lfgttpgw396jf69iogfbsg 2420068 2420067 2026-05-27T08:00:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93181 and previous config saved to /var/cache/conftool/dbconfig/20260527-080046-fceratto.json 2420068 wikitext text/x-wiki pibjwso0ir07xaziv38jjmvv29lmw5p 2420069 2420068 2026-05-27T08:05:45Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2420069 wikitext text/x-wiki 99tot0fsmvrh8fj2nl66m1sim680nxh 2420070 2420069 2026-05-27T08:07:26Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2420070 wikitext text/x-wiki 2j8oufkc70cx60zq4wktfx8kajpyv3j 2420072 2420070 2026-05-27T08:10:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 (T426633)', diff saved to https://phabricator.wikimedia.org/P93182 and previous config saved to /var/cache/conftool/dbconfig/20260527-081054-fceratto.json 2420072 wikitext text/x-wiki f2lt184yc23yfwa5gcaijqtwx890270 2420073 2420072 2026-05-27T08:11:07Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance 2420073 wikitext text/x-wiki mxeqhuk6zysz7z8rnsytl2hfa91gcdu 2420074 2420073 2026-05-27T08:11:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2153 (T426633)', diff saved to https://phabricator.wikimedia.org/P93183 and previous config saved to /var/cache/conftool/dbconfig/20260527-081112-fceratto.json 2420074 wikitext text/x-wiki 2wt56suml3s1b3y7mncyvqp7bvkx8c4 2420075 2420074 2026-05-27T08:11:59Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.4 refs T423913 2420075 wikitext text/x-wiki jdwuwshcu1eyahc0pohu81cfyffh0ue 2420077 2420075 2026-05-27T08:15:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2051.codfw.wmnet with reason: host reimage 2420077 wikitext text/x-wiki 9duuagqaytkzkwlifap88xcwwur2qv6 2420078 2420077 2026-05-27T08:18:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2051.codfw.wmnet with reason: host reimage 2420078 wikitext text/x-wiki 8sa8sq0gi9tzygqm44x6w2q2r8nzzmh 2420079 2420078 2026-05-27T08:19:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T426633)', diff saved to https://phabricator.wikimedia.org/P93184 and previous config saved to /var/cache/conftool/dbconfig/20260527-081942-fceratto.json 2420079 wikitext text/x-wiki 9ajt5hye4wurre782am2vezda47sf8z 2420081 2420079 2026-05-27T08:27:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org 2420081 wikitext text/x-wiki 3492nc0ir6b97kcef0x94n3dqfhzfv6 2420082 2420081 2026-05-27T08:29:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93185 and previous config saved to /var/cache/conftool/dbconfig/20260527-082950-fceratto.json 2420082 wikitext text/x-wiki n44ho91amgq1d51iz2perls75v7uuft 2420083 2420082 2026-05-27T08:31:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org 2420083 wikitext text/x-wiki cnxciinnzy836y5n7yfssqhmptse54p 2420084 2420083 2026-05-27T08:32:09Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1004.wikimedia.org 2420084 wikitext text/x-wiki m3y20f3grwbp8hreluxqb9g5fvem94o 2420085 2420084 2026-05-27T08:33:47Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420085 wikitext text/x-wiki ie27mu6v2dw1jkuqh5y2n1rbp0p434n 2420086 2420085 2026-05-27T08:33:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2166: Upgrading db2166.codfw.wmnet 2420086 wikitext text/x-wiki oycfytte8glzsuzggosxo7q3wmgfmqe 2420087 2420086 2026-05-27T08:34:18Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2166: Upgrading db2166.codfw.wmnet 2420087 wikitext text/x-wiki 20lqz6a6c58fqo0dsyk568d592iearl 2420088 2420087 2026-05-27T08:35:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2051.codfw.wmnet with OS trixie 2420088 wikitext text/x-wiki pjq8xpx75wqrx85zxsi2vr350d4y1m3 2420089 2420088 2026-05-27T08:35:39Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2166.codfw.wmnet with OS trixie 2420089 wikitext text/x-wiki ahovklno4eyuty0solk0bl027cdgene 2420090 2420089 2026-05-27T08:36:02Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420090 wikitext text/x-wiki g200rt226efisb7lss4djjyfbn2cwbj 2420091 2420090 2026-05-27T08:36:23Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1203: Upgrading db1203.eqiad.wmnet 2420091 wikitext text/x-wiki 6i9kp689fh7aqyiokwiw453wfpwrhgp 2420092 2420091 2026-05-27T08:36:38Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1004.wikimedia.org 2420092 wikitext text/x-wiki ngbv7biinmjdc4s2nl5vcblt7t0offf 2420093 2420092 2026-05-27T08:36:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1203: Upgrading db1203.eqiad.wmnet 2420093 wikitext text/x-wiki 3sy9alpwmz7saov6i4zk5g2hnbn2g63 2420094 2420093 2026-05-27T08:37:45Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2420094 wikitext text/x-wiki cegpl237g0de8zn9riia082gffj048r 2420095 2420094 2026-05-27T08:38:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2051: repool after maintenance 2420095 wikitext text/x-wiki mpgx98veb301yeyudxypuaabcjjyxrb 2420097 2420095 2026-05-27T08:40:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93189 and previous config saved to /var/cache/conftool/dbconfig/20260527-083957-fceratto.json 2420097 wikitext text/x-wiki sra01axlw4v5po1axbmijbklww4qia4 2420098 2420097 2026-05-27T08:41:30Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1203.eqiad.wmnet with OS trixie 2420098 wikitext text/x-wiki dwciiywnpww1hyjjvz5shydo3m0b300 2420099 2420098 2026-05-27T08:50:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T426633)', diff saved to https://phabricator.wikimedia.org/P93190 and previous config saved to /var/cache/conftool/dbconfig/20260527-085005-fceratto.json 2420099 wikitext text/x-wiki l3jhgj74afgkbor184iovlzv80dairj 2420100 2420099 2026-05-27T08:50:18Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance 2420100 wikitext text/x-wiki 9xn9gkgf4th9da9ghsspcc6d29q94y1 2420101 2420100 2026-05-27T08:50:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2170 (T426633)', diff saved to https://phabricator.wikimedia.org/P93191 and previous config saved to /var/cache/conftool/dbconfig/20260527-085024-fceratto.json 2420101 wikitext text/x-wiki sm404agavhs3hhii6mclrr7l8840a3o 2420102 2420101 2026-05-27T08:50:32Z Stashbot 7414 fabfur: depooling and installing haproxy-awslc on cp3074 and cp3066 (T419825) 2420102 wikitext text/x-wiki 6nmv3tniv4y2ujtbjt2cufgbbkz6kzc 2420103 2420102 2026-05-27T08:51:18Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2420103 wikitext text/x-wiki 8a88lm6h3grtkf7ji86om125p8spihs 2420104 2420103 2026-05-27T08:51:30Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.* 2420104 wikitext text/x-wiki 0kphwv0bpr3gdxvpobni7kv56jztukr 2420105 2420104 2026-05-27T08:51:43Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* 2420105 wikitext text/x-wiki f8d2cuaqcc0uoq369g35irhf78ikpw2 2420106 2420105 2026-05-27T08:51:45Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2420106 wikitext text/x-wiki 7awpjkonbvx7k1j1l1zmrg4gdntg34q 2420107 2420106 2026-05-27T08:51:48Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2420107 wikitext text/x-wiki l6ed04hz1950c67cilzmijesb4lr4sc 2420109 2420107 2026-05-27T08:52:05Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2420109 wikitext text/x-wiki bsa3hsixuprzg22zv8hvnijmdqndl8f 2420110 2420109 2026-05-27T08:52:10Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2420110 wikitext text/x-wiki 538684op0u3trqmrhgnjplt045lew58 2420111 2420110 2026-05-27T08:52:25Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2420111 wikitext text/x-wiki dkyh9p0ryncelkh6da7ch1s7lnetxk5 2420112 2420111 2026-05-27T08:52:30Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2420112 wikitext text/x-wiki m4foov4l9gg0hmpshevlkr7cqliwqo1 2420113 2420112 2026-05-27T08:52:48Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2420113 wikitext text/x-wiki op1fperq1gmm1rfvlkeb72dfn03d8jj 2420114 2420113 2026-05-27T08:52:53Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2420114 wikitext text/x-wiki 91p7cupdggbp80u2m98apvfv4fg76f9 2420115 2420114 2026-05-27T08:53:09Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2420115 wikitext text/x-wiki t6ywu15nd7kqs2ap11ypcus41ny6ihj 2420116 2420115 2026-05-27T08:53:13Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2420116 wikitext text/x-wiki l4nlz08y1kk7v47khtmfnas489pkxcb 2420117 2420116 2026-05-27T08:53:31Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2420117 wikitext text/x-wiki cb1d58ds8vc8rve8ov82taq5pzml957 2420118 2420117 2026-05-27T08:53:35Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2420118 wikitext text/x-wiki hww8hnw7eqjneh5h9aeavz2pid645jn 2420119 2420118 2026-05-27T08:54:00Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2420119 wikitext text/x-wiki 0xyqqyltnu5rqu544bb7gwjc332qzyg 2420120 2420119 2026-05-27T08:54:04Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2420120 wikitext text/x-wiki bvw3qz2qk8usokv1lho4zhr7ln22a17 2420121 2420120 2026-05-27T08:54:06Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2166.codfw.wmnet with reason: host reimage 2420121 wikitext text/x-wiki i8up82obio7w5m1x2yx7tx0jcrgs8h1 2420122 2420121 2026-05-27T08:54:22Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2420122 wikitext text/x-wiki rha3yf0qlnhj1xt3ioowj39sete5kpk 2420123 2420122 2026-05-27T08:54:43Z Stashbot 7414 Emperor: restart swift on ms-fe2011 T360913 2420123 wikitext text/x-wiki nlflj7cnsxnp0m3aktdu09kx7rv4mfq 2420124 2420123 2026-05-27T08:55:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage 2420124 wikitext text/x-wiki 5u5oioa32tf2qmyb3611yyhdhje1va5 2420125 2420124 2026-05-27T08:57:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T426633)', diff saved to https://phabricator.wikimedia.org/P93193 and previous config saved to /var/cache/conftool/dbconfig/20260527-085751-fceratto.json 2420125 wikitext text/x-wiki giyo46pjaiwm67vqg2mzyxcb3utzo4s 2420128 2420125 2026-05-27T08:59:24Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: host reimage 2420128 wikitext text/x-wiki fyr47dvqrvbcts2f3t3edu6j0l0drq6 2420129 2420128 2026-05-27T09:02:33Z Stashbot 7414 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp6015.* 2420129 wikitext text/x-wiki qqx341n455477p9pmm5wiojb958hhrp 2420130 2420129 2026-05-27T09:02:40Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage 2420130 wikitext text/x-wiki 7mdpjqt4u7edr3txml74st1y4g8k3r9 2420131 2420130 2026-05-27T09:02:48Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp6015.drmrs.wmnet 2420131 wikitext text/x-wiki ft5r7o78htvtdlkyfkl16jh0es8ak6a 2420132 2420131 2026-05-27T09:02:51Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp6015.drmrs.wmnet 2420132 wikitext text/x-wiki d1cfm2xervdet9chfy6igdjn8gh3y2u 2420133 2420132 2026-05-27T09:03:01Z Stashbot 7414 fabfur: repooling cp3074 and cp3066 (T419825) 2420133 wikitext text/x-wiki gj2jfyim2nic3gb82xwad9y3lhqwc87 2420134 2420133 2026-05-27T09:03:11Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* 2420134 wikitext text/x-wiki hxxghywoue8hmz62i00lorq5ehx3bu9 2420135 2420134 2026-05-27T09:03:17Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.* 2420135 wikitext text/x-wiki fgf1nk90c3tf5z019vcd1jagdoistoo 2420136 2420135 2026-05-27T09:08:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93194 and previous config saved to /var/cache/conftool/dbconfig/20260527-090759-fceratto.json 2420136 wikitext text/x-wiki 0mfpexfzg3c4mndou98upnuhvi3kdb0 2420138 2420136 2026-05-27T09:16:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2166.codfw.wmnet with OS trixie 2420138 wikitext text/x-wiki jfw1wlzx1s726gmkle6o81r21vbxjx7 2420139 2420138 2026-05-27T09:18:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93196 and previous config saved to /var/cache/conftool/dbconfig/20260527-091806-fceratto.json 2420139 wikitext text/x-wiki stt1xjz8ywe8hxtpn5eu6jryewewjcb 2420140 2420139 2026-05-27T09:20:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1203.eqiad.wmnet with OS trixie 2420140 wikitext text/x-wiki rv0irt06191qb7278u0nkb0v8ahtopw 2420141 2420140 2026-05-27T09:23:52Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: repool after maintenance 2420141 wikitext text/x-wiki 1g8sm4xp8mpehhwiq6gzsd3xsdky2rs 2420142 2420141 2026-05-27T09:24:32Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2166: Migration of db2166.codfw.wmnet completed 2420142 wikitext text/x-wiki o3opxa96ce271j6y19qehumogluzrp4 2420143 2420142 2026-05-27T09:25:21Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance 2420143 wikitext text/x-wiki 5nxzi26sk3ihjdycp3bgb77ouq40k65 2420144 2420143 2026-05-27T09:25:24Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance 2420144 wikitext text/x-wiki 782vf05mhywu6zdy89ojanrka8c67wg 2420145 2420144 2026-05-27T09:25:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420145 wikitext text/x-wiki avx2zknzxk7yj0addny4rhslzxisn1l 2420146 2420145 2026-05-27T09:25:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1050: Upgrading es1050.eqiad.wmnet 2420146 wikitext text/x-wiki l2081xh71rs607ihcyldk9q427gsuiy 2420147 2420146 2026-05-27T09:26:15Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1050: Upgrading es1050.eqiad.wmnet 2420147 wikitext text/x-wiki nciisz90zd050s31mqvp0ab1yizcdp9 2420148 2420147 2026-05-27T09:27:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1050.eqiad.wmnet with OS trixie 2420148 wikitext text/x-wiki q3lsewih9wrho0wnclnxvgcmhflz37u 2420149 2420148 2026-05-27T09:28:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T426633)', diff saved to https://phabricator.wikimedia.org/P93200 and previous config saved to /var/cache/conftool/dbconfig/20260527-092814-fceratto.json 2420149 wikitext text/x-wiki o1alrpabynx9pmis68qkd52c65dov4t 2420150 2420149 2026-05-27T09:28:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1203: Migration of db1203.eqiad.wmnet completed 2420150 wikitext text/x-wiki 6irznju6ynmoaqb3eo85kiuahxfkbqd 2420151 2420150 2026-05-27T09:28:37Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance 2420151 wikitext text/x-wiki niw8ko52tv6ja9x2uzj42jep3t5hkrh 2420152 2420151 2026-05-27T09:28:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2173 (T426633)', diff saved to https://phabricator.wikimedia.org/P93202 and previous config saved to /var/cache/conftool/dbconfig/20260527-092842-fceratto.json 2420152 wikitext text/x-wiki 70nm8p5ntintmckcl28aeiddaj7f5wx 2420154 2420152 2026-05-27T09:34:50Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2420154 wikitext text/x-wiki mufr9qbiltl92jajxv2a3py6qqvhsgw 2420156 2420154 2026-05-27T09:36:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T426633)', diff saved to https://phabricator.wikimedia.org/P93203 and previous config saved to /var/cache/conftool/dbconfig/20260527-093609-fceratto.json 2420156 wikitext text/x-wiki 0y1ye90mup6sfh3hohn7tj2mmgzyp41 2420157 2420156 2026-05-27T09:36:20Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2420157 wikitext text/x-wiki s11w124ixx3il1gr5pnb5u430ya68zu 2420158 2420157 2026-05-27T09:37:02Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2420158 wikitext text/x-wiki q2phrvcs3ss0namti1p7nxesz7rogf4 2420159 2420158 2026-05-27T09:37:18Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2420159 wikitext text/x-wiki e6m6fyqv5dzfn2g2xg9dcs3jy7v2nua 2420160 2420159 2026-05-27T09:38:06Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2420160 wikitext text/x-wiki oxab8sr4ry181m18zs476k6anhvjqe1 2420161 2420160 2026-05-27T09:38:21Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2420161 wikitext text/x-wiki fufliuv0j1n0yyypvonzo2qwi7pocwk 2420162 2420161 2026-05-27T09:41:48Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage 2420162 wikitext text/x-wiki 3ob4stiwss6lkstnr865e86fykll511 2420164 2420162 2026-05-27T09:43:52Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2420164 wikitext text/x-wiki bv8l97x29mdqv3tsc43w9hqgx5249i9 2420165 2420164 2026-05-27T09:46:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage 2420165 wikitext text/x-wiki lvtsa5no3mfrpn4gt5qsdqs4bzh4tkl 2420166 2420165 2026-05-27T09:46:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93206 and previous config saved to /var/cache/conftool/dbconfig/20260527-094616-fceratto.json 2420166 wikitext text/x-wiki hqbe6f1v2mu66m2kv3x92ag53h7aef8 2420167 2420166 2026-05-27T09:47:27Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2420167 wikitext text/x-wiki haur5scng3853x5hh5m0we3gboxxb5e 2420170 2420167 2026-05-27T09:56:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93208 and previous config saved to /var/cache/conftool/dbconfig/20260527-095624-fceratto.json 2420170 wikitext text/x-wiki fk5fmjzyml6icl3xb5hf86j1c5ly1wj 2420171 2420170 2026-05-27T10:02:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1050.eqiad.wmnet with OS trixie 2420171 wikitext text/x-wiki flcfagjpcjku8elg8o67wdqt0iv7on5 2420173 2420171 2026-05-27T10:04:41Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2420173 wikitext text/x-wiki mnmwyntargcttys2sur81xtd2v6e90v 2420174 2420173 2026-05-27T10:05:11Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance 2420174 wikitext text/x-wiki 9cfkf1yyfasft19fdtqogfm139cp5y2 2420175 2420174 2026-05-27T10:06:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T426633)', diff saved to https://phabricator.wikimedia.org/P93211 and previous config saved to /var/cache/conftool/dbconfig/20260527-100632-fceratto.json 2420175 wikitext text/x-wiki bxiesc01ezvb628nz0hgrsmeyjs988v 2420176 2420175 2026-05-27T10:06:56Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance 2420176 wikitext text/x-wiki 1vgt7sah1zszq9lgjlvxxlzzj8yfb6n 2420177 2420176 2026-05-27T10:07:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2174 (T426633)', diff saved to https://phabricator.wikimedia.org/P93212 and previous config saved to /var/cache/conftool/dbconfig/20260527-100701-fceratto.json 2420177 wikitext text/x-wiki okuag0ul6fk6zhmr18g2hek946or9mx 2420178 2420177 2026-05-27T10:08:41Z Stashbot 7414 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2420178 wikitext text/x-wiki 748yvx6pkvtz092ri156f84oiniqf7f 2420179 2420178 2026-05-27T10:10:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Migration of db2166.codfw.wmnet completed 2420179 wikitext text/x-wiki sz4zkw4340i6fnaanzlbx8rcsy2etqe 2420180 2420179 2026-05-27T10:10:04Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2420180 wikitext text/x-wiki t9paoytnepeiuojnlrtgbfak9gb6iic 2420181 2420180 2026-05-27T10:10:32Z Stashbot 7414 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2420181 wikitext text/x-wiki nslvw1earbg3vprze7vh93w6s9sd7hd 2420182 2420181 2026-05-27T10:14:03Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1203: Migration of db1203.eqiad.wmnet completed 2420182 wikitext text/x-wiki qbeidnjyr5p5lgy7734jzqjik68xxx0 2420183 2420182 2026-05-27T10:14:07Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2420183 wikitext text/x-wiki 0620g9q6iou3rtvxp7jm3eknmftfok4 2420184 2420183 2026-05-27T10:14:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T426633)', diff saved to https://phabricator.wikimedia.org/P93215 and previous config saved to /var/cache/conftool/dbconfig/20260527-101426-fceratto.json 2420184 wikitext text/x-wiki skfu6ou30xpw3wcbhwy8d5rkogrjhag 2420187 2420184 2026-05-27T10:21:39Z Stashbot 7414 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2420187 wikitext text/x-wiki 4n8e55r1uj7z9wjzbjwfjb4bg5161ro 2420188 2420187 2026-05-27T10:22:04Z Stashbot 7414 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2420188 wikitext text/x-wiki 5s8xwutqsmygk83goz357ontfeewohx 2420190 2420188 2026-05-27T10:24:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93217 and previous config saved to /var/cache/conftool/dbconfig/20260527-102434-fceratto.json 2420190 wikitext text/x-wiki nle8erpbnhx2y4ra9zkryye1a1eay5e 2420191 2420190 2026-05-27T10:29:21Z Stashbot 7414 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2420191 wikitext text/x-wiki ernfsko490hioexs5a2gqb7nrohmeud 2420192 2420191 2026-05-27T10:29:43Z Stashbot 7414 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2420192 wikitext text/x-wiki c57jx9c5wonnnvq845ir3141jsrfae4 2420193 2420192 2026-05-27T10:34:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93218 and previous config saved to /var/cache/conftool/dbconfig/20260527-103441-fceratto.json 2420193 wikitext text/x-wiki asz2e894pbva6bmh9beylxr6mz6cuda 2420194 2420193 2026-05-27T10:34:54Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420194 wikitext text/x-wiki j63fu7czxuuqa5adpvvt8y5a513pjq3 2420195 2420194 2026-05-27T10:35:03Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2165: Upgrading db2165.codfw.wmnet 2420195 wikitext text/x-wiki 4hxh24v5jimsky1q5bis96q75m8ixs3 2420197 2420195 2026-05-27T10:35:22Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2165: Upgrading db2165.codfw.wmnet 2420197 wikitext text/x-wiki 3belw0kjkj9mjprz1nk7trnzrt1x1mj 2420198 2420197 2026-05-27T10:35:31Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420198 wikitext text/x-wiki sud5l31s124kgzlo4a8y5tfoqrc34s1 2420199 2420198 2026-05-27T10:35:52Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1193: Upgrading db1193.eqiad.wmnet 2420199 wikitext text/x-wiki 94j2m8yje8jlpsze4r4foo7f79583og 2420200 2420199 2026-05-27T10:36:31Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1193: Upgrading db1193.eqiad.wmnet 2420200 wikitext text/x-wiki sbkln3s936djsduss5ftjh2mlgqjvb6 2420201 2420200 2026-05-27T10:38:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1193.eqiad.wmnet with OS trixie 2420201 wikitext text/x-wiki e0w9rk5u652gqv55dllwzhgjn7ta0tf 2420202 2420201 2026-05-27T10:39:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS trixie 2420202 wikitext text/x-wiki 4o30g7mf1zmv88as5bbh265qvyndlxx 2420203 2420202 2026-05-27T10:44:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T426633)', diff saved to https://phabricator.wikimedia.org/P93222 and previous config saved to /var/cache/conftool/dbconfig/20260527-104449-fceratto.json 2420203 wikitext text/x-wiki eevjkoenjgl3q9rkd1lhu58c3dg2adb 2420204 2420203 2026-05-27T10:45:13Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance 2420204 wikitext text/x-wiki mkjig4c7mlh7282813nkgpmv92wd7j6 2420205 2420204 2026-05-27T10:45:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2176 (T426633)', diff saved to https://phabricator.wikimedia.org/P93223 and previous config saved to /var/cache/conftool/dbconfig/20260527-104518-fceratto.json 2420205 wikitext text/x-wiki 3zc6ck08i4mi6y0sth4gshvtc9euwc4 2420206 2420205 2026-05-27T10:50:36Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance 2420206 wikitext text/x-wiki f0k4evpfvyn1bridjw95wz25fqjsmqm 2420207 2420206 2026-05-27T10:52:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage 2420207 wikitext text/x-wiki n6twlr8yz0mb6jfxbtw9d8kj3jvva1z 2420208 2420207 2026-05-27T10:52:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T426633)', diff saved to https://phabricator.wikimedia.org/P93225 and previous config saved to /var/cache/conftool/dbconfig/20260527-105235-fceratto.json 2420208 wikitext text/x-wiki bbvwtnarb6m81gx247sb3t1f7a0rg0a 2420209 2420208 2026-05-27T10:56:38Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2420209 wikitext text/x-wiki da3dmu9v88k4ukir7nw0mbd879q5o9n 2420210 2420209 2026-05-27T10:57:54Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage 2420210 wikitext text/x-wiki 2f7zzoxpbs4c3u54np4xus4thb2xhhd 2420211 2420210 2026-05-27T10:58:48Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage 2420211 wikitext text/x-wiki dnrkz7j7208rncevzqgxy408po2xp70 2420212 2420211 2026-05-27T11:00:18Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2189', diff saved to https://phabricator.wikimedia.org/P93226 and previous config saved to /var/cache/conftool/dbconfig/20260527-110016-marostegui.json 2420212 wikitext text/x-wiki ob6xp66vv3pmvp1wp23vnbfmrr2gtox 2420213 2420212 2026-05-27T11:01:46Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage 2420213 wikitext text/x-wiki kzab9p15bvd2miyf5qoksf9c36huyhy 2420214 2420213 2026-05-27T11:01:51Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2420214 wikitext text/x-wiki tmbc42s288f0jg75ejz8xlrqs8ib5wx 2420215 2420214 2026-05-27T11:02:08Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary 2420215 wikitext text/x-wiki so9sw128q2vb2yhy4jkacp5bz1i9obf 2420217 2420215 2026-05-27T11:02:42Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2420217 wikitext text/x-wiki 8alfahwnye6i2mmevf7jdqutkqctwlt 2420218 2420217 2026-05-27T11:02:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93227 and previous config saved to /var/cache/conftool/dbconfig/20260527-110242-fceratto.json 2420218 wikitext text/x-wiki cvv7mqqohv5rmt5zwd8tpndeln3d7k0 2420219 2420218 2026-05-27T11:03:00Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2420219 wikitext text/x-wiki ccqf6126bgzssdpezygkfgj29iqidmx 2420220 2420219 2026-05-27T11:08:39Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2420220 wikitext text/x-wiki ozh4ke82cvg9pjgk8asoenrur7nx5pv 2420221 2420220 2026-05-27T11:08:43Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2420221 wikitext text/x-wiki fer2a40nyvdlu2whsd3zrkbibfql43u 2420222 2420221 2026-05-27T11:10:47Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2420222 wikitext text/x-wiki 4prnw4tox3ndnyedtw1hw20kyc8coop 2420223 2420222 2026-05-27T11:10:57Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2420223 wikitext text/x-wiki tli0zowg3hnm2lexk2jds78g2yzll2t 2420224 2420223 2026-05-27T11:12:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93229 and previous config saved to /var/cache/conftool/dbconfig/20260527-111250-fceratto.json 2420224 wikitext text/x-wiki 2ydfr91jpfagulwbaadc3uz37lymcj8 2420225 2420224 2026-05-27T11:15:32Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1193.eqiad.wmnet with OS trixie 2420225 wikitext text/x-wiki rbskwj9b1h0lzp62mcsjqyk9atxzsfe 2420226 2420225 2026-05-27T11:19:43Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS trixie 2420226 wikitext text/x-wiki gahs43duvjdrnyep0zm5zb675k18t8b 2420228 2420226 2026-05-27T11:23:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T426633)', diff saved to https://phabricator.wikimedia.org/P93230 and previous config saved to /var/cache/conftool/dbconfig/20260527-112257-fceratto.json 2420228 wikitext text/x-wiki p2jaflq2foc1k00gf8iwk07in65nuw4 2420229 2420228 2026-05-27T11:23:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance 2420229 wikitext text/x-wiki hzi3nw0eq8xwfl0tkiliy10z21wt1dw 2420230 2420229 2026-05-27T11:23:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2188 (T426633)', diff saved to https://phabricator.wikimedia.org/P93231 and previous config saved to /var/cache/conftool/dbconfig/20260527-112327-fceratto.json 2420230 wikitext text/x-wiki mjh5wvk83qocd4gkec6d5khqt7fxye0 2420231 2420230 2026-05-27T11:24:21Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1193: Migration of db1193.eqiad.wmnet completed 2420231 wikitext text/x-wiki g2keh3skpb3wepw47t3depdjun0nga0 2420232 2420231 2026-05-27T11:29:19Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2165: Migration of db2165.codfw.wmnet completed 2420232 wikitext text/x-wiki r90kzp4gfdhcl2xfxfr59jzqypuspdy 2420233 2420232 2026-05-27T11:31:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T426633)', diff saved to https://phabricator.wikimedia.org/P93235 and previous config saved to /var/cache/conftool/dbconfig/20260527-113142-fceratto.json 2420233 wikitext text/x-wiki qpsujt89rdk7atgfiupyv1vryxzxmqd 2420234 2420233 2026-05-27T11:41:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93237 and previous config saved to /var/cache/conftool/dbconfig/20260527-114149-fceratto.json 2420234 wikitext text/x-wiki 9pl7fhtdwbon5h0hqktkxu4s4ku7cxo 2420235 2420234 2026-05-27T11:52:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93239 and previous config saved to /var/cache/conftool/dbconfig/20260527-115157-fceratto.json 2420235 wikitext text/x-wiki bwthp68tj91g5soqm9rnbrjqtz8o6hz 2420236 2420235 2026-05-27T11:56:40Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2420236 wikitext text/x-wiki syjc79vt4y7jrcfolaxbk7k2y4o0vzq 2420237 2420236 2026-05-27T11:58:25Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" 2420237 wikitext text/x-wiki pgap049ofhxgqh9u5hixur0drg4x55r 2420238 2420237 2026-05-27T11:58:31Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" 2420238 wikitext text/x-wiki cr6lryj74oa024epnfd9tz23uc5sw0x 2420239 2420238 2026-05-27T11:58:45Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary 2420239 wikitext text/x-wiki 7xex92u3ip923bbg6dlj6xv40ninxga 0 17290 2419344 2419123 2026-05-26T13:15:00Z Stashbot 7414 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 981, 985, 987 2419344 wikitext text/x-wiki i9yp5sfq77ptlhdcxhq2jisybl5r6am 2419396 2419344 2026-05-26T14:07:08Z Stashbot 7414 hashar: Updated integration-quibble-* jobs in order to validate running tests with Supervisord not managing memcached # T397810 2419396 wikitext text/x-wiki 1mpidbo6ws0o7xdm4756usvvp35yc8y 2419438 2419396 2026-05-26T14:46:17Z Stashbot 7414 hashar: Updating Quibble Jenkins jobs to stop Supervisord from spawning Memcached # T397810 2419438 wikitext text/x-wiki eftwxxyoc1ycel1cpavjoqkqrkhx226 2419490 2419438 2026-05-26T15:26:00Z Stashbot 7414 James_F: Zuul: Add PHP 8.5 to a few missing PHP pipelines, oops 2419490 wikitext text/x-wiki 0cohy3gaxnnoypnmhm4wyugs88ju1vx 2420080 2419490 2026-05-27T08:25:40Z Stashbot 7414 codders: integration-castor06: sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ 2420080 wikitext text/x-wiki kn93g2em0zwjxhne01o3prbb7dnsuhu 2420168 2420080 2026-05-27T09:49:10Z Stashbot 7414 hashar: Updated helm-lint Jenkins job to use releng/helm-linter:0.8.0 image # T424824 2420168 wikitext text/x-wiki fm0ad9hr70zpkh6uxd0028akqtgdur5 0 24216 2419525 2371712 2026-05-26T15:56:46Z BKing (WMF) 29017 /* Depool all nodes in a specific datacenter */ details on how to verify a datacenter depool 2419525 wikitext text/x-wiki emq0ic77gul2tydssxex4r8gk3dxyt1 0 29021 2419351 2368669 2026-05-26T13:24:14Z Simon04 2094 Fix phab=aqs2.0 2419351 wikitext text/x-wiki 0inbrzxqg8xkpqlta0i71kxuionrq1a 498 30942 2419567 2418674 2026-05-26T16:30:56Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service} AND NOT P{F:kernelversion = 6.12.88}' 2419567 wikitext text/x-wiki ndxfqg4gdks8skc4uuydx2qo6n6o56r 2419568 2419567 2026-05-26T16:31:00Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service} AND NOT P{F:kernelversion = 6.12.88}' 2419568 wikitext text/x-wiki i480kiiwkv5tk00chduwpp4rpi369rs 2419734 2419568 2026-05-26T18:16:13Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2419734 wikitext text/x-wiki bhe0zrt1kp7vpxuv2zzkpubk4mdhmws 2419735 2419734 2026-05-26T18:16:42Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=97) 2419735 wikitext text/x-wiki nzywu5f5ohjd0mhqfzyhdx1w1azqn0l 2419743 2419735 2026-05-26T18:21:18Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2419743 wikitext text/x-wiki pv2r2nzsjf4g896hse8m461ea872f18 2419744 2419743 2026-05-26T18:21:58Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=97) 2419744 wikitext text/x-wiki rlactd748y3gbe197vtrqqcgjrk3jxf 2419745 2419744 2026-05-26T18:22:30Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2419745 wikitext text/x-wiki 8wvnrii9ias00b4ub56qjbwsb0fv8k0 2419758 2419745 2026-05-26T18:33:41Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) 2419758 wikitext text/x-wiki o7mjzbhim9pefgv13ayidwvys2dx6x8 2419768 2419758 2026-05-26T18:46:09Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2419768 wikitext text/x-wiki pvv12u4ywb81s5or2167cipw8qypjp6 2419770 2419768 2026-05-26T18:47:36Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=97) 2419770 wikitext text/x-wiki fg7eika59k654k1dpbvdr92pgpmxvuh 2419771 2419770 2026-05-26T18:50:53Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds 2419771 wikitext text/x-wiki 552wl487j23rera6nl3pllmojng1u1b 2419780 2419771 2026-05-26T19:11:42Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) 2419780 wikitext text/x-wiki nv1nqpda674y0q0x20oriw91v5ax260 0 440900 2420227 2416675 2026-05-27T11:20:38Z Taavi-WMF 41365 /* cloudservicesXXXX */ use batch cookbook 2420227 wikitext text/x-wiki jcw74e9xx7qhkmprgxqlf4qz68bj1sz 498 443949 2419689 2413369 2026-05-26T17:31:32Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy d86cbb7 (Add/Expand module-level docstrings) 2419689 wikitext text/x-wiki shmm68plxl4mdjy0e6udn7mvo9xhhna 2419690 2419689 2026-05-26T17:31:33Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 55c2fbe (Reorganize slightly README and CONTRIBUTING.md files) 2419690 wikitext text/x-wiki 94ps33da82e7y2g9yq3guc2z9mwducp 2419691 2419690 2026-05-26T17:31:34Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 18b6a34 (Fix test_index_page unit test) 2419691 wikitext text/x-wiki er4o72xs1qwm7ebdaf3oddpyqvs11us 0 446705 2420169 2418697 2026-05-27T09:54:23Z BJensen-WMF 48942 /* Packaging */ add instructions for checking out operations/debs/kubernetes 2420169 wikitext text/x-wiki avrc5dbvurw9ujxvpgmho30mgurvk7p 2420172 2420169 2026-05-27T10:04:03Z BJensen-WMF 48942 s/bullseye/bookworm/ 2420172 wikitext text/x-wiki b2jodr0ke2bq1s425b0c6ulvlauvqr1 2420185 2420172 2026-05-27T10:14:50Z BJensen-WMF 48942 /* Publishing */ clarify pbuilder-result alias 2420185 wikitext text/x-wiki b2lmh6yk2ijgbp3gpf7dssoweo7zx6d 2420186 2420185 2026-05-27T10:17:47Z BJensen-WMF 48942 /* Publishing */ update apt host 2420186 wikitext text/x-wiki mkdkrwn0ot1wl58s8cbsa3gs501task 2420189 2420186 2026-05-27T10:23:12Z BJensen-WMF 48942 /* Publishing */ add a pointer to apt components 2420189 wikitext text/x-wiki ep7mwy6exa5nisdhde0wax3ezey5nmm 0 446795 2420071 2415425 2026-05-27T08:08:47Z FCeratto-WMF 39104 Add decommissioning cookbook 2420071 wikitext text/x-wiki 5yzggrkjhhsplyamt36oaqecvop53tl 0 449160 2419405 2419267 2026-05-26T14:10:29Z Dexbot 30554 Bot: Updating the report 2419405 wikitext text/x-wiki cvz92vyuqyjt4iuvqf7h9a5hbkx97u5 2419724 2419405 2026-05-26T18:05:05Z Dexbot 30554 Bot: Updating the report 2419724 wikitext text/x-wiki 87q5pit6wejrvl8x0kgbfr0ykjoci6y 2419891 2419724 2026-05-26T21:53:40Z Dexbot 30554 Bot: Updating the report 2419891 wikitext text/x-wiki a526ylf1siekssemlvivl0blfzindug 2419940 2419891 2026-05-27T00:03:00Z Dexbot 30554 Bot: Updating the report 2419940 wikitext text/x-wiki cntyejmux9ukqfqxit5amyfkmxz6zis 2419945 2419940 2026-05-27T00:24:04Z Dexbot 30554 Bot: Updating the report 2419945 wikitext text/x-wiki sqset1w5uztn8eczdmjy5ee4buaba5d 2420076 2419945 2026-05-27T08:13:15Z Dexbot 30554 Bot: Updating the report 2420076 wikitext text/x-wiki cxnk7a8g8goxh0yp2yabwijqkuv6tct 0 451190 2419914 2387048 2026-05-26T22:44:07Z EMcnaughton (WMF) 6074 /* Intact */ spelling 2419914 wikitext text/x-wiki 4t3rv2826tyezi55n7qd3tmhegbxmfz 1 451462 2419387 2029266 2026-05-26T13:58:00Z Peter F. Patel-Schneider 42591 /* legacy service? */ new section 2419387 wikitext text/x-wiki aos92078new7unlq1sc0f8821mr2ylu 0 451490 2419779 2405353 2026-05-26T19:11:40Z LSandergreen-WMF 46345 /* Thank you , monthly convert, recurring failure notification and end of year email text & translations */ add data retention policy 2419779 wikitext text/x-wiki bm9h0hyg6ahtthw9yz6eov5vg2h52ha 0 451533 2419431 2411111 2026-05-26T14:34:16Z Filippo Giunchedi 2939 /* Next meeting agenda */ 2419431 wikitext text/x-wiki 5u8n4wfbehe3ynz1j47ubxc1gcrkn3j 2419507 2419431 2026-05-26T15:33:53Z FNegri-WMF 32595 /* Next meeting agenda */ add next meeting date and clear agenda 2419507 wikitext text/x-wiki qfhl7ww8m2r6v3dpfup9coykhbuldnz 2419510 2419507 2026-05-26T15:34:15Z FNegri-WMF 32595 /* Next meeting agenda */ properly clear agenda :) 2419510 wikitext text/x-wiki oemxe1xvmfq1pu8nhjmdjtojel4g2ht 0 452340 2420196 2394593 2026-05-27T10:35:04Z CGoubert-WMF 32775 Add REST Gateway diagram 2420196 wikitext text/x-wiki t8hdzm8zqrbuoigql3dcq9otgfdide1 116 453906 2419302 2418925 2026-05-26T12:09:16Z Gitlabaccountapprovalbot 37332 charisad was rejected. 2419302 wikitext text/x-wiki bq985runie4dbif9unmu1ae3nxq0ljz 0 455928 2419912 2409101 2026-05-26T22:42:51Z SFaci-WMF 35637 /* Create an instrument */ 2419912 wikitext text/x-wiki 58kbwafo0iclyv44pcw34j8oyio1i0z 2419913 2419912 2026-05-26T22:42:59Z SFaci-WMF 35637 /* Send a click event */ 2419913 wikitext text/x-wiki dghxzf47tv3ddf9aii08sxkeqpo3411 2419915 2419913 2026-05-26T22:45:10Z SFaci-WMF 35637 /* mw.testKitchen.getExperiment */ 2419915 wikitext text/x-wiki 980o2xpallhuih7o9tezqp9dqla8fjj 0 455989 2419925 2418383 2026-05-26T22:56:32Z SFaci-WMF 35637 /* Experiment design: user traffic per wiki */ 2419925 wikitext text/x-wiki 652wfy7ncdbc2xpf6nvdclqmipcshq6 0 456039 2419922 2409102 2026-05-26T22:48:13Z SFaci-WMF 35637 /* Client-side instrumentation */ 2419922 wikitext text/x-wiki kqtxl762bmqw8nghvgbxeop7owoo5z5 2419923 2419922 2026-05-26T22:49:04Z SFaci-WMF 35637 /* Client-side instrumentation */ 2419923 wikitext text/x-wiki 52oc4qtc4abno590yz0d5wcazzqr1lf 0 457961 2419686 2413524 2026-05-26T17:30:00Z HNowlan (WMF) 15760 /* Access control */ 2419686 wikitext text/x-wiki 3d97v2sipw78dwg7xsbk807rc780f4t 0 458789 2419919 2381449 2026-05-26T22:46:18Z SFaci-WMF 35637 /* Instrumentation */ 2419919 wikitext text/x-wiki 82rhs4l6yiv78xiep7opljf4dsvcp16 2419920 2419919 2026-05-26T22:47:36Z SFaci-WMF 35637 /* Analytics instrumentation */ 2419920 wikitext text/x-wiki czgrji2tsgnhnr9zvq28se3gq01le5u 2419921 2419920 2026-05-26T22:47:46Z SFaci-WMF 35637 /* Experiment instrumentation: analytics */ 2419921 wikitext text/x-wiki k5358hh0wb9glbhz0xisyp501fsuswo 0 459138 2419339 2417227 2026-05-26T13:07:52Z KSoučková-WMF 35993 /* Contact information */ remove ServiceOps 2419339 wikitext text/x-wiki t6gulq6tn2zxlabtxepz79ajio4jikt 0 459903 2419439 2388887 2026-05-26T14:46:39Z JMeybohm (WMF) 16709 2419439 wikitext text/x-wiki cssa7npciy7dqxzt0phnov6nxy0febt 0 459948 2420126 2419185 2026-05-27T08:58:47Z LToscano (WMF) 7191 /* Kubernetes API (control plane) */ 2420126 wikitext text/x-wiki 4thrhyp6kf992tga1ft28uiddx6wpm4 2420127 2420126 2026-05-27T08:58:59Z LToscano (WMF) 7191 /* Kubernetes API (control plane) */ 2420127 wikitext text/x-wiki lc9hqwkrk2ej1iow2s37q7cb9iuu16o 0 459966 2419916 2406229 2026-05-26T22:45:44Z SFaci-WMF 35637 /* Scenario: Footer */ 2419916 wikitext text/x-wiki n6rgfhj0kt406fjz65qvhszrcx4khk2 2419917 2419916 2026-05-26T22:46:00Z SFaci-WMF 35637 /* Scenario: Special page */ 2419917 wikitext text/x-wiki 63ltriip7zqgrb0ub9ykoldv9xr40ao 0 459993 2419517 2396903 2026-05-26T15:52:12Z APaskulin (WMF) 14880 Fix typos 2419517 wikitext text/x-wiki gm5qwhe4a9ceua3xd9sdbz7twjvdkcb 116 460070 2419883 2418618 2026-05-26T21:47:09Z EtherpadBackupBot 54504 South_Asia_Community_Call archived by Gnoeee 2419883 wikitext text/x-wiki 07hni10a8g4v8hp43r6tt86nrg14c7i 2419884 2419883 2026-05-26T21:47:24Z EtherpadBackupBot 54504 Wikimedia_Hackathon_2026_Closing_Showcase archived by Gnoeee 2419884 wikitext text/x-wiki knejpnftqttuc5o72zzxo7zzyysyy92 498 460126 2419480 2412206 2026-05-26T15:12:29Z Stashbot 7414 wmbot~gergesshamon@tools-bastion-15: Restarting buildservice 2419480 wikitext text/x-wiki qw1skdxbdcsfuefcuj5d0iikas4mrjv 3 460219 2419375 2026-05-26T13:43:17Z StrikerBot 8475 Welcome to Toolforge! 2419375 wikitext text/x-wiki lx6e13y6hrgjyawqs0nhfhf1fob5piy 3 460220 2419376 2026-05-26T13:43:52Z StrikerBot 8475 Welcome to Toolforge! 2419376 wikitext text/x-wiki 34p8wfm05pc0tw28updljcdc0gsmf6m 3 460221 2419377 2026-05-26T13:44:08Z StrikerBot 8475 Welcome to Toolforge! 2419377 wikitext text/x-wiki gcyncsjtcl9ixatud175fnumo60x93u 0 460222 2419421 2026-05-26T14:21:33Z AWesterinen-WMF 54189 Created page with "This document describes a means to categorize SPARQL queries sent to the Wikidata eqiad and codfw servers. It is intended to serve as a multi-dimensional framework for both system testing and performance evaluation. Following the methodology described by Zhang, et al. [1], queries are categorized by their SPARQL algebra [2], BGPs (graph-pattern shape and join structure), and other, “hybrid” features (for example, combining specific algebraic and pattern features, o..." 2419421 wikitext text/x-wiki 3om6kua57hq65okubgug5l1st097a8v 2419688 2419421 2026-05-26T17:31:14Z AWesterinen-WMF 54189 2419688 wikitext text/x-wiki io2kend021q5bhfpwfzd9j1d5ui84fc 2419710 2419688 2026-05-26T17:46:15Z AWesterinen-WMF 54189 2419710 wikitext text/x-wiki 55bn1bm565eh9bag61jpvwydh3h1qsm 2419711 2419710 2026-05-26T17:50:28Z AWesterinen-WMF 54189 /* SPARQL Classification Architecture */ 2419711 wikitext text/x-wiki ryawyf6uvc86brrmcugc3qxt6wfjv8n 2419717 2419711 2026-05-26T17:55:46Z AWesterinen-WMF 54189 2419717 wikitext text/x-wiki h8caicvc033rfdejyvhst2ffe0jwduu 2419723 2419717 2026-05-26T18:04:19Z AWesterinen-WMF 54189 2419723 wikitext text/x-wiki e8yxk9twdhcblnp6vi5bxnzxcjx5qvo 2419727 2419723 2026-05-26T18:09:42Z AWesterinen-WMF 54189 2419727 wikitext text/x-wiki oeyt92o7yer9a2f32cvihwat5q9gmf9 2419728 2419727 2026-05-26T18:10:11Z AWesterinen-WMF 54189 /* SPARQL Classification Architecture */ 2419728 wikitext text/x-wiki icw5vwpcb0mm3ekocvrwjah3v6fyz5z 2419729 2419728 2026-05-26T18:12:15Z AWesterinen-WMF 54189 2419729 wikitext text/x-wiki trxhfqjyrbccdiju97a15ljra5nsjg3 2419736 2419729 2026-05-26T18:18:58Z AWesterinen-WMF 54189 2419736 wikitext text/x-wiki l7bzpmd6nr3o8u281ff1t60mhakxzy5 2419738 2419736 2026-05-26T18:19:51Z AWesterinen-WMF 54189 2419738 wikitext text/x-wiki pkrgrkui6wq5iu9ofuohe2arzjm1go0 2419746 2419738 2026-05-26T18:22:32Z AWesterinen-WMF 54189 2419746 wikitext text/x-wiki q976uyspfhy2ot64sjjngq828y2bbuf 2419748 2419746 2026-05-26T18:24:46Z AWesterinen-WMF 54189 /* Executive Summary */ 2419748 wikitext text/x-wiki jv6fv5r5p561exf8ph9yh1tlgj0ucze 2419749 2419748 2026-05-26T18:25:41Z AWesterinen-WMF 54189 /* SPARQL Classification Features */ 2419749 wikitext text/x-wiki sabhx2f1hnlbtgv8ivxcwyyg31uwp1k 2419750 2419749 2026-05-26T18:27:28Z AWesterinen-WMF 54189 /* SPARQL Classification Features */ 2419750 wikitext text/x-wiki 8ig5r53proni5u4961te5gtx01bq5w0 2419751 2419750 2026-05-26T18:28:53Z AWesterinen-WMF 54189 /* SPARQL Classification Features */ 2419751 wikitext text/x-wiki eujyu1rhgx7mnakompuxvg9haravk8u 2419757 2419751 2026-05-26T18:31:18Z AWesterinen-WMF 54189 /* SPARQL Classification Features */ 2419757 wikitext text/x-wiki dnkuupxj2yoojb0si06rb1c40mj7neg 2419759 2419757 2026-05-26T18:34:35Z AWesterinen-WMF 54189 /* Estimating SPARQL “Cost” */ 2419759 wikitext text/x-wiki 046q8aty4qpa3ss7ykyp4e3voo5ijcw 0 460223 2419429 2026-05-26T14:30:13Z AWesterinen-WMF 54189 Created page with "This document outlines the testing strategy to validate the Wikidata infrastructure as part of the migration from Blazegraph to QLever. The work is defined in [https://phabricator.wikimedia.org/T422097 Phabricator ticket T422097]. The goal is to define the technical components required to validate data correctness, evaluate performance, and categorize output comparisons across the two environments. == Executive Summary & Overall Architecture == Migration testing is cri..." 2419429 wikitext text/x-wiki 32nfqzppczjor7ngteflsisqwtkp80i 2419466 2419429 2026-05-26T15:02:34Z AWesterinen-WMF 54189 /* Pre-Requisite Assumptions and Analyses */ 2419466 wikitext text/x-wiki 3lvyw3qch1uhqo6occaay5ye591j7i7 2419469 2419466 2026-05-26T15:04:09Z AWesterinen-WMF 54189 2419469 wikitext text/x-wiki fs4c0hsu3tbxgaxdxgr8e753an6zcrv 2419472 2419469 2026-05-26T15:05:34Z AWesterinen-WMF 54189 /* Pre-Requisites */ 2419472 wikitext text/x-wiki t3hk5ochxpymvscbn5qkz0zoz0n188c 2419476 2419472 2026-05-26T15:08:34Z AWesterinen-WMF 54189 /* Pre-Requisites */ 2419476 wikitext text/x-wiki eyrdqma08ud4wttubx00f3wu4k91ili 2419509 2419476 2026-05-26T15:34:07Z AWesterinen-WMF 54189 2419509 wikitext text/x-wiki 8amdsqdbshvqzrsk592go4zpnyllqvk 2419511 2419509 2026-05-26T15:35:03Z AWesterinen-WMF 54189 /* Pre-Requisites */ 2419511 wikitext text/x-wiki fbvwvk5wyheg6j47qo9osoxvac2bwyp 2419514 2419511 2026-05-26T15:38:51Z AWesterinen-WMF 54189 /* Software Development Tasks */ 2419514 wikitext text/x-wiki 7khgcdkbjdniqo7qw5camzfcq95zjzh 2419530 2419514 2026-05-26T15:57:55Z AWesterinen-WMF 54189 /* Executive Summary & Overall Architecture */ 2419530 wikitext text/x-wiki t1z3b5lgxyi99mz5fgdh1looxkdbs0w 2419531 2419530 2026-05-26T15:59:06Z AWesterinen-WMF 54189 /* Executive Summary & Overall Architecture */ 2419531 wikitext text/x-wiki rqrsdyfzc7wehwr460oev6dbl93m8i5 2419532 2419531 2026-05-26T15:59:33Z AWesterinen-WMF 54189 /* Prerequisites */ 2419532 wikitext text/x-wiki 8317396v3pmojpx9cd7ilgametmpo2u 2419533 2419532 2026-05-26T16:00:09Z AWesterinen-WMF 54189 /* Prerequisites */ 2419533 wikitext text/x-wiki 49zmcwfbgd8b8qpv2ork2itefxi2orc 2419538 2419533 2026-05-26T16:01:18Z AWesterinen-WMF 54189 /* Prerequisites */ 2419538 wikitext text/x-wiki 6u8ax2v2sto38x3jcigt8lcuqaq7pkv 2419539 2419538 2026-05-26T16:02:25Z AWesterinen-WMF 54189 /* Software Development Tasks */ 2419539 wikitext text/x-wiki cb862l88wtzfekre05bwe1mctc79bpx 2419541 2419539 2026-05-26T16:02:54Z AWesterinen-WMF 54189 /* Software Development Tasks */ 2419541 wikitext text/x-wiki rjodkqaz09t0dywbrkxmj03ldvn1vmx 2419550 2419541 2026-05-26T16:05:15Z AWesterinen-WMF 54189 /* Software Development Tasks */ 2419550 wikitext text/x-wiki mvf8muz435vuiij36u8g3jaz3mgsd9b 2419555 2419550 2026-05-26T16:06:41Z AWesterinen-WMF 54189 /* Software Development Tasks */ 2419555 wikitext text/x-wiki 6x0shdo9j2sh56p53f3mtg2mj8811h5 2419557 2419555 2026-05-26T16:09:13Z AWesterinen-WMF 54189 /* Software Development Tasks */ 2419557 wikitext text/x-wiki 9o1220wv3ws9939qs5owx1tfu8zkgiv 0 460224 2419512 2026-05-26T15:35:19Z FNegri-WMF 32595 Archive meeting notes from Google Doc 2419512 wikitext text/x-wiki 50mja1o61u8ugen1zo1bc461hvtawgc 2419513 2419512 2026-05-26T15:36:46Z FNegri-WMF 32595 Improve formatting 2419513 wikitext text/x-wiki oizaxce8n6miyq4zodl1oazgg9jr8i0 1 460225 2420003 2026-05-27T05:53:57Z KartikMistry 1115 /* Signed-By: */ new section 2420003 wikitext text/x-wiki fbupk0xt7c7nl890suazlfrayz58ojn 2420004 2420003 2026-05-27T05:54:20Z KartikMistry 1115 ~ 2420004 wikitext text/x-wiki c51i75f2nxgkhh03ijj8l6iyyqa2a6p