Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.24 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2470811 2468866 2026-10-19T15:43:55Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf 2470811 wikitext text/x-wiki rb1f82lpg5sjg3d2gk9cpzrwymt7epk 2470812 2470811 2026-10-19T15:44:22Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf 2470812 wikitext text/x-wiki 0vb1o5px5pp14i1fxf1q65lm5p3kskv 0 4108 2470670 2470245 2026-10-19T14:35:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1356887]] to Monday, October 19 UTC late backport window 2470670 wikitext text/x-wiki ma7iofqnq13f6eoan00j8rewjun96ci 2470671 2470670 2026-10-19T14:35:58Z ScheduleDeploymentBot 37566 Add [[gerrit:1356885]] to Monday, October 19 UTC late backport window 2470671 wikitext text/x-wiki 2pw55ezfahuu5f232txdbs9bmuxrsbs 2470857 2470671 2026-10-19T16:55:34Z ScheduleDeploymentBot 37566 Add [[gerrit:1356949]] to Monday, October 19 UTC late backport window 2470857 wikitext text/x-wiki gmew53sf4rssr7fmcsi5scrgslvypkp 2470917 2470857 2026-10-19T20:02:31Z Krinkle 44 /* Week of October 19 */ done earlier today already 2470917 wikitext text/x-wiki hjqu31cynanlg3cc4acyksurx9nwxdq 498 6726 2470803 2468865 2026-10-19T15:39:59Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf 2470803 wikitext text/x-wiki djjm15quqw5oxpw2hcbm2u278apqrie 2470805 2470803 2026-10-19T15:41:52Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf 2470805 wikitext text/x-wiki 198amjmdcjo7ndlccejk054bl9vslrf 0 7797 2470804 2337450 2026-10-19T15:40:12Z Neil Shah-Quinn (WMF) 36601 Add info about not upgrading to Hadoop 3. Reorganize so user documentation is at the top. Copyedit 2470804 wikitext text/x-wiki kp643e5pg22cnl3pac38sz09ul7ap8k 0 7919 2470344 2470342 2026-10-19T11:59:21Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2006-dev.codfw.wmnet 2470344 wikitext text/x-wiki gbaiockxkysp7n9jiidtkw5dv1walul 2470345 2470344 2026-10-19T11:59:24Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcontrol2005-dev.codfw.wmnet 2470345 wikitext text/x-wiki krxgw8faspooqimivtdqkacy2gx4ii3 2470346 2470345 2026-10-19T12:00:05Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync 2470346 wikitext text/x-wiki 2fmtwvyy4rpi60rufdma6746xbzn014 2470347 2470346 2026-10-19T12:00:08Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync 2470347 wikitext text/x-wiki jmlostmpephm3mwgwa5q9tclukaha07 2470348 2470347 2026-10-19T12:00:12Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync 2470348 wikitext text/x-wiki fw72feobxahfipnruss4l4f5q0ws42u 2470349 2470348 2026-10-19T12:00:20Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync 2470349 wikitext text/x-wiki 9eme6zcn1on8yonzz8izo81ybnjat94 2470350 2470349 2026-10-19T12:00:26Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4003.ulsfo.wmnet 2470350 wikitext text/x-wiki qor6oui5ld350pgfi5wdqjptbmp77j9 2470351 2470350 2026-10-19T12:05:59Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2005-dev.codfw.wmnet 2470351 wikitext text/x-wiki 5zae4b8bmi5izoaykw4uy6vvbi68ur6 2470352 2470351 2026-10-19T12:06:03Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1002-dev.eqiad.wmnet 2470352 wikitext text/x-wiki nhybassla1ee3g1fjr0t5smsrzq7r5t 2470353 2470352 2026-10-19T12:07:20Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2470353 wikitext text/x-wiki 42c825b2m5sy78g2wzkorgqbzc9rcwk 2470354 2470353 2026-10-19T12:07:40Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2470354 wikitext text/x-wiki ewj3uejd8fyaw3wrc1wj7qeuro40cwx 2470355 2470354 2026-10-19T12:07:43Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid2003.codfw.wmnet 2470355 wikitext text/x-wiki drrsp0lpj87zj0cb7fc5iofvwcd02es 2470356 2470355 2026-10-19T12:08:18Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus5003.eqsin.wmnet 2470356 wikitext text/x-wiki quhtmf0om6rebw8wi60khxk1jz4hlbh 2470357 2470356 2026-10-19T12:09:52Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet 2470357 wikitext text/x-wiki li44ut767xtv1k7oomutu60hmx1bo0i 2470358 2470357 2026-10-19T12:09:55Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet 2470358 wikitext text/x-wiki pmwrqxcwd88lbeh13iqps3x1q29sx9w 2470359 2470358 2026-10-19T12:10:01Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2005-dev.codfw.wmnet 2470359 wikitext text/x-wiki oko3el9i77iddyrztn6fshft068a7ya 2470360 2470359 2026-10-19T12:10:20Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P96800 and previous config saved to /var/cache/conftool/dbconfig/20261019-121018-marostegui.json 2470360 wikitext text/x-wiki 4qt586hbvwuw2x7gpmhyx8m11arwmkc 2470361 2470360 2026-10-19T12:10:28Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7004.magru.wmnet 2470361 wikitext text/x-wiki bi8wvv6fndlkt0sxz7qv7r6x10gz5w1 2470362 2470361 2026-10-19T12:11:24Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet 2470362 wikitext text/x-wiki pyr5n5yd4j7lgi0xmjpnv1o1xrp0rkv 2470363 2470362 2026-10-19T12:11:32Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2003.codfw.wmnet 2470363 wikitext text/x-wiki i7i89y2mm73p6m92aaclfz0vuh0dosx 2470364 2470363 2026-10-19T12:11:46Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2470364 wikitext text/x-wiki kmvtqerx9eck47dhdcb415e1djkkdx8 2470365 2470364 2026-10-19T12:12:01Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid1003.eqiad.wmnet 2470365 wikitext text/x-wiki 6189jkbo4onesglsjbrqr896zr1valc 2470366 2470365 2026-10-19T12:12:21Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2470366 wikitext text/x-wiki gmtarm7yl1hr82m93oi6hzmia9l01mm 2470367 2470366 2026-10-19T12:13:52Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet 2470367 wikitext text/x-wiki k1u3mjzyxl662sbdug88817in1k5rsz 2470368 2470367 2026-10-19T12:14:33Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5003.eqsin.wmnet 2470368 wikitext text/x-wiki 9rqcnwbvfiv5rj2ihi7ll33t6a408qb 2470369 2470368 2026-10-19T12:15:36Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet 2470369 wikitext text/x-wiki 0z1op1zh50e12mye795cwujng00vnay 2470370 2470369 2026-10-19T12:15:56Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1003.eqiad.wmnet 2470370 wikitext text/x-wiki s5y2kupqo77cy5gu0rkurueqk4sr8ls 2470371 2470370 2026-10-19T12:16:07Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet 2470371 wikitext text/x-wiki g8cnut7s2yfsfqj1n1yw37rtxj4q1wr 2470372 2470371 2026-10-19T12:16:10Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2006-dev.codfw.wmnet 2470372 wikitext text/x-wiki 9el4d42veihxw8h0q414a3mpprckodo 2470373 2470372 2026-10-19T12:16:36Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus6002.drmrs.wmnet 2470373 wikitext text/x-wiki 5k8mabkg0f5y8pfdxkbsqju2kvxnbri 2470374 2470373 2026-10-19T12:17:51Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7004.magru.wmnet 2470374 wikitext text/x-wiki 5025dj8lo7i7u7efo0jqarpeks8wa44 2470375 2470374 2026-10-19T12:18:46Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet 2470375 wikitext text/x-wiki kof3xk8x3n7gkvcl6ohw4k7yyhh2c9r 2470376 2470375 2026-10-19T12:19:51Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2470376 wikitext text/x-wiki qqm6bte65mifpxqbaom1ets6s0ojzg6 2470377 2470376 2026-10-19T12:20:17Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2470377 wikitext text/x-wiki m5f5hwbtycx3gvxg346bpnej61vq3s1 2470378 2470377 2026-10-19T12:22:12Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet 2470378 wikitext text/x-wiki hsqeks99hda5botce2k9e61sl1xkh9v 2470379 2470378 2026-10-19T12:22:15Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephmon2007-dev.codfw.wmnet 2470379 wikitext text/x-wiki t27fl1knujlkhdrv2hx5wsfmwhxifms 2470380 2470379 2026-10-19T12:22:40Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6002.drmrs.wmnet 2470380 wikitext text/x-wiki adj6quxnoeqc6r8sckj418h1utcekkm 2470381 2470380 2026-10-19T12:22:51Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet 2470381 wikitext text/x-wiki rzmx5v0kkfvkicylkztmze33010m995 2470382 2470381 2026-10-19T12:25:11Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1003.eqiad.wmnet 2470382 wikitext text/x-wiki 7v98pb0d1v1i7beopo6l1kvxjwz0dqt 2470383 2470382 2026-10-19T12:25:22Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P96801 and previous config saved to /var/cache/conftool/dbconfig/20261019-122520-marostegui.json 2470383 wikitext text/x-wiki h63z3pzqegcz67idw6kz27l4mx55s6f 2470384 2470383 2026-10-19T12:25:26Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply 2470384 wikitext text/x-wiki sk8jzogqpdwxpvscc5psab3xvdo73pl 2470385 2470384 2026-10-19T12:26:03Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply 2470385 wikitext text/x-wiki 5rch12mj9e5aukscvozdx4w63xu6c4i 2470386 2470385 2026-10-19T12:26:05Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow5003.eqsin.wmnet 2470386 wikitext text/x-wiki i424iy9bht1caxf75fvqb7oi4la7yaj 2470387 2470386 2026-10-19T12:26:07Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7004.magru.wmnet 2470387 wikitext text/x-wiki dows5e7m2s0a1oaxbjdnbynnv3fu0vg 2470388 2470387 2026-10-19T12:26:11Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7004.magru.wmnet 2470388 wikitext text/x-wiki r5db2zann7gbidetqnboaiiu4sc74cg 2470389 2470388 2026-10-19T12:28:09Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7003.magru.wmnet 2470389 wikitext text/x-wiki rth1f8v26e34oj9u71rhfw0o0htaig7 2470390 2470389 2026-10-19T12:28:34Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2007-dev.codfw.wmnet 2470390 wikitext text/x-wiki 499zfpz93d8nvrzu8phg67372xwjesi 2470391 2470390 2026-10-19T12:28:37Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2004-dev.codfw.wmnet 2470391 wikitext text/x-wiki 5jmbue0tao1tex4v73mir1rxcuv3n8h 2470392 2470391 2026-10-19T12:29:38Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus7002.magru.wmnet 2470392 wikitext text/x-wiki rwf35g14symyh1ktiju8gq2v666jmu0 2470393 2470392 2026-10-19T12:30:31Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5003.eqsin.wmnet 2470393 wikitext text/x-wiki kzle4vs2jnlz9epksm5jtvuf8lu75qe 2470394 2470393 2026-10-19T12:31:59Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7003.magru.wmnet 2470394 wikitext text/x-wiki bb8ueyqbzyl2ixe2d8gevqmidpg8n6v 2470395 2470394 2026-10-19T12:32:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet 2470395 wikitext text/x-wiki q93h6vttwylw4xhytjjqbmh3yyarbaw 2470396 2470395 2026-10-19T12:32:47Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host etherpad1004.eqiad.wmnet 2470396 wikitext text/x-wiki tn5dtmgdy25qlz8luugwwacztambshx 2470397 2470396 2026-10-19T12:32:50Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2470397 wikitext text/x-wiki tvu08poignx128eyetewz836rbmfvdk 2470398 2470397 2026-10-19T12:32:52Z Stashbot 7414 ayounsi@cumin1004: END (ERROR) - Cookbook sre.netbox.update-extras (exit_code=97) rolling restart_daemons on A:netbox-canary 2470398 wikitext text/x-wiki b2hvec4pcy3hkt49j9wh3hgv2s4mzn8 2470399 2470398 2026-10-19T12:33:01Z Stashbot 7414 arnaudb@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host etherpad1004.eqiad.wmnet 2470399 wikitext text/x-wiki 4irskonqesp98y3mwudzihbx3m8ud4g 2470400 2470399 2026-10-19T12:33:03Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2470400 wikitext text/x-wiki oupihsbtecpbb03vqlz1z3s4m6feylv 2470401 2470400 2026-10-19T12:33:22Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host etherpad1004.eqiad.wmnet 2470401 wikitext text/x-wiki s71qs48oxu44b47ilhhtdtln45imyel 2470402 2470401 2026-10-19T12:33:42Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idm-test1001.wikimedia.org 2470402 wikitext text/x-wiki rqx1mo7emavx8p15nc985b6n9flu0qh 2470403 2470402 2026-10-19T12:33:57Z Stashbot 7414 slyngshede@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host idm-test1001.wikimedia.org 2470403 wikitext text/x-wiki nc4h0e3ymr2j7wur0ieb3hjb7mcxwax 2470404 2470403 2026-10-19T12:34:17Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1003.eqiad.wmnet 2470404 wikitext text/x-wiki 7xdla186ehg5tolljt9mga9ttvy1erq 2470405 2470404 2026-10-19T12:34:20Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1004.eqiad.wmnet 2470405 wikitext text/x-wiki nc7z37u35rct2j8n8d2gbt1vzizwwiy 2470406 2470405 2026-10-19T12:34:40Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary 2470406 wikitext text/x-wiki 50nnsmstgved2zarizd7yb1h8qeb9sx 2470407 2470406 2026-10-19T12:34:55Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox 2470407 wikitext text/x-wiki edwhgocdoow6bf84ar2e1jo22agifxm 2470408 2470407 2026-10-19T12:35:24Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox 2470408 wikitext text/x-wiki bagyi6tfwfvifeq7evv4gg9gc9wphtu 2470409 2470408 2026-10-19T12:35:44Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus7002.magru.wmnet 2470409 wikitext text/x-wiki j2bngrta6qlj4fcx1wn2wcyap3wqqda 2470410 2470409 2026-10-19T12:36:12Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idm-test1001.wikimedia.org 2470410 wikitext text/x-wiki ksh0pi8hz0cophgqegmmd51i4oku70q 2470411 2470410 2026-10-19T12:36:22Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet 2470411 wikitext text/x-wiki q5ociit58cwxv1h01dhjy84upule0je 2470412 2470411 2026-10-19T12:36:24Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet 2470412 wikitext text/x-wiki de41y31etewjguu7stt6qemjovmxv4a 2470413 2470412 2026-10-19T12:36:26Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2005-dev.codfw.wmnet 2470413 wikitext text/x-wiki jz1o70jvwjotjv6hsj87pgxuznv30cv 2470414 2470413 2026-10-19T12:37:10Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad1004.eqiad.wmnet 2470414 wikitext text/x-wiki 32ezygod6408lnyoea0ynpvchj0f93q 2470415 2470414 2026-10-19T12:37:14Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host etherpad2002.codfw.wmnet 2470415 wikitext text/x-wiki pil79d1uutp1nug01t03vfasshgh241 2470416 2470415 2026-10-19T12:38:54Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus1005.eqiad.wmnet 2470416 wikitext text/x-wiki 6ep2pogtqy7lnusorxamztb4q7y0pq5 2470417 2470416 2026-10-19T12:39:05Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus1007.eqiad.wmnet 2470417 wikitext text/x-wiki pg13oidutq3equa7kviytuosos5m0v5 2470418 2470417 2026-10-19T12:39:18Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus2005.codfw.wmnet 2470418 wikitext text/x-wiki pfytgreriv43id2mo3wvbkx6ugfurdm 2470419 2470418 2026-10-19T12:39:27Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus2007.codfw.wmnet 2470419 wikitext text/x-wiki kt9wqivog0ni9ixmf1nk6pt7uhoa8is 2470420 2470419 2026-10-19T12:39:57Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm-test1001.wikimedia.org 2470420 wikitext text/x-wiki kshxmye6gmuz1tz08ilgr1huqldhhgp 2470421 2470420 2026-10-19T12:40:13Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7003.magru.wmnet 2470421 wikitext text/x-wiki 5gbvr62ub36gh5ylylne21mf6j4xh7s 2470422 2470421 2026-10-19T12:40:25Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T441197)', diff saved to https://phabricator.wikimedia.org/P96802 and previous config saved to /var/cache/conftool/dbconfig/20261019-124022-marostegui.json 2470422 wikitext text/x-wiki gdxesv4v94jlzmg5n4n3be9dcpb0zue 2470423 2470422 2026-10-19T12:40:28Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7003.magru.wmnet 2470423 wikitext text/x-wiki 7ygvhwqki3mcnic3qvaudgehm3475uz 2470424 2470423 2026-10-19T12:40:39Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1247.eqiad.wmnet with reason: Maintenance 2470424 wikitext text/x-wiki 8e6r11nevy6355ymtk2yfwyvzbhq17v 2470425 2470424 2026-10-19T12:40:42Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1247 (T441197)', diff saved to https://phabricator.wikimedia.org/P96803 and previous config saved to /var/cache/conftool/dbconfig/20261019-124039-marostegui.json 2470425 wikitext text/x-wiki fbjswkw98vksn8aojdprb9oqfv9hfef 2470426 2470425 2026-10-19T12:41:12Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad2002.codfw.wmnet 2470426 wikitext text/x-wiki d043idsqk6lu3ymicol0txaj02zim2d 2470427 2470426 2026-10-19T12:41:18Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org 2470427 wikitext text/x-wiki kxb9wmwgw31tp0e1o61o0s0uflsrjrk 2470428 2470427 2026-10-19T12:42:14Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host doc1004.eqiad.wmnet 2470428 wikitext text/x-wiki abtuxr7kv43numfz8y0f0sshvqimxpy 2470429 2470428 2026-10-19T12:42:41Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow3004.esams.wmnet 2470429 wikitext text/x-wiki 5r3sjhqy1rokoo24akd9qae99n1ony6 2470430 2470429 2026-10-19T12:42:45Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7002.magru.wmnet 2470430 wikitext text/x-wiki kpiyl5vszbsegnwoac9q8zuux61gj89 2470431 2470430 2026-10-19T12:43:20Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1004.eqiad.wmnet 2470431 wikitext text/x-wiki izjqspxfjhstxj1ct1vwc1m8g5o0wmv 2470432 2470431 2026-10-19T12:43:22Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2470432 wikitext text/x-wiki 0g4t178guvh8hgwpfindiywm57m1q07 2470433 2470432 2026-10-19T12:43:24Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup2003.codfw.wmnet 2470433 wikitext text/x-wiki ggr8wqb2vkq9qmaiplzffsvaiegegnp 2470434 2470433 2026-10-19T12:44:08Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2005-dev.codfw.wmnet 2470434 wikitext text/x-wiki 71spcplafgs1z3ps4r7orjr5tbs6ghc 2470435 2470434 2026-10-19T12:44:11Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2006-dev.codfw.wmnet 2470435 wikitext text/x-wiki azaoowp8h6a5nahgh1sjmzgaa2opzp5 2470436 2470435 2026-10-19T12:44:13Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idm2001.wikimedia.org 2470436 wikitext text/x-wiki 7evaxctqdllmsklg7zn1hxpgtlfla9s 2470437 2470436 2026-10-19T12:45:43Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7002.magru.wmnet 2470437 wikitext text/x-wiki kwi68oambdqj2mjwtbvxgku10b74oku 2470438 2470437 2026-10-19T12:46:11Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc1004.eqiad.wmnet 2470438 wikitext text/x-wiki jf0iaxqjypoapllym1aexzc0erkh5r4 2470439 2470438 2026-10-19T12:46:17Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host doc2003.codfw.wmnet 2470439 wikitext text/x-wiki dt5mv3oip7s7bmu3wfm6fy2shy7h3jn 2470440 2470439 2026-10-19T12:46:53Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3004.esams.wmnet 2470440 wikitext text/x-wiki 168vqlkdyh59rj5go6xvvkhwf8k54s6 2470441 2470440 2026-10-19T12:47:12Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2007.codfw.wmnet 2470441 wikitext text/x-wiki n0h6f632yij9n3h5nhon3tw4orx9ij4 2470442 2470441 2026-10-19T12:47:16Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1007.eqiad.wmnet 2470442 wikitext text/x-wiki 7pn5k1xizrhzdcpo0lg9a7vdh7lypte 2470443 2470442 2026-10-19T12:47:19Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4008.ulsfo.wmnet 2470443 wikitext text/x-wiki ab4oedlnn29mzf79q5wh8qgxt9pbs9c 2470444 2470443 2026-10-19T12:47:33Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org 2470444 wikitext text/x-wiki 968tw816a3ah6zevsjum2ynfht87c2p 2470445 2470444 2026-10-19T12:48:13Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm2001.wikimedia.org 2470445 wikitext text/x-wiki 0m8a75kjbmd5u0k1wwfyjvimi6qb6kd 2470446 2470445 2026-10-19T12:48:45Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T441197)', diff saved to https://phabricator.wikimedia.org/P96804 and previous config saved to /var/cache/conftool/dbconfig/20261019-124842-marostegui.json 2470446 wikitext text/x-wiki cwmp71qrmutw3izy1pgbin726n0is7z 2470447 2470446 2026-10-19T12:48:58Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2005.codfw.wmnet 2470447 wikitext text/x-wiki 9mrcuwznflor3dsz19ydti7a53monwk 2470448 2470447 2026-10-19T12:50:17Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc2003.codfw.wmnet 2470448 wikitext text/x-wiki nn96ibt3khb9wqafm4u0js6r0qdktho 2470449 2470448 2026-10-19T12:51:13Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow2003.codfw.wmnet 2470449 wikitext text/x-wiki 58vbtedyvk4ci9zqwy936vgoko7o271 2470450 2470449 2026-10-19T12:51:31Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org 2470450 wikitext text/x-wiki 3uykv17qbub66ik9nou3x33uplg8h6i 2470451 2470450 2026-10-19T12:51:40Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2006-dev.codfw.wmnet 2470451 wikitext text/x-wiki fprkrtjcb0sp9ujqfwdvd7jartqjg0j 2470452 2470451 2026-10-19T12:51:42Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host vrts1004.eqiad.wmnet 2470452 wikitext text/x-wiki ekrh8mj6vg205wu01dbvpiaq7y2txg3 2470453 2470452 2026-10-19T12:51:44Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcephosd2007-dev.codfw.wmnet 2470453 wikitext text/x-wiki f7gaq98yuhdld1cryv93r0g3ukomc7f 2470454 2470453 2026-10-19T12:51:47Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host phab1006.eqiad.wmnet 2470454 wikitext text/x-wiki qu7j4w4sh9e226jbwxszvmtif00d69z 2470455 2470454 2026-10-19T12:51:50Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1005.eqiad.wmnet 2470455 wikitext text/x-wiki 60opm2hp8flgjrbtsse8j10zhan9cy6 2470456 2470455 2026-10-19T12:52:18Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host aphlict2001.codfw.wmnet 2470456 wikitext text/x-wiki 79izayy80nrrv1ju3ncic8zjst5c9ys 2470457 2470456 2026-10-19T12:52:33Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host lists2001.wikimedia.org 2470457 wikitext text/x-wiki 9chqyzf0u9kjk45lm1vetws8n07lrzn 2470458 2470457 2026-10-19T12:52:35Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2470458 wikitext text/x-wiki 3mvlbma97ouiqq0lpdky9xbnqpatdz0 2470459 2470458 2026-10-19T12:52:43Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host planet2003.codfw.wmnet 2470459 wikitext text/x-wiki tt658q7m4kmy4r8wlcsepeuwr8310ex 2470460 2470459 2026-10-19T12:52:57Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host stewards2001.codfw.wmnet 2470460 wikitext text/x-wiki 3rihtd1q4ydasn5wmx29xog298ykmsw 2470461 2470460 2026-10-19T12:53:49Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7002.magru.wmnet 2470461 wikitext text/x-wiki 64ouw651qxb562nwjr9d9hs4drm3zhw 2470462 2470461 2026-10-19T12:53:53Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7002.magru.wmnet 2470462 wikitext text/x-wiki lxu2564cac03354ouepgzhac0pzyuec 2470463 2470462 2026-10-19T12:54:01Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet 2470463 wikitext text/x-wiki 29b67m8icl3w86opyw73coot5cr2j69 2470464 2470463 2026-10-19T12:54:43Z Stashbot 7414 moritzm: failover Ganeti master in magru to ganeti7004 2470464 wikitext text/x-wiki g4xb1916ibz2st1cwee95z7uv21bu9u 2470465 2470464 2026-10-19T12:54:54Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2470465 wikitext text/x-wiki jcs7m46qu6zi56xq9ef2x8xw58id9sl 2470466 2470465 2026-10-19T12:55:14Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2003.codfw.wmnet 2470466 wikitext text/x-wiki etb7ud7vr1wlm3zvqmvm2vdw7p89yx4 2470467 2470466 2026-10-19T12:55:28Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1004.eqiad.wmnet 2470467 wikitext text/x-wiki 5ih3vu0w1j7585afj3fgmv4u9ih7f0u 2470468 2470467 2026-10-19T12:55:49Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow2004.codfw.wmnet 2470468 wikitext text/x-wiki l8q0r2mpng0azcqafg9g5upkyngc6su 2470469 2470468 2026-10-19T12:56:03Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host vrts2002.codfw.wmnet 2470469 wikitext text/x-wiki mklnf6x3ec5jygxvfpmhf1xwkk2ab4r 2470470 2470469 2026-10-19T12:56:17Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict2001.codfw.wmnet 2470470 wikitext text/x-wiki lhghpfvhhs47wxvsxyn5hnv4567wrlw 2470471 2470470 2026-10-19T12:56:31Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet2003.codfw.wmnet 2470471 wikitext text/x-wiki 58fskdyfv50kze2un4o3vo0asxub16p 2470472 2470471 2026-10-19T12:56:35Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host codesearch2001.codfw.wmnet 2470472 wikitext text/x-wiki 5r4bd71xafcv6f3fusjno0zlymnefri 2470473 2470472 2026-10-19T12:56:45Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards2001.codfw.wmnet 2470473 wikitext text/x-wiki 9js2gt9dm8wzzgnrz492n0luxn1bnm4 2470474 2470473 2026-10-19T12:57:22Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab1006.eqiad.wmnet 2470474 wikitext text/x-wiki huu8gi9ycsxbog3bw8c1a6aia9y59w1 2470475 2470474 2026-10-19T12:57:25Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host contint2003.wikimedia.org 2470475 wikitext text/x-wiki rm54udaxbqg8ibi2w6vrmlul6koeawz 2470476 2470475 2026-10-19T12:57:27Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host codesearch2001.codfw.wmnet 2470476 wikitext text/x-wiki sd6ig6no1fkzwr7ggd3myyzu0yo4886 2470477 2470476 2026-10-19T12:57:37Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org 2470477 wikitext text/x-wiki nnw91h4jgu55s1221lfk9suympefbsn 2470478 2470477 2026-10-19T12:57:47Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1005.eqiad.wmnet 2470478 wikitext text/x-wiki tppz7ujwi0lugas9ah82o49669yuuqo 2470479 2470478 2026-10-19T12:57:50Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host people1005.eqiad.wmnet 2470479 wikitext text/x-wiki 2m80svf1klf846dccb1b1kw3izyzuoq 2470480 2470479 2026-10-19T12:57:53Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup2003.codfw.wmnet 2470480 wikitext text/x-wiki htn9hilxxuwlb8u3wom70lww4l9h7p9 2470481 2470480 2026-10-19T12:57:56Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup2004.codfw.wmnet 2470481 wikitext text/x-wiki 3iceeesbpn9ovubkp2vi8pznrpg6w6t 2470482 2470481 2026-10-19T12:58:25Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host phab2003.codfw.wmnet 2470482 wikitext text/x-wiki 7q6jmmqpduskxrpw663cc0mgoqw9qdf 2470483 2470482 2026-10-19T12:59:01Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2007-dev.codfw.wmnet 2470483 wikitext text/x-wiki tb5qi0lvr8yjslerkdvinomqgr42ue5 2470484 2470483 2026-10-19T12:59:03Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists2001.wikimedia.org 2470484 wikitext text/x-wiki a1mt22asvm8er0p8ckuqqeoc9j69lby 2470485 2470484 2026-10-19T12:59:05Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host releases2003.codfw.wmnet 2470485 wikitext text/x-wiki 1xc6zf4mon0kymhxwihokbn5mkj5cy8 2470486 2470485 2026-10-19T13:00:20Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet 2470486 wikitext text/x-wiki 6bjyzpke8jtg4dkq3xkl7rssyb9p0ib 2470487 2470486 2026-10-19T13:00:26Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet 2470487 wikitext text/x-wiki clrsqgc5btxea4d5txuhfhsgtwfj72c 2470488 2470487 2026-10-19T13:01:42Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1005.eqiad.wmnet 2470488 wikitext text/x-wiki ajbotaku8ajuwrh82bqjln6gxxok4a0 2470489 2470488 2026-10-19T13:02:25Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idm1001.wikimedia.org 2470489 wikitext text/x-wiki 2a2ubh2th3nphf4x59i86b2k4fjdvlh 2470490 2470489 2026-10-19T13:02:40Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts2002.codfw.wmnet 2470490 wikitext text/x-wiki na9vro1ig7v6qbanu2jmy0xf7yulbdd 2470491 2470490 2026-10-19T13:03:05Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases2003.codfw.wmnet 2470491 wikitext text/x-wiki l46d8ayb32l4ymf1ox9ja2a0yme57ou 2470493 2470491 2026-10-19T13:03:46Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P96805 and previous config saved to /var/cache/conftool/dbconfig/20261019-130344-marostegui.json 2470493 wikitext text/x-wiki eqjl510hvnlrn6ohtpajtp39uuqhlz2 2470494 2470493 2026-10-19T13:03:47Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus1006.eqiad.wmnet 2470494 wikitext text/x-wiki 6gjcgtvqax1o6zgrhl1rm53ngo72192 2470495 2470494 2026-10-19T13:03:50Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus1008.eqiad.wmnet 2470495 wikitext text/x-wiki gb84h80ff58inysp92v3ujq4xa6amrg 2470496 2470495 2026-10-19T13:03:54Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus2006.codfw.wmnet 2470496 wikitext text/x-wiki c3n26znevto7letwm96yp1lpfer41u7 2470497 2470496 2026-10-19T13:03:57Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1005.eqiad.wmnet 2470497 wikitext text/x-wiki 9zw16uf7p14arqqt0dd2fbyzopteilq 2470498 2470497 2026-10-19T13:03:59Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host prometheus2008.codfw.wmnet 2470498 wikitext text/x-wiki re9s0025g6ti3qns5t8bm7585vhxujs 2470499 2470498 2026-10-19T13:04:02Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint2003.wikimedia.org 2470499 wikitext text/x-wiki 2nva4srikcptg97kdohfpuz4y0b5lml 2470500 2470499 2026-10-19T13:04:04Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab2003.codfw.wmnet 2470500 wikitext text/x-wiki cj4rzenollipjne9pufkp6y26caa9rz 2470501 2470500 2026-10-19T13:04:09Z Stashbot 7414 XioNoX: upgrade spicerack on cumin2003 2470501 wikitext text/x-wiki sz60wz4mjxbrze2mgst2xgyg3kcq13e 2470502 2470501 2026-10-19T13:04:20Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul2001.codfw.wmnet 2470502 wikitext text/x-wiki auk3duux9kf4165isum205ulss1cjik 2470505 2470502 2026-10-19T13:06:23Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm1001.wikimedia.org 2470505 wikitext text/x-wiki 175ip2yczv4zfaz5qcny2rjm6nhyzkj 2470506 2470505 2026-10-19T13:06:36Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7001.magru.wmnet 2470506 wikitext text/x-wiki 4xs0g6a19ab1lky299ybue6sy852ml0 2470507 2470506 2026-10-19T13:06:45Z Stashbot 7414 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1356389|Config change for opting out cswikiversity from LL suppression. (T441437)]] 2470507 wikitext text/x-wiki r8arvpo556uyhjmziriqi49mvbwgdte 2470508 2470507 2026-10-19T13:06:47Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4007.ulsfo.wmnet 2470508 wikitext text/x-wiki h9b0jm1ntqip3zviuni8l5g3rq0y9dv 2470509 2470508 2026-10-19T13:08:19Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2001.codfw.wmnet 2470509 wikitext text/x-wiki tw5hq0h4vlww2ql17ocg8ce0j4p7hs0 2470510 2470509 2026-10-19T13:08:22Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup2004.codfw.wmnet 2470510 wikitext text/x-wiki ehvwcedzbo10gyf1ewxnlhivoyi72z9 2470511 2470510 2026-10-19T13:08:28Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul2002.codfw.wmnet 2470511 wikitext text/x-wiki oq5o694vkrgm3ol2s7cqf74gn22peqt 2470512 2470511 2026-10-19T13:08:48Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp-test1005.wikimedia.org 2470512 wikitext text/x-wiki 31yxamg63td5c3gkbw9i9ttokczum7c 2470513 2470512 2026-10-19T13:11:22Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy1001.eqiad.wmnet 2470513 wikitext text/x-wiki k9jxiqjsrmj2z8cqgxdg5l65htr71tc 2470514 2470513 2026-10-19T13:11:33Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1008.eqiad.wmnet 2470514 wikitext text/x-wiki 80af855mkv8ljml4iw1d2ll387qzydm 2470515 2470514 2026-10-19T13:11:59Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2008.codfw.wmnet 2470515 wikitext text/x-wiki hiu1bvl5jdzhfedh7hrnju7xto8bat6 2470516 2470515 2026-10-19T13:12:27Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2002.codfw.wmnet 2470516 wikitext text/x-wiki g9oagdqramguvxnlfucabyjs3fmro2r 2470517 2470516 2026-10-19T13:12:43Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1005.wikimedia.org 2470517 wikitext text/x-wiki krs0xfdxe4c1l0qn13zl49w3jo9nj5a 2470518 2470517 2026-10-19T13:13:02Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul2003.codfw.wmnet 2470518 wikitext text/x-wiki 54kxm4f9nkwq2xbifcg6ux7mjjji1ee 2470519 2470518 2026-10-19T13:13:37Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2006.codfw.wmnet 2470519 wikitext text/x-wiki ciboir26nq56j9z5h0mkghxivvdmtaz 2470520 2470519 2026-10-19T13:14:36Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet 2470520 wikitext text/x-wiki ryv24jonz9lm94br14q6oooxapet0xw 2470521 2470520 2026-10-19T13:15:34Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org 2470521 wikitext text/x-wiki 0cxpqrafu9zs1i2w56k0zgbh6374sz6 2470522 2470521 2026-10-19T13:15:49Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy1001.eqiad.wmnet 2470522 wikitext text/x-wiki g9657xtcf6gsxpon5j672p7kqreo9al 2470523 2470522 2026-10-19T13:16:01Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti4007.ulsfo.wmnet 2470523 wikitext text/x-wiki lvv25xihev2fi293bde9fky5laf54yo 2470524 2470523 2026-10-19T13:16:05Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7001.magru.wmnet 2470524 wikitext text/x-wiki d8quecpjby40z9all374d19a4fjsq4t 2470525 2470524 2026-10-19T13:16:18Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy1002.eqiad.wmnet 2470525 wikitext text/x-wiki nd8ko8xyi553qdg7irrntrkmsqqaydt 2470526 2470525 2026-10-19T13:16:53Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcumin1001.eqiad.wmnet 2470526 wikitext text/x-wiki bojsvcc6g4buxonxt5fapv08c8mpnvy 2470527 2470526 2026-10-19T13:17:02Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2003.codfw.wmnet 2470527 wikitext text/x-wiki qh5hq0a8luig8hd3e02r8ucmksixsqp 2470528 2470527 2026-10-19T13:17:17Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1006.eqiad.wmnet 2470528 wikitext text/x-wiki n5aphre3rnifmdygn0fnhsifk0k2svl 2470529 2470528 2026-10-19T13:18:26Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin2001.codfw.wmnet 2470529 wikitext text/x-wiki 0gum4ow4v5kks0j3jz47zqz1n49p1e8 2470530 2470529 2026-10-19T13:18:42Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner1002.eqiad.wmnet} and (A:gitlab-runner) 2470530 wikitext text/x-wiki t32lnbwqzax0yv5qvnqeq91qdx1sg77 2470531 2470530 2026-10-19T13:18:48Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P96806 and previous config saved to /var/cache/conftool/dbconfig/20261019-131846-marostegui.json 2470531 wikitext text/x-wiki av84q5sktbuwqsooru2jni1pp78ciqj 2470532 2470531 2026-10-19T13:19:19Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org 2470532 wikitext text/x-wiki p3wsk05dim37zz5r8f563ryvoywnhq7 2470533 2470532 2026-10-19T13:19:34Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp2005.wikimedia.org 2470533 wikitext text/x-wiki 1xxr2bqc5d818sovkbxgodoqwnxc6cn 2470534 2470533 2026-10-19T13:20:09Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.presto.reboot-workers for Presto an-presto cluster: Reboot Presto nodes 2470534 wikitext text/x-wiki i2dcvjkconiks0rndd13k0qhl4f7s35 2470535 2470534 2026-10-19T13:20:41Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy1002.eqiad.wmnet 2470535 wikitext text/x-wiki ivb8q9oxg8ljxsy3jriv4q77ufobgl3 2470536 2470535 2026-10-19T13:20:49Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin1001.eqiad.wmnet 2470536 wikitext text/x-wiki a7zlf3hounqndgu57hx1tf30f5wrgwk 2470537 2470536 2026-10-19T13:21:32Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host planet1003.eqiad.wmnet 2470537 wikitext text/x-wiki j8230r138h5c0paozsfiln3enl8mc2l 2470538 2470537 2026-10-19T13:21:34Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host people2004.codfw.wmnet 2470538 wikitext text/x-wiki p39jdwkjo6fto2wek0xa036gc6ouzpd 2470539 2470538 2026-10-19T13:21:48Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet 2470539 wikitext text/x-wiki sd0is37pbio5q32yhgme500gtruu261 2470540 2470539 2026-10-19T13:21:53Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host stewards1001.eqiad.wmnet 2470540 wikitext text/x-wiki 45e049rzemftnkue3g8cv0jrztliaxu 2470541 2470540 2026-10-19T13:22:05Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host codesearch1001.eqiad.wmnet 2470541 wikitext text/x-wiki 5v291pgqf36qmpjvkggk0le359bqas4 2470542 2470541 2026-10-19T13:22:57Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host codesearch1001.eqiad.wmnet 2470542 wikitext text/x-wiki fer1k2tu5fu3q30qluf4ob2te59i8ip 2470543 2470542 2026-10-19T13:22:59Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy2001.codfw.wmnet 2470543 wikitext text/x-wiki dhwxpcv50mcqz1aoegupbj9o4p0bu6h 2470544 2470543 2026-10-19T13:23:19Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2005.wikimedia.org 2470544 wikitext text/x-wiki adeasasim18qecjal7i9a0p377c07ae 2470545 2470544 2026-10-19T13:24:17Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4007.ulsfo.wmnet 2470545 wikitext text/x-wiki 1nimmhq25nu5nt4bo01twx3tr7f3ego 2470546 2470545 2026-10-19T13:24:22Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4007.ulsfo.wmnet 2470546 wikitext text/x-wiki coxdwa19fg4mdo8ls1uf5ttp61mcizj 2470547 2470546 2026-10-19T13:24:53Z Stashbot 7414 XioNoX: upgrade spicerack on cumin1004 2470547 wikitext text/x-wiki avh1fhfp5pscn8095w9l6l4oj5hkwn8 2470548 2470547 2026-10-19T13:25:29Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet1003.eqiad.wmnet 2470548 wikitext text/x-wiki rznkld0dp5cfjl7cgxxdapd0l1x6wli 2470549 2470548 2026-10-19T13:25:32Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people2004.codfw.wmnet 2470549 wikitext text/x-wiki olnynndb932m7dxnfb8kr6xfkz3zv7j 2470550 2470549 2026-10-19T13:25:34Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner1002.eqiad.wmnet} and (A:gitlab-runner) 2470550 wikitext text/x-wiki ap7mvpg1trpa80rq911yv459p13vuyt 2470551 2470550 2026-10-19T13:25:36Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards1001.eqiad.wmnet 2470551 wikitext text/x-wiki g6qaiuccakpg7ecljhgik0iy147h5rr 2470552 2470551 2026-10-19T13:27:03Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet 2470552 wikitext text/x-wiki fvn9qawbrrx219055kjc7kpbtqiyrt9 2470553 2470552 2026-10-19T13:27:21Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy2001.codfw.wmnet 2470553 wikitext text/x-wiki 0nf9qimqia2dkp5qqpfpv4biqbvp99m 2470554 2470553 2026-10-19T13:27:30Z Stashbot 7414 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1356389|Config change for opting out cswikiversity from LL suppression. (T441437)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2470554 wikitext text/x-wiki s6cjwg6fzc74v16d31to6slb4veodfw 2470555 2470554 2026-10-19T13:28:05Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner1003.eqiad.wmnet} and (A:gitlab-runner) 2470555 wikitext text/x-wiki 6hav3g93lxzzd0rbxhjfctspwf6twa2 2470556 2470555 2026-10-19T13:28:41Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy2002.codfw.wmnet 2470556 wikitext text/x-wiki s7ozyr7vfqlbd58cjh6k08utfa94if6 2470557 2470556 2026-10-19T13:29:55Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4006.ulsfo.wmnet 2470557 wikitext text/x-wiki 9q6tmjo80x0k0wcgxvnnlcnjc5yrtym 2470558 2470557 2026-10-19T13:30:23Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2470558 wikitext text/x-wiki r0mjzexurmaglhdkawmu8zvk5govp3s 2470559 2470558 2026-10-19T13:32:28Z Stashbot 7414 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment 2470559 wikitext text/x-wiki a795am1f077tk3n3nr1tz1sok5pe9mq 2470560 2470559 2026-10-19T13:32:47Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2470560 wikitext text/x-wiki 5mzmiyr901uky250b6rogjy3o5si1oh 2470561 2470560 2026-10-19T13:33:15Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy2002.codfw.wmnet 2470561 wikitext text/x-wiki p993jpwvcq8thy4m0rcwq70nqw5zbro 2470562 2470561 2026-10-19T13:33:29Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy2002.codfw.wmnet 2470562 wikitext text/x-wiki 8wtz9im1dai2jti1vvixfw6rf4vdqq4 2470563 2470562 2026-10-19T13:33:49Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2004.codfw.wmnet 2470563 wikitext text/x-wiki c3lo5eom5wubel7ob8ebw6swuddjodo 2470564 2470563 2026-10-19T13:33:52Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T441197)', diff saved to https://phabricator.wikimedia.org/P96807 and previous config saved to /var/cache/conftool/dbconfig/20261019-133348-marostegui.json 2470564 wikitext text/x-wiki 412g74kr01z5ins0km12qmul5bkg5dm 2470565 2470564 2026-10-19T13:34:05Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1248.eqiad.wmnet with reason: Maintenance 2470565 wikitext text/x-wiki c5fu9lmqy5yfhoxpknhpqe9rvk01jyp 2470566 2470565 2026-10-19T13:34:09Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1248 (T441197)', diff saved to https://phabricator.wikimedia.org/P96808 and previous config saved to /var/cache/conftool/dbconfig/20261019-133406-marostegui.json 2470566 wikitext text/x-wiki f5cvtbg7vi3ue6cugop6z70zrrf5ysx 2470567 2470566 2026-10-19T13:34:18Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti4006.ulsfo.wmnet 2470567 wikitext text/x-wiki qo82rxyzmb7ywlmo30kd19tj3mf8hpj 2470568 2470567 2026-10-19T13:34:42Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow2005.codfw.wmnet 2470568 wikitext text/x-wiki ncolsmdf5m5wkoot28h3zs2jkjb8oew 2470569 2470568 2026-10-19T13:35:43Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner1003.eqiad.wmnet} and (A:gitlab-runner) 2470569 wikitext text/x-wiki n5d274xfrct1bumw1hi8b5u4s80zn2g 2470570 2470569 2026-10-19T13:35:54Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner1004.eqiad.wmnet} and (A:gitlab-runner) 2470570 wikitext text/x-wiki i2zgbpnm72wqzcut4ilo5dbl5ez9qt5 2470571 2470570 2026-10-19T13:38:01Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy2002.codfw.wmnet 2470571 wikitext text/x-wiki 2brzancp9nin72eslgos1pxrpzfuf8t 2470572 2470571 2026-10-19T13:38:37Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo 2470572 wikitext text/x-wiki q86dpcyvdw9iei7et19w85guysnqonh 2470573 2470572 2026-10-19T13:39:24Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp1005.wikimedia.org 2470573 wikitext text/x-wiki 7klrghjtpdnfo96l21fg4r3g0wzg1cy 2470574 2470573 2026-10-19T13:39:29Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on P{dns7001*} and (A:dnsbox) 2470574 wikitext text/x-wiki cpo2pkh25zfha1mvea0yc6f80qbiijz 2470575 2470574 2026-10-19T13:39:32Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns7001.wikimedia.org 2470575 wikitext text/x-wiki 8mj0kmgc3dxq5xpbhv7zp32wmqrzgcj 2470576 2470575 2026-10-19T13:39:37Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti7001.magru.wmnet 2470576 wikitext text/x-wiki ngpgcrq7siwgw8ua8x2ehacy7nmlfhw 2470577 2470576 2026-10-19T13:39:39Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti7001.magru.wmnet 2470577 wikitext text/x-wiki 8s3tzaf4k94zgbr7k99qah9g2a38gyj 2470578 2470577 2026-10-19T13:40:03Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2005.codfw.wmnet 2470578 wikitext text/x-wiki 19skm11fqwt73588ms0cuy61d6v82hu 2470579 2470578 2026-10-19T13:40:16Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum 2470579 wikitext text/x-wiki 5m2jf5m7ixa6jqgrnqrjt8s549qgche 2470580 2470579 2026-10-19T13:40:44Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4006.ulsfo.wmnet 2470580 wikitext text/x-wiki 0c9okxdmq2goqmggnk799lijoqpzujn 2470581 2470580 2026-10-19T13:40:49Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet 2470581 wikitext text/x-wiki iwjvopnzlpd2qs963bdpaddybo6bhnx 2470582 2470581 2026-10-19T13:41:00Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo 2470582 wikitext text/x-wiki eyippde52mwziq46q8a9zzwlzhmtylj 2470583 2470582 2026-10-19T13:41:03Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough 2470583 wikitext text/x-wiki 3cjw4tyhcfnl2268zr77a2e4ke4jow8 2470584 2470583 2026-10-19T13:41:05Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance 2470584 wikitext text/x-wiki 3apxby58pd0mhtsmgqw0c52tbyxapbf 2470585 2470584 2026-10-19T13:41:08Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2171 (T441633)', diff saved to https://phabricator.wikimedia.org/P96809 and previous config saved to /var/cache/conftool/dbconfig/20261019-134105-ladsgroup.json 2470585 wikitext text/x-wiki ngr9x9ditwcedxy78ts23ymg08l1ozw 2470586 2470585 2026-10-19T13:43:06Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp7010* or cp7002*} and A:cp 2470586 wikitext text/x-wiki 72y36brxshdlc03sv3o3wk82avrp1we 2470587 2470586 2026-10-19T13:43:09Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1005.wikimedia.org 2470587 wikitext text/x-wiki jhh1kaij3571kp4tx1rwo8z0utm7smw 2470589 2470587 2026-10-19T13:43:48Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner1004.eqiad.wmnet} and (A:gitlab-runner) 2470589 wikitext text/x-wiki opgc41dhko9o5flod8maxnqym4z70eq 2470590 2470589 2026-10-19T13:43:59Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner2002.codfw.wmnet} and (A:gitlab-runner) 2470590 wikitext text/x-wiki qmgd71xtec0zryllkcavrvi6ko1tdja 2470591 2470590 2026-10-19T13:44:40Z Stashbot 7414 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356389|Config change for opting out cswikiversity from LL suppression. (T441437)]] (duration: 37m 55s) 2470591 wikitext text/x-wiki os2qchgajke5ssx7o9wb8h1io4ix4qk 2470592 2470591 2026-10-19T13:44:44Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T441197)', diff saved to https://phabricator.wikimedia.org/P96810 and previous config saved to /var/cache/conftool/dbconfig/20261019-134441-marostegui.json 2470592 wikitext text/x-wiki 5zzfwfrx0awauet9s6qkm72ed098d6z 2470593 2470592 2026-10-19T13:44:57Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy 2470593 wikitext text/x-wiki hb89pfmv4pq4ciipo54xn0vtpu2qv1x 2470594 2470593 2026-10-19T13:46:34Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on P{ncredir7004*} and A:ncredir 2470594 wikitext text/x-wiki j72twfet7wt1um3bb2d1tpkrnepvu6w 2470596 2470594 2026-10-19T13:47:52Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T441633)', diff saved to https://phabricator.wikimedia.org/P96811 and previous config saved to /var/cache/conftool/dbconfig/20261019-134750-ladsgroup.json 2470596 wikitext text/x-wiki e8gqxrcu1ilbyhnk2cod7twwwmimjmq 2470597 2470596 2026-10-19T13:47:55Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs7003.magru.wmnet 2470597 wikitext text/x-wiki pyfv81ec916x1xshcu0576gsg0odt1s 2470598 2470597 2026-10-19T13:48:08Z Stashbot 7414 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host lvs7003.magru.wmnet 2470598 wikitext text/x-wiki 27rvqtxeav082d62809ovpb3cxnp0kt 2470599 2470598 2026-10-19T13:48:39Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs7003.magru.wmnet 2470599 wikitext text/x-wiki 4belhi0kyuioo6eoto1oufd2p2tvofm 2470600 2470599 2026-10-19T13:49:55Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy3001.esams.wmnet 2470600 wikitext text/x-wiki 72pe549q2xhuyxgluwxvwgehqmsrpjh 2470601 2470600 2026-10-19T13:50:47Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on P{ncredir7004*} and A:ncredir 2470601 wikitext text/x-wiki qrvw957moe7a3ah6qco3nt471km5l6g 2470602 2470601 2026-10-19T13:50:59Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner2002.codfw.wmnet} and (A:gitlab-runner) 2470602 wikitext text/x-wiki o5niw5o7jfazzrjjhzt71gz1ppd7yjd 2470603 2470602 2026-10-19T13:51:07Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns7001.wikimedia.org 2470603 wikitext text/x-wiki cbzhu3dsilvk6flouusz55dmgw11xog 2470604 2470603 2026-10-19T13:51:09Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P{dns7001*} and (A:dnsbox) 2470604 wikitext text/x-wiki pm4lfm7teh1rnxgvn0ihx84a4xv9ty7 2470605 2470604 2026-10-19T13:51:11Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner2003.codfw.wmnet} and (A:gitlab-runner) 2470605 wikitext text/x-wiki 44qw8u3hqt3ymuhep2zkshw1zpfbg6y 2470606 2470605 2026-10-19T13:52:38Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352919|knwikisource: lift IP cap for workshop on 24-October-2026 (T440546)]], [[gerrit:1355244|mlwikivoyage: add logo, wordmark and set sitename, timezone (T441173)]], [[gerrit:1356681|ruwikisource: add 'A' as NamespaceAliases for author namespace (T439568)]] 2470606 wikitext text/x-wiki 7ry48cgp4364ckaugtymm8y0ufzc7b0 2470607 2470606 2026-10-19T13:52:48Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host stewards1001.eqiad.wmnet 2470607 wikitext text/x-wiki 22db5ffqh621m7fc3kt2li2kifl8vfm 2470608 2470607 2026-10-19T13:52:56Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org 2470608 wikitext text/x-wiki adwozu934ocrwo1bf4faahh6ba03ndw 2470609 2470608 2026-10-19T13:53:05Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet 2470609 wikitext text/x-wiki 2m8kkehyvkyftkuf1tie4cpv49l97rm 2470610 2470609 2026-10-19T13:54:31Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy3001.esams.wmnet 2470610 wikitext text/x-wiki 8fnn0olst7i5e8r8qdsp2qm7jacfdgt 2470611 2470610 2026-10-19T13:55:02Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7002.magru.wmnet 2470611 wikitext text/x-wiki s7k48stcq5p5mhh361cfp3dlii2wclz 2470612 2470611 2026-10-19T13:55:15Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2470612 wikitext text/x-wiki 99qpzzoo8u7wpvp5q1sn3ap31eizcqt 2470613 2470612 2026-10-19T13:55:18Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . 2470613 wikitext text/x-wiki aes2cg8zn98yj0nt0dtcl0a4nptdb33 2470614 2470613 2026-10-19T13:55:32Z Stashbot 7414 moritzm: failover Ganeti master in ulsfo to ganeti4008 2470614 wikitext text/x-wiki jx88wv9s313g6ywphuvo8m91c4yx9zm 2470615 2470614 2026-10-19T13:56:13Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs7003.magru.wmnet 2470615 wikitext text/x-wiki edp52hgniwylm9cyw1zjhfd4kav0wc0 2470616 2470615 2026-10-19T13:56:34Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 11 hosts with reason: Kernel reboots 2470616 wikitext text/x-wiki h7lswsptl1c8g626lkz868ocy8zq4on 2470617 2470616 2026-10-19T13:56:44Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards1001.eqiad.wmnet 2470617 wikitext text/x-wiki 8b4mj77z2io141341ok4lw2o8j424fh 2470618 2470617 2026-10-19T13:56:51Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: anzx, lucaswerkmeister-wmde: Backport for [[gerrit:1352919|knwikisource: lift IP cap for workshop on 24-October-2026 (T440546)]], [[gerrit:1355244|mlwikivoyage: add logo, wordmark and set sitename, timezone (T441173)]], [[gerrit:1356681|ruwikisource: add 'A' as NamespaceAliases for author namespace (T439568)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug) 2470618 wikitext text/x-wiki gyy9awrdhkxq2n0nnlez5ar3j8pa29t 2470619 2470618 2026-10-19T13:56:56Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1002.eqiad.wmnet 2470619 wikitext text/x-wiki bhyy6vz7kzszadsc3h5u72yip10stmv 2470620 2470619 2026-10-19T13:57:54Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P96812 and previous config saved to /var/cache/conftool/dbconfig/20261019-135752-ladsgroup.json 2470620 wikitext text/x-wiki hpk9195kib6uuuau2txe0t0j47o8tjm 2470621 2470620 2026-10-19T13:58:38Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner2003.codfw.wmnet} and (A:gitlab-runner) 2470621 wikitext text/x-wiki 3tr8d936067if4arwq2m5y0ghb2m2mm 2470622 2470621 2026-10-19T13:58:49Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on P{gitlab-runner2004.codfw.wmnet} and (A:gitlab-runner) 2470622 wikitext text/x-wiki nxb55ota76u00re2e51gshz4xgkdjiv 2470623 2470622 2026-10-19T13:58:59Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: anzx, lucaswerkmeister-wmde: Continuing with deployment 2470623 wikitext text/x-wiki 60uz1o1rjsnbkvl4hfz8coyexw5095m 2470624 2470623 2026-10-19T13:59:35Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org 2470624 wikitext text/x-wiki 389rezv4azu0943smc9j9j6zg4s4ib7 2470625 2470624 2026-10-19T13:59:44Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P96813 and previous config saved to /var/cache/conftool/dbconfig/20261019-135943-marostegui.json 2470625 wikitext text/x-wiki 4z3g9ni1w062zltil7vr1awwid5h2pu 2470626 2470625 2026-10-19T13:59:47Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet 2470626 wikitext text/x-wiki 7sxyndvjbd77k6kukroiwelniqtzl6e 2470627 2470626 2026-10-19T14:00:45Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1002.eqiad.wmnet 2470627 wikitext text/x-wiki 0fb73npntwft7p710bnetbna1ppakwt 2470628 2470627 2026-10-19T14:00:55Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy3002.esams.wmnet 2470628 wikitext text/x-wiki aatlc6le6zc4p0olkxr3wop9hwzvegl 2470629 2470628 2026-10-19T14:02:28Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1022.eqiad.wmnet with reason: Kernel reboots 2470629 wikitext text/x-wiki c19pf8r522now4txbwuk09zstvpl9my 2470630 2470629 2026-10-19T14:04:34Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352919|knwikisource: lift IP cap for workshop on 24-October-2026 (T440546)]], [[gerrit:1355244|mlwikivoyage: add logo, wordmark and set sitename, timezone (T441173)]], [[gerrit:1356681|ruwikisource: add 'A' as NamespaceAliases for author namespace (T439568)]] (duration: 11m 57s) 2470630 wikitext text/x-wiki off7lqor0kc573v9auuuvsy7riit94x 2470631 2470630 2026-10-19T14:05:07Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on P{gitlab-runner2004.codfw.wmnet} and (A:gitlab-runner) 2470631 wikitext text/x-wiki m57j6xj4o5m9j5zi0o0bz8sj1u57yxq 2470632 2470631 2026-10-19T14:05:34Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy3002.esams.wmnet 2470632 wikitext text/x-wiki dhfd517wdqnjd8uhfcx3a2z9ldxmwbz 2470633 2470632 2026-10-19T14:05:48Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy4003.ulsfo.wmnet 2470633 wikitext text/x-wiki k2arakzzgykq5ivlhxun25yd7mm17wr 2470634 2470633 2026-10-19T14:06:28Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1025.eqiad.wmnet with reason: Kernel reboots 2470634 wikitext text/x-wiki 6iml7985rm08lb2ukiral5z44hjwveh 2470635 2470634 2026-10-19T14:07:56Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P96814 and previous config saved to /var/cache/conftool/dbconfig/20261019-140754-ladsgroup.json 2470635 wikitext text/x-wiki 85a1fxbbr7phms5po350hvhz08um0a8 2470636 2470635 2026-10-19T14:10:24Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy4003.ulsfo.wmnet 2470636 wikitext text/x-wiki funqkvew59sk6pednroqu4druw9077n 2470637 2470636 2026-10-19T14:10:38Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy4004.ulsfo.wmnet 2470637 wikitext text/x-wiki k2wi7zpe7bdtyw7ih2uybctebnjxxag 2470638 2470637 2026-10-19T14:11:49Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1028.eqiad.wmnet with reason: Kernel reboots 2470638 wikitext text/x-wiki 7e79cxeb396jt8d2895zzycex5qtxh3 2470639 2470638 2026-10-19T14:13:18Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1027.eqiad.wmnet with reason: Kernel reboots 2470639 wikitext text/x-wiki f32u9pc4ch3twk7i7zpt1qu0gsoqlfk 2470640 2470639 2026-10-19T14:14:46Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P96815 and previous config saved to /var/cache/conftool/dbconfig/20261019-141445-marostegui.json 2470640 wikitext text/x-wiki 3suv1mwxzgxtu6fyexva73ycwd2n3ko 2470641 2470640 2026-10-19T14:15:03Z Stashbot 7414 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2470641 wikitext text/x-wiki sbazwxutmlaurjrmx1ud6t5scycrmer 2470642 2470641 2026-10-19T14:15:05Z Stashbot 7414 kamila@deploy1003: Started scap deploy-service shellbox: pilot PHP 8.5 on -timeline and -video 2470642 wikitext text/x-wiki s4ahcesmirueq538b4t5fwblzet9g2g 2470643 2470642 2026-10-19T14:15:14Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy4004.ulsfo.wmnet 2470643 wikitext text/x-wiki pqapsj07hv9t48ewtnys9jel3vcguhp 2470644 2470643 2026-10-19T14:15:31Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy5003.eqsin.wmnet 2470644 wikitext text/x-wiki d3blxcwdj7w897hfuhzchq7exglg7uh 2470645 2470644 2026-10-19T14:15:38Z Stashbot 7414 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2470645 wikitext text/x-wiki mfq8d5zm32fbkgdz6tcidfl6dxdxfj7 2470646 2470645 2026-10-19T14:17:58Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T441633)', diff saved to https://phabricator.wikimedia.org/P96816 and previous config saved to /var/cache/conftool/dbconfig/20261019-141756-ladsgroup.json 2470646 wikitext text/x-wiki 4ji8dxudyy7yqbp1ua0i6oy98xx1kr7 2470647 2470646 2026-10-19T14:18:06Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance 2470647 wikitext text/x-wiki 6bz4h1cuad8qhfkdlyvf2hrdj2n9jng 2470648 2470647 2026-10-19T14:18:09Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2178 (T441633)', diff saved to https://phabricator.wikimedia.org/P96817 and previous config saved to /var/cache/conftool/dbconfig/20261019-141807-ladsgroup.json 2470648 wikitext text/x-wiki 8aynu70kw4wkr2hczyqwv7r00dykkzs 2470649 2470648 2026-10-19T14:18:19Z Stashbot 7414 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2470649 wikitext text/x-wiki qx439jtao35wvkdntq1j51vqo4kqtmu 2470650 2470649 2026-10-19T14:19:39Z Stashbot 7414 kamila@deploy1003: Finished scap deploy-service shellbox: pilot PHP 8.5 on -timeline and -video (duration: 04m 35s) 2470650 wikitext text/x-wiki 3yer3040ylt6bykbtul7ne6658dz7kw 2470651 2470650 2026-10-19T14:20:28Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy5003.eqsin.wmnet 2470651 wikitext text/x-wiki 5catetys3tn59c5x34vnry17hxiz9eh 2470652 2470651 2026-10-19T14:20:44Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy5004.eqsin.wmnet 2470652 wikitext text/x-wiki qn0pzx2jogsjqlq6kk780ff32falgwr 2470653 2470652 2026-10-19T14:22:22Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet 2470653 wikitext text/x-wiki tbs8zf3rcyyqozde7do10rqj2gdi30n 2470654 2470653 2026-10-19T14:24:41Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T441633)', diff saved to https://phabricator.wikimedia.org/P96818 and previous config saved to /var/cache/conftool/dbconfig/20261019-142440-ladsgroup.json 2470654 wikitext text/x-wiki ehjennt5dlprcqg1g42b15a5yo10icl 2470655 2470654 2026-10-19T14:25:26Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy5004.eqsin.wmnet 2470655 wikitext text/x-wiki 8adp1mjvn4iuyrr5qmowgb7pf38dul2 2470656 2470655 2026-10-19T14:25:41Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy6001.drmrs.wmnet 2470656 wikitext text/x-wiki 0nxxt7vfj9qu36r4w4w2yzpqtycef42 2470657 2470656 2026-10-19T14:26:02Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin 2470657 wikitext text/x-wiki evtt6qxrry8ismu2t30qfsgqm980nrh 2470659 2470657 2026-10-19T14:27:38Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet 2470659 wikitext text/x-wiki 2ydne9iwxx1izb3xv3yfxj1brg8mxjh 2470660 2470659 2026-10-19T14:29:40Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin 2470660 wikitext text/x-wiki kugrkirllonatsrxnismn43gee2d7g2 2470661 2470660 2026-10-19T14:29:50Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T441197)', diff saved to https://phabricator.wikimedia.org/P96819 and previous config saved to /var/cache/conftool/dbconfig/20261019-142947-marostegui.json 2470661 wikitext text/x-wiki c5sr6481dci0xfotbf63iohr8ahjdce 2470662 2470661 2026-10-19T14:30:04Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1249.eqiad.wmnet with reason: Maintenance 2470662 wikitext text/x-wiki hfcflg04z2ajd4pcg8wj0jqg7xp5pn1 2470663 2470662 2026-10-19T14:30:07Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1249 (T441197)', diff saved to https://phabricator.wikimedia.org/P96820 and previous config saved to /var/cache/conftool/dbconfig/20261019-143004-marostegui.json 2470663 wikitext text/x-wiki 2njfkmrsbwf43nmnalqo1qae6efif22 2470664 2470663 2026-10-19T14:30:19Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy6001.drmrs.wmnet 2470664 wikitext text/x-wiki bw41ygglwled9ldeztd7enof3fb4b8p 2470665 2470664 2026-10-19T14:30:33Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy6002.drmrs.wmnet 2470665 wikitext text/x-wiki ks0dxsj8hlu4qmf6k1iuphregq5wr0g 2470666 2470665 2026-10-19T14:32:14Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy6002.drmrs.wmnet 2470666 wikitext text/x-wiki mrgi6nlui9kibs5afast6v34lbev1cm 2470667 2470666 2026-10-19T14:32:29Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy7001.magru.wmnet 2470667 wikitext text/x-wiki hrqd4r2gbs2iqp07szfn0xh3kd80ct7 2470668 2470667 2026-10-19T14:32:42Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1356889|hCaptcha: Remove ResourceLoader dependency from HCaptchaOutput (T441681)]] (1.47.0-wmf.24) 2470668 wikitext text/x-wiki b7u0da8ypvq81j7foyddfges6a1cad7 2470669 2470668 2026-10-19T14:34:43Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P96821 and previous config saved to /var/cache/conftool/dbconfig/20261019-143442-ladsgroup.json 2470669 wikitext text/x-wiki 9r8s0el4svphhtg2qqkfk8mzu3zjx7b 2470672 2470669 2026-10-19T14:36:03Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet 2470672 wikitext text/x-wiki f7ayweemi05jpyz34ajocrk95wf92q8 2470673 2470672 2026-10-19T14:36:23Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet 2470673 wikitext text/x-wiki mrhrzb9d8axtfzb2md2mxemh3ntajyq 2470674 2470673 2026-10-19T14:37:01Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7010.magru.wmnet 2470674 wikitext text/x-wiki sqab39vqbgsp375bygo25vn4j983mxj 2470675 2470674 2026-10-19T14:37:02Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp7010* or cp7002*} and A:cp 2470675 wikitext text/x-wiki fdf05j1tztlu2qg53dsp5j8vpnwmpk1 2470676 2470675 2026-10-19T14:37:10Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy7001.magru.wmnet 2470676 wikitext text/x-wiki 9vjt7k9jpmu5hayboo9yq5y3ojbgtvf 2470677 2470676 2026-10-19T14:37:24Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host tcp-proxy7002.magru.wmnet 2470677 wikitext text/x-wiki hny8gyhkrard0xuckggoxtthcjsxgkb 2470678 2470677 2026-10-19T14:37:54Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T441197)', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20261019-143751-marostegui.json 2470678 wikitext text/x-wiki llfa5lisa2enu50odr0qn63xesxbr3e 2470679 2470678 2026-10-19T14:38:21Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1356889|hCaptcha: Remove ResourceLoader dependency from HCaptchaOutput (T441681)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2470679 wikitext text/x-wiki j3lb7xf4h9svrjqntgo7qhq8xhki8tn 2470680 2470679 2026-10-19T14:39:02Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2470680 wikitext text/x-wiki mzs8z1rhts8ni1a1d4iutnjxmfh72qa 2470681 2470680 2026-10-19T14:39:29Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs 2470681 wikitext text/x-wiki b9upxgu5q4fhdkgmwl3lvo6dgjoxq1b 2470682 2470681 2026-10-19T14:40:43Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2470682 wikitext text/x-wiki gjolyf4dtnst54mewcrqo3p4q7ya1ai 2470683 2470682 2026-10-19T14:41:20Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2470683 wikitext text/x-wiki oe32qxf957rjdozy9pywhfdj3o7nwo1 2470684 2470683 2026-10-19T14:41:35Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2470684 wikitext text/x-wiki 1pyol25scdrwd989f6a4q5f483bdk9z 2470685 2470684 2026-10-19T14:41:39Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet 2470685 wikitext text/x-wiki 7ps5jhzj2jrob6xz3556lef4afkxl3d 2470686 2470685 2026-10-19T14:41:55Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs 2470686 wikitext text/x-wiki 6ddesj2c14bcx2gv68kuhniv2p9dfjl 2470687 2470686 2026-10-19T14:42:05Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy7002.magru.wmnet 2470687 wikitext text/x-wiki 32cb7blz9mnxy1qys2bjixyily6a0wh 2470688 2470687 2026-10-19T14:42:17Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams 2470688 wikitext text/x-wiki 7bu14fvgardko9vy9ypjni3m0p2z25e 2470689 2470688 2026-10-19T14:42:24Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2470689 wikitext text/x-wiki sd8ct6apchcqaj2tak0pn741c71n98g 2470690 2470689 2026-10-19T14:43:05Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply 2470690 wikitext text/x-wiki syfrvyjee6ntwibutum4knparjgkjxy 2470691 2470690 2026-10-19T14:43:57Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply 2470691 wikitext text/x-wiki efrr6iovvg5b4xdtdmt4535ny5zwlkc 2470692 2470691 2026-10-19T14:44:32Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356889|hCaptcha: Remove ResourceLoader dependency from HCaptchaOutput (T441681)]] (1.47.0-wmf.24) (duration: 11m 50s) 2470692 wikitext text/x-wiki nvor8uexpomv2eyfi38559lqayfl5r0 2470693 2470692 2026-10-19T14:44:34Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams 2470693 wikitext text/x-wiki 8egh7acv8b3a6yf1smzhmr8gd66zuon 2470694 2470693 2026-10-19T14:44:36Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2470694 wikitext text/x-wiki td1c5tns12fzbml7hdifkswam5rkes0 2470695 2470694 2026-10-19T14:44:45Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P96822 and previous config saved to /var/cache/conftool/dbconfig/20261019-144443-ladsgroup.json 2470695 wikitext text/x-wiki cqqr1x7rexivw8yffax7eby8b0p637r 2470696 2470695 2026-10-19T14:45:09Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2470696 wikitext text/x-wiki rz5hw3pvwh2njkqijtz9nkzj1f9gubs 2470697 2470696 2026-10-19T14:45:33Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2470697 wikitext text/x-wiki mz29hulp9a8g6ud3ucvuixt65yjamtn 2470698 2470697 2026-10-19T14:46:23Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2470698 wikitext text/x-wiki ohtn7bf6nrbjundyli9sobu90x9cx60 2470699 2470698 2026-10-19T14:46:27Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough 2470699 wikitext text/x-wiki plqs3b3uaf3bhrxyks9e4eynh7eqi8o 2470700 2470699 2026-10-19T14:46:54Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum 2470700 wikitext text/x-wiki lbk9d5fxuhpfrcxc38h18xco05j2ne1 2470701 2470700 2026-10-19T14:47:44Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org 2470701 wikitext text/x-wiki p63b3hn7vw33g8dm4def39t95rot9z2 2470702 2470701 2026-10-19T14:50:15Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy 2470702 wikitext text/x-wiki 8en3ned7ep90zrc44zcxdg8xlnmzig9 2470703 2470702 2026-10-19T14:50:22Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply 2470703 wikitext text/x-wiki iayiibbtt7chyzcvra4l3g72diselab 2470704 2470703 2026-10-19T14:51:01Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1356815|IS: Enable PersonalDashboard on test2wiki]] 2470704 wikitext text/x-wiki mbzy1r4mmfakyxdf39qzyq3621z46rq 2470705 2470704 2026-10-19T14:51:19Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply 2470705 wikitext text/x-wiki bawwx491ta4l1px04tipk58mj0mc8js 2470706 2470705 2026-10-19T14:52:55Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P96823 and previous config saved to /var/cache/conftool/dbconfig/20261019-145253-marostegui.json 2470706 wikitext text/x-wiki 339prddcq3sfd0akoqqn4mcqga4afxo 2470707 2470706 2026-10-19T14:53:00Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2470707 wikitext text/x-wiki ky0xdx067u7y7scefugth88tnp356qu 2470708 2470707 2026-10-19T14:53:29Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2470708 wikitext text/x-wiki 383ol5jtq3g8gcgx79ndfqoh569hzrh 2470709 2470708 2026-10-19T14:54:21Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org 2470709 wikitext text/x-wiki in9437i8t5f6nidtg1gm87ey6xtoheh 2470710 2470709 2026-10-19T14:54:35Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org 2470710 wikitext text/x-wiki nhbl3wp4d9sswui3qzr8gau62996yta 2470711 2470710 2026-10-19T14:54:47Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T441633)', diff saved to https://phabricator.wikimedia.org/P96824 and previous config saved to /var/cache/conftool/dbconfig/20261019-145445-ladsgroup.json 2470711 wikitext text/x-wiki 4vb6fsza8mo6you0lpak8t3ul0gzuxz 2470712 2470711 2026-10-19T14:55:03Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2470712 wikitext text/x-wiki cf4pbdyfvwhp5pap1j1oans2rfa2cvi 2470713 2470712 2026-10-19T14:55:05Z Stashbot 7414 samtar@deploy1003: samtar: Backport for [[gerrit:1356815|IS: Enable PersonalDashboard on test2wiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2470713 wikitext text/x-wiki qrg8cwrgjaytmvm6vq2zoe0ea2rfugm 2470714 2470713 2026-10-19T14:55:06Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance 2470714 wikitext text/x-wiki 23l07xpu2hyfaew09h2n7o025mmqd4t 2470715 2470714 2026-10-19T14:55:09Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2192 (T441633)', diff saved to https://phabricator.wikimedia.org/P96825 and previous config saved to /var/cache/conftool/dbconfig/20261019-145507-ladsgroup.json 2470715 wikitext text/x-wiki babp8j8vgnl2lrd7johvp567655tgge 2470716 2470715 2026-10-19T14:55:41Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2470716 wikitext text/x-wiki ozwwh51vk33fors3y02i4ea2w07rxu3 2470717 2470716 2026-10-19T14:55:57Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply 2470717 wikitext text/x-wiki oheo0djucmp1j3kn9xc95pxhdbkul66 2470718 2470717 2026-10-19T14:56:51Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply 2470718 wikitext text/x-wiki lp6t3is94ysygtikz37t8krpew50vv5 2470719 2470718 2026-10-19T14:57:50Z Stashbot 7414 samtar@deploy1003: samtar: Continuing with deployment 2470719 wikitext text/x-wiki q79c8nlys97o298vbrbaa55brto7ne9 2470720 2470719 2026-10-19T14:58:23Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster 2470720 wikitext text/x-wiki a4ws65hx3q5hiecfnm2zlcsnx8yj1zv 2470721 2470720 2026-10-19T14:58:24Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.presto.reboot-workers (exit_code=0) for Presto an-presto cluster: Reboot Presto nodes 2470721 wikitext text/x-wiki oifhzjlqlxisvek0yc1ed1kzvisp8a5 2470722 2470721 2026-10-19T14:58:49Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2470722 wikitext text/x-wiki i60p9tee07ilu9rhk3lbbwr2evthftc 2470723 2470722 2026-10-19T14:59:39Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes 2470723 wikitext text/x-wiki 4upmi9n90uud5yw96lwkvjd5k6zdpk3 2470724 2470723 2026-10-19T14:59:58Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2470724 wikitext text/x-wiki rawhe7rlfam2xcrznm5zuqn9q98xxvd 2470725 2470724 2026-10-19T15:00:52Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4005.ulsfo.wmnet 2470725 wikitext text/x-wiki 3lqj10brn890u03huarxim3ro4xylnq 2470726 2470725 2026-10-19T15:01:15Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1003.eqiad.wmnet 2470726 wikitext text/x-wiki rtvgsch3lsugasetqzwntgp50rwza2g 2470727 2470726 2026-10-19T15:01:18Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org 2470727 wikitext text/x-wiki j3d638mw88phge698jg23l4k41g8sr8 2470728 2470727 2026-10-19T15:01:32Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit2003.wikimedia.org 2470728 wikitext text/x-wiki q1buuc511rfkr5ue4c4qmnm8ym4i4l9 2470729 2470728 2026-10-19T15:01:41Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2470729 wikitext text/x-wiki hky44erda3fv2u63s1syudpr0madhbo 2470730 2470729 2026-10-19T15:01:51Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T441633)', diff saved to https://phabricator.wikimedia.org/P96826 and previous config saved to /var/cache/conftool/dbconfig/20261019-150149-ladsgroup.json 2470730 wikitext text/x-wiki 0dnl0zpqom60e31a4ogcab19x3gk28q 2470731 2470730 2026-10-19T15:02:17Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2470731 wikitext text/x-wiki mzcpe66683gb9f07km1pnbdvxs44e5h 2470732 2470731 2026-10-19T15:02:43Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply 2470732 wikitext text/x-wiki 7yhxac0weffanolhpqik3zhrwx8mcc3 2470733 2470732 2026-10-19T15:03:17Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356815|IS: Enable PersonalDashboard on test2wiki]] (duration: 12m 16s) 2470733 wikitext text/x-wiki i7eh1p9fngs6paa97he7rk2cy5mxim5 2470734 2470733 2026-10-19T15:03:22Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2470734 wikitext text/x-wiki nh3e0adqgyi8378v4cd98rozp8ejwcl 2470735 2470734 2026-10-19T15:04:05Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2470735 wikitext text/x-wiki pi4j9pb3gbfj8o9h4z4makdg6rthjf4 2470736 2470735 2026-10-19T15:04:38Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti4005.ulsfo.wmnet 2470736 wikitext text/x-wiki t5w8gu69wyoq793l0g6hrmac3mmqrwj 2470737 2470736 2026-10-19T15:04:44Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2470737 wikitext text/x-wiki rdw4m1tqbo4ge6f1j33luzsf366k8uk 2470738 2470737 2026-10-19T15:04:52Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2470738 wikitext text/x-wiki s1hb3hasj3h5u5eeuvgn0nsu9d4n2gh 2470739 2470738 2026-10-19T15:05:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet 2470739 wikitext text/x-wiki tej4pt39k7yabptux5c7enfpjecf3ha 2470740 2470739 2026-10-19T15:05:27Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2470740 wikitext text/x-wiki ez0at4xychv2u6mgsesvef3af75rbo6 2470741 2470740 2026-10-19T15:05:44Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2470741 wikitext text/x-wiki mr8vt4xs1hx7i7ikssff7spz5yanewc 2470742 2470741 2026-10-19T15:06:09Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2470742 wikitext text/x-wiki 9au2qx2usbhc9bdqal05hkdej6ekjqc 2470743 2470742 2026-10-19T15:06:31Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2470743 wikitext text/x-wiki dwky91u8cwjac7niucyhfx6lcna6ibj 2470744 2470743 2026-10-19T15:07:01Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2470744 wikitext text/x-wiki 389bvtzv5chuyxxv9zuhanpoj7lf2xr 2470746 2470744 2026-10-19T15:07:57Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P96827 and previous config saved to /var/cache/conftool/dbconfig/20261019-150755-marostegui.json 2470746 wikitext text/x-wiki nsas1vhow7xddv6lldqovo3z46lha3k 2470747 2470746 2026-10-19T15:08:12Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2470747 wikitext text/x-wiki lrrg62dwhwa25j9yynlid44thz4az3s 2470748 2470747 2026-10-19T15:08:14Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2003.wikimedia.org 2470748 wikitext text/x-wiki te6biqolx7pz1auqfyyaeame73etyn7 2470749 2470748 2026-10-19T15:08:27Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1006.eqiad.wmnet 2470749 wikitext text/x-wiki sig40cpmmp11m7df8zx4og4fknez24i 2470750 2470749 2026-10-19T15:08:39Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1003.eqiad.wmnet 2470750 wikitext text/x-wiki g2o4ut9gdxec9lly2g321ltqpqj7vud 2470751 2470750 2026-10-19T15:08:55Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2470751 wikitext text/x-wiki imv8xtbnbz0pboy3bw3yprc4omcodt2 2470752 2470751 2026-10-19T15:09:10Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2470752 wikitext text/x-wiki t6v3wu9cthpaf9e7qn4i5dk03511s21 2470753 2470752 2026-10-19T15:09:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1004.eqiad.wmnet 2470753 wikitext text/x-wiki 3r12b4d31mc6wpbmzh81mvttfs1btq6 2470754 2470753 2026-10-19T15:10:01Z Stashbot 7414 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. 2470754 wikitext text/x-wiki 8nyh3nxuh2m6qetcg01t48xphantldb 2470755 2470754 2026-10-19T15:10:25Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply 2470755 wikitext text/x-wiki d6lxfbm5w3921nf02hzvxjshmf5ufao 2470756 2470755 2026-10-19T15:10:30Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply 2470756 wikitext text/x-wiki 3pdk63g23t4m9ef2ktkk7pfawf68soe 2470757 2470756 2026-10-19T15:11:17Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet 2470757 wikitext text/x-wiki a2fh765eueu8nfg26eo32fs516iuzeq 2470758 2470757 2026-10-19T15:11:53Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P96828 and previous config saved to /var/cache/conftool/dbconfig/20261019-151151-ladsgroup.json 2470758 wikitext text/x-wiki ogyczjd4y78rlfz5suwgfc9g323n7gu 2470759 2470758 2026-10-19T15:11:54Z Stashbot 7414 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2470759 wikitext text/x-wiki klvfu9vp026cxvkklk8hb86vm355058 2470760 2470759 2026-10-19T15:13:14Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2470760 wikitext text/x-wiki 1at9cjvwlkm3rbxcvmtjxra56bh1ag7 2470761 2470760 2026-10-19T15:13:17Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . 2470761 wikitext text/x-wiki 87r4gpw7u03i4dhd3w3eg7274vkcqyo 2470762 2470761 2026-10-19T15:13:51Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1006.eqiad.wmnet 2470762 wikitext text/x-wiki l8qy9w8ec3bs8knnxhuzj5xcgtqpuij 2470763 2470762 2026-10-19T15:14:04Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1007.eqiad.wmnet 2470763 wikitext text/x-wiki 81d2wly4m3sj37kmnwmlj0na76pzuua 2470764 2470763 2026-10-19T15:14:48Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1004.eqiad.wmnet 2470764 wikitext text/x-wiki edzgebftbe93rki3de9zdmwkv2eucjo 2470765 2470764 2026-10-19T15:18:29Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1007.eqiad.wmnet 2470765 wikitext text/x-wiki hvv36s2qeyinsjckvr2864jw5aj56ms 2470766 2470765 2026-10-19T15:19:22Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2470766 wikitext text/x-wiki sxjrpi3xro2kw3f05r7wcgrz96zrv0b 2470767 2470766 2026-10-19T15:20:11Z Stashbot 7414 mpostoronca@deploy1003: Started scap sync-world: Backport for [[gerrit:1356799|SECURITY: AbuseReview: Respect hidden usernames in username filter (T439752)]] (1.47.0-wmf.24) 2470767 wikitext text/x-wiki qay9hxdjtas28dpy0ci8kdcubxkagza 2470768 2470767 2026-10-19T15:20:32Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2470768 wikitext text/x-wiki d5hbq4m63y9u0s6k9z2j0smfg8ssadk 2470769 2470768 2026-10-19T15:21:55Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P96830 and previous config saved to /var/cache/conftool/dbconfig/20261019-152153-ladsgroup.json 2470769 wikitext text/x-wiki 1nrmlvjhhth7loh9phxebehuj6iy5jh 2470770 2470769 2026-10-19T15:23:01Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T441197)', diff saved to https://phabricator.wikimedia.org/P96831 and previous config saved to /var/cache/conftool/dbconfig/20261019-152257-marostegui.json 2470770 wikitext text/x-wiki t20j5mnrpwtjqrd9qr12i7lv86ugdne 2470771 2470770 2026-10-19T15:23:14Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1252.eqiad.wmnet with reason: Maintenance 2470771 wikitext text/x-wiki 96462at2gxc93jarr0gvr5e71ilii6w 2470772 2470771 2026-10-19T15:23:18Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1252 (T441197)', diff saved to https://phabricator.wikimedia.org/P96832 and previous config saved to /var/cache/conftool/dbconfig/20261019-152315-marostegui.json 2470772 wikitext text/x-wiki qmfp5mtdim8wa5bgzz6fld7pph5c9mr 2470773 2470772 2026-10-19T15:24:14Z Stashbot 7414 mpostoronca@deploy1003: mpostoronca: Backport for [[gerrit:1356799|SECURITY: AbuseReview: Respect hidden usernames in username filter (T439752)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2470773 wikitext text/x-wiki rd2vxru5izcc4af0i3aoglwn673syn2 2470774 2470773 2026-10-19T15:25:00Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2470774 wikitext text/x-wiki 1r4qshvn8rrqmfgzuax3dqnvwoirqa3 2470775 2470774 2026-10-19T15:26:09Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2470775 wikitext text/x-wiki sbb2i8poy2pv6hd2mkuxrc1fgib0qcl 2470776 2470775 2026-10-19T15:26:27Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti4005.ulsfo.wmnet 2470776 wikitext text/x-wiki n3n3ucydktejeybajbf2jlg8bnu09rf 2470777 2470776 2026-10-19T15:26:28Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti4005.ulsfo.wmnet 2470777 wikitext text/x-wiki 52tkee08pcoemrfhzx9yp20srfmrx3r 2470778 2470777 2026-10-19T15:26:42Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2470778 wikitext text/x-wiki i05tsl2buca5jamewfg2r4hkkak6uhs 2470779 2470778 2026-10-19T15:26:49Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2470779 wikitext text/x-wiki s1ytjhd92yrymc5c9mjv433hpkozvvp 2470780 2470779 2026-10-19T15:26:55Z Stashbot 7414 mpostoronca@deploy1003: mpostoronca: Continuing with deployment 2470780 wikitext text/x-wiki 03gidtwpmvne5qhc8ad0hs15s3pk0mh 2470781 2470780 2026-10-19T15:27:20Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2470781 wikitext text/x-wiki ehg9zddqiifbv8nl9t4pilq27so255e 2470782 2470781 2026-10-19T15:27:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint1001.eqiad.wmnet 2470782 wikitext text/x-wiki 8i45ubfk14bgrjg7678ulq492poq36t 2470783 2470782 2026-10-19T15:28:03Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2470783 wikitext text/x-wiki cx0i7b10izmehz1e3qltajmddatief6 2470784 2470783 2026-10-19T15:28:37Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2470784 wikitext text/x-wiki nyceilxwzyc0c2qr1jhv09b2pdxu5wq 2470785 2470784 2026-10-19T15:28:59Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2470785 wikitext text/x-wiki rix4unae3966y67gk0xkye2ud3umw18 2470786 2470785 2026-10-19T15:29:57Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2470786 wikitext text/x-wiki 1ciguhrvmv2v2zy11j01nzklf7ad6tq 2470787 2470786 2026-10-19T15:30:50Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2470787 wikitext text/x-wiki 3ebframcrq4ck8hyvbc9doravl7qv6h 2470788 2470787 2026-10-19T15:31:20Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint1001.eqiad.wmnet 2470788 wikitext text/x-wiki 4iqrooucgmfsixwypum5r9n3aekwevo 2470789 2470788 2026-10-19T15:31:38Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2470789 wikitext text/x-wiki 1xgmeikklzjh43wm4pgv2yxllvjg6o9 2470790 2470789 2026-10-19T15:31:57Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T441633)', diff saved to https://phabricator.wikimedia.org/P96833 and previous config saved to /var/cache/conftool/dbconfig/20261019-153155-ladsgroup.json 2470790 wikitext text/x-wiki b0kxem872lel8g1vzj1cwfihkse4aw7 2470791 2470790 2026-10-19T15:32:17Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: Maintenance 2470791 wikitext text/x-wiki nwp27u5zwx16x66fj92td3jqw4433kl 2470792 2470791 2026-10-19T15:32:18Z Stashbot 7414 mpostoronca@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356799|SECURITY: AbuseReview: Respect hidden usernames in username filter (T439752)]] (1.47.0-wmf.24) (duration: 12m 12s) 2470792 wikitext text/x-wiki 092jqqfujwgjykifv3op6iuy7k6032g 2470793 2470792 2026-10-19T15:32:23Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T441197)', diff saved to https://phabricator.wikimedia.org/P96834 and previous config saved to /var/cache/conftool/dbconfig/20261019-153219-marostegui.json 2470793 wikitext text/x-wiki t1brsos85i6htxrryl9txsy1yu9s4h5 2470794 2470793 2026-10-19T15:32:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint2001.codfw.wmnet 2470794 wikitext text/x-wiki gtlj8k183o59pdip63aubkzflv80v3r 2470796 2470794 2026-10-19T15:33:25Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2470796 wikitext text/x-wiki ekptup9buxgbfkqenucesrnp9k1bo2r 2470797 2470796 2026-10-19T15:34:08Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2470797 wikitext text/x-wiki fd9b7ly60jjuciw65vw98ox1ymz6el3 2470798 2470797 2026-10-19T15:34:39Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.300.0" for 3 host(s) 2470798 wikitext text/x-wiki a5pv59r9eil2v64o03nooh8ibdur0e8 2470799 2470798 2026-10-19T15:36:02Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint2001.codfw.wmnet 2470799 wikitext text/x-wiki fdtqwt8ne3rmf4hmdjsuxs0grgrp46z 2470800 2470799 2026-10-19T15:36:34Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.300.0" completed for 3 hosts 2470800 wikitext text/x-wiki ejnnttk9j3bxrepeaftoh6k4k50uu4y 2470801 2470800 2026-10-19T15:37:35Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance 2470801 wikitext text/x-wiki om0b1edsforltamzd0uwad8pgl6iy9q 2470802 2470801 2026-10-19T15:37:37Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2211 (T441633)', diff saved to https://phabricator.wikimedia.org/P96835 and previous config saved to /var/cache/conftool/dbconfig/20261019-153735-ladsgroup.json 2470802 wikitext text/x-wiki 0sso2lnyfxtztfvm2j43cmwl59s8olu 2470808 2470802 2026-10-19T15:43:19Z Stashbot 7414 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1356905|Bumping portals to master (T128546)]] 2470808 wikitext text/x-wiki djzahxggm2gk7n2uzk2alx2yu75itk0 2470813 2470808 2026-10-19T15:44:31Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T441633)', diff saved to https://phabricator.wikimedia.org/P96836 and previous config saved to /var/cache/conftool/dbconfig/20261019-154429-ladsgroup.json 2470813 wikitext text/x-wiki 1776qqa17kel9ctsj1ynjqtm2i5qp8q 2470814 2470813 2026-10-19T15:46:28Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet 2470814 wikitext text/x-wiki lmk96bpml73pryddhn8hc6f3dgp2y4k 2470815 2470814 2026-10-19T15:47:23Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P96837 and previous config saved to /var/cache/conftool/dbconfig/20261019-154721-marostegui.json 2470815 wikitext text/x-wiki dzlb5chpvrh4wxlwcg4xg93qo41qrc0 2470816 2470815 2026-10-19T15:47:27Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1356905|Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2470816 wikitext text/x-wiki p54lbpy9zl9ybi63f73tcwgkm7yiw2e 2470817 2470816 2026-10-19T15:50:49Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Continuing with deployment 2470817 wikitext text/x-wiki ehr3tq1ciizo3y1jkycknmsfmil8npc 2470818 2470817 2026-10-19T15:54:33Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P96838 and previous config saved to /var/cache/conftool/dbconfig/20261019-155431-ladsgroup.json 2470818 wikitext text/x-wiki 4y5s4yxjvjkoalzdjcbfs51j532a2rz 2470819 2470818 2026-10-19T15:55:36Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup1003.eqiad.wmnet 2470819 wikitext text/x-wiki jcu0e8rcuupszu2u08rs65ituwdej8h 2470820 2470819 2026-10-19T15:55:50Z Stashbot 7414 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ms-backup1003.eqiad.wmnet 2470820 wikitext text/x-wiki 6g9j8k2nm774jqf4nlxufbgzo5klxwf 2470821 2470820 2026-10-19T15:56:12Z Stashbot 7414 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356905|Bumping portals to master (T128546)]] (duration: 12m 54s) 2470821 wikitext text/x-wiki d7d2a06iyk3lukf86t1n5xftfjm9kv1 2470822 2470821 2026-10-19T15:56:58Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup1003.eqiad.wmnet 2470822 wikitext text/x-wiki hkw17xvechk9shvahbalxpzya58lp90 2470823 2470822 2026-10-19T15:59:35Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-backup1003.eqiad.wmnet 2470823 wikitext text/x-wiki n1fr72ugie5lm3kbavqmm9yuy5diqw1 2470824 2470823 2026-10-19T16:02:25Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P96839 and previous config saved to /var/cache/conftool/dbconfig/20261019-160224-marostegui.json 2470824 wikitext text/x-wiki c3ctyndrl6ir0tfnovf8000amjh1wul 2470825 2470824 2026-10-19T16:03:08Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup2003.codfw.wmnet 2470825 wikitext text/x-wiki mapyb022edifv96n9gxpkbrh3jbk1n4 2470826 2470825 2026-10-19T16:04:34Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P96840 and previous config saved to /var/cache/conftool/dbconfig/20261019-160433-ladsgroup.json 2470826 wikitext text/x-wiki t9nrqojwpfoz960rtvhsfty3g315ml1 2470827 2470826 2026-10-19T16:05:57Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-backup2003.codfw.wmnet 2470827 wikitext text/x-wiki pbllis4gycbmonrmsri580os8lddkxv 2470828 2470827 2026-10-19T16:11:27Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid analytics cluster: Reboot Druid nodes 2470828 wikitext text/x-wiki e2clyx71s0b2uzk7gkklwt1zzf4s1ok 2470829 2470828 2026-10-19T16:14:00Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup2004.codfw.wmnet 2470829 wikitext text/x-wiki hygs45f4f522eqmaenipw26hfbkpvco 2470830 2470829 2026-10-19T16:14:37Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T441633)', diff saved to https://phabricator.wikimedia.org/P96841 and previous config saved to /var/cache/conftool/dbconfig/20261019-161435-ladsgroup.json 2470830 wikitext text/x-wiki lb97cvznn4prb6egzipkf3xu4j9qz89 2470831 2470830 2026-10-19T16:14:45Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host ms-backup1004.eqiad.wmnet 2470831 wikitext text/x-wiki hg30x98l78zuyzi6ghjlrlfe4zqnr7q 2470832 2470831 2026-10-19T16:14:56Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance 2470832 wikitext text/x-wiki egyp0d0ju86rone0hr462h32z07cpmz 2470833 2470832 2026-10-19T16:14:58Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2213 (T441633)', diff saved to https://phabricator.wikimedia.org/P96842 and previous config saved to /var/cache/conftool/dbconfig/20261019-161456-ladsgroup.json 2470833 wikitext text/x-wiki 85dr7xv1rt0r4ublwkk2tsswrrk097j 2470834 2470833 2026-10-19T16:16:42Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes 2470834 wikitext text/x-wiki 69ka57fty7h4g1usv6xjsf271fmmgb6 2470835 2470834 2026-10-19T16:16:49Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-backup2004.codfw.wmnet 2470835 wikitext text/x-wiki 93fj5c9mtnzgwgjoqz5i8sjcfhni0xd 2470836 2470835 2026-10-19T16:17:29Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T441197)', diff saved to https://phabricator.wikimedia.org/P96843 and previous config saved to /var/cache/conftool/dbconfig/20261019-161725-marostegui.json 2470836 wikitext text/x-wiki 776lc26vvuxze7qgll3hjnqv9ovoih8 2470837 2470836 2026-10-19T16:17:33Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-backup1004.eqiad.wmnet 2470837 wikitext text/x-wiki 6o1h6x43fn4d6vyurzfxgrhnhu99835 2470838 2470837 2026-10-19T16:17:42Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1265.eqiad.wmnet with reason: Maintenance 2470838 wikitext text/x-wiki ncxbsc4dyfdp55cg0lxj8awr341h36i 2470839 2470838 2026-10-19T16:21:45Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance 2470839 wikitext text/x-wiki 0alxsndju23gi33nc70fur54ut18gpf 2470840 2470839 2026-10-19T16:21:48Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T441633)', diff saved to https://phabricator.wikimedia.org/P96844 and previous config saved to /var/cache/conftool/dbconfig/20261019-162147-ladsgroup.json 2470840 wikitext text/x-wiki 26rxs6xyynem2e7ombl08fbzg8ztmj4 2470841 2470840 2026-10-19T16:25:39Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1015.eqiad.wmnet 2470841 wikitext text/x-wiki s6f96ae44qb34f5qvqwobtiv95wjsqk 2470842 2470841 2026-10-19T16:26:48Z Stashbot 7414 herron@cumin1004: START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:logstash-codfw 2470842 wikitext text/x-wiki jhdahf97iq3uobi55eck91ojnkae7ye 2470843 2470842 2026-10-19T16:28:38Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1015.eqiad.wmnet 2470843 wikitext text/x-wiki fq4mh3qnz9c8sdwces8mwuew7k8u98e 2470844 2470843 2026-10-19T16:31:50Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P96845 and previous config saved to /var/cache/conftool/dbconfig/20261019-163148-ladsgroup.json 2470844 wikitext text/x-wiki m07zu3gowtcqiti7cdpur9ubitmmqra 2470845 2470844 2026-10-19T16:33:03Z Stashbot 7414 kamila@deploy1003: Started scap deploy-service shellbox: roll out PHP 8.5 for -timeline and -video 2470845 wikitext text/x-wiki kytenbecf4ezsfjitxcqqet5iarcrxc 2470846 2470845 2026-10-19T16:33:05Z Stashbot 7414 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2470846 wikitext text/x-wiki pdw72m4hz4503fzweqq981tq9j04dgd 2470847 2470846 2026-10-19T16:33:52Z Stashbot 7414 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2470847 wikitext text/x-wiki cbh5fph35evljah3iqncck36y19rydo 2470848 2470847 2026-10-19T16:35:24Z Stashbot 7414 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2470848 wikitext text/x-wiki clad95i36eci2ocmvozjssruc1cgm6a 2470850 2470848 2026-10-19T16:38:01Z Stashbot 7414 kamila@deploy1003: Finished scap deploy-service shellbox: roll out PHP 8.5 for -timeline and -video (duration: 04m 57s) 2470850 wikitext text/x-wiki a28hf0j03zx9xiydg6vadbr3b9nbmv0 2470852 2470850 2026-10-19T16:41:52Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P96846 and previous config saved to /var/cache/conftool/dbconfig/20261019-164150-ladsgroup.json 2470852 wikitext text/x-wiki svyd7jok5zlqukyb0cjuzvudu7h4p19 2470854 2470852 2026-10-19T16:51:54Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T441633)', diff saved to https://phabricator.wikimedia.org/P96847 and previous config saved to /var/cache/conftool/dbconfig/20261019-165152-ladsgroup.json 2470854 wikitext text/x-wiki oghe3b5azpjfla3bqbfrhg0jao31b2h 2470855 2470854 2026-10-19T16:52:13Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance 2470855 wikitext text/x-wiki onr3gv04bevxq9icejno6gh5pcys9o3 2470856 2470855 2026-10-19T16:52:16Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2223 (T441633)', diff saved to https://phabricator.wikimedia.org/P96848 and previous config saved to /var/cache/conftool/dbconfig/20261019-165214-ladsgroup.json 2470856 wikitext text/x-wiki 9mvxdhx5aoooxi940lo334gliwjn0i6 2470858 2470856 2026-10-19T16:58:44Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T441633)', diff saved to https://phabricator.wikimedia.org/P96849 and previous config saved to /var/cache/conftool/dbconfig/20261019-165842-ladsgroup.json 2470858 wikitext text/x-wiki dn6ipi6chrpl12t6boes0hgf6c8wbyy 2470859 2470858 2026-10-19T17:03:35Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1221.eqiad.wmnet with reason: Maintenance 2470859 wikitext text/x-wiki ie5trg294cnoru9qji3919myfha3uh7 2470860 2470859 2026-10-19T17:03:45Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on 6 hosts with reason: Maintenance 2470860 wikitext text/x-wiki hkh8mdhvsvymv87hgj3f6o95wde5u9x 2470861 2470860 2026-10-19T17:03:48Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96850 and previous config saved to /var/cache/conftool/dbconfig/20261019-170345-marostegui.json 2470861 wikitext text/x-wiki 59qmqnakggzmogyqdvubx7wowpwrn0h 2470862 2470861 2026-10-19T17:04:10Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2015.codfw.wmnet 2470862 wikitext text/x-wiki h7hnxqh7vb7bp23it1hch8c60vo8y7q 2470863 2470862 2026-10-19T17:06:56Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1016.eqiad.wmnet 2470863 wikitext text/x-wiki 6wcd54y4geb4tp7y5f82qjtizjrfcro 2470864 2470863 2026-10-19T17:06:58Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2015.codfw.wmnet 2470864 wikitext text/x-wiki 1k4uxv5ysq8hbmjrkwcxb9ikwt0jlj5 2470865 2470864 2026-10-19T17:07:57Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2016.codfw.wmnet 2470865 wikitext text/x-wiki 6273sj3rovgui2trbemjbws6deqw3xz 2470866 2470865 2026-10-19T17:08:45Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P96851 and previous config saved to /var/cache/conftool/dbconfig/20261019-170844-ladsgroup.json 2470866 wikitext text/x-wiki 5flv5z8qiutlzshkgw0jairlilkxiis 2470867 2470866 2026-10-19T17:09:45Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1016.eqiad.wmnet 2470867 wikitext text/x-wiki mzwl1f1mq0dtyea6xvgkn55cmllcc3p 2470868 2470867 2026-10-19T17:10:48Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1017.eqiad.wmnet 2470868 wikitext text/x-wiki 81b02fn7reeml5lo1vcuakcdvzuqkcy 2470869 2470868 2026-10-19T17:10:52Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2016.codfw.wmnet 2470869 wikitext text/x-wiki tep6igdx68c6j9v69adjxd98ybe20sm 2470870 2470869 2026-10-19T17:11:39Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96852 and previous config saved to /var/cache/conftool/dbconfig/20261019-171136-marostegui.json 2470870 wikitext text/x-wiki 3lhxr147w3olypyc1jz7dksug4rdzxc 2470871 2470870 2026-10-19T17:11:56Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2017.codfw.wmnet 2470871 wikitext text/x-wiki ru87ftbzxkl86x8bnguqj0p2jyxski2 2470872 2470871 2026-10-19T17:13:39Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1017.eqiad.wmnet 2470872 wikitext text/x-wiki 2u0eiwune5bbpdh44yjbs1skle4f65s 2470873 2470872 2026-10-19T17:14:42Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1018.eqiad.wmnet 2470873 wikitext text/x-wiki j7ne8ww130zlr5f9j7i91j5xcmddbpc 2470874 2470873 2026-10-19T17:14:52Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2017.codfw.wmnet 2470874 wikitext text/x-wiki mk493gyzv0l1qnfmg38ej0g1n2bozuo 2470875 2470874 2026-10-19T17:15:55Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2018.codfw.wmnet 2470875 wikitext text/x-wiki euwhg93uvaqkodeqm73ifmruglpf7yh 2470876 2470875 2026-10-19T17:17:38Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1018.eqiad.wmnet 2470876 wikitext text/x-wiki 7guzyzxszd0okoo54xh5gvlxjez5som 2470877 2470876 2026-10-19T17:18:42Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1019.eqiad.wmnet 2470877 wikitext text/x-wiki gd2dmn6hlteb04ziss53xhc3loj26uj 2470878 2470877 2026-10-19T17:18:47Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P96853 and previous config saved to /var/cache/conftool/dbconfig/20261019-171846-ladsgroup.json 2470878 wikitext text/x-wiki icbludoipvqa91debhke0dczcfp4o7u 2470879 2470878 2026-10-19T17:18:58Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2018.codfw.wmnet 2470879 wikitext text/x-wiki nl3d3l8w36bswimec88vov2mraxql5b 2470880 2470879 2026-10-19T17:20:02Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2019.codfw.wmnet 2470880 wikitext text/x-wiki 1ifh146evhp8vqtqehyzbruebstvxj8 2470881 2470880 2026-10-19T17:21:31Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1019.eqiad.wmnet 2470881 wikitext text/x-wiki beskw7lm9gurmtwcnnguauoh0okk7jl 2470882 2470881 2026-10-19T17:22:27Z Stashbot 7414 herron@deploy1003: Started scap sync-world: bump rsyslog to 8.2608.0-4 T438375 2470882 wikitext text/x-wiki 89j9ehs6o66ds2e3i1dvvr23ohc5oib 2470883 2470882 2026-10-19T17:22:34Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1020.eqiad.wmnet 2470883 wikitext text/x-wiki 1c7bh49z0vhrdznphe8ghvw5zgst4yh 2470884 2470883 2026-10-19T17:22:58Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2019.codfw.wmnet 2470884 wikitext text/x-wiki 7igg3ukjys2ov00zsekobwkgl80fhp8 2470885 2470884 2026-10-19T17:23:36Z Stashbot 7414 dzahn@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet 2470885 wikitext text/x-wiki evtg1ovg8jx58mc2b59mc6ucmwz5x66 2470886 2470885 2026-10-19T17:23:50Z Stashbot 7414 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host zuul1004.eqiad.wmnet 2470886 wikitext text/x-wiki hsaod0gl6p279mrnva4owbrsa3sftxb 2470887 2470886 2026-10-19T17:24:02Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2020.codfw.wmnet 2470887 wikitext text/x-wiki tiipgvaywos5obdw45anbouwcqgg9v2 2470888 2470887 2026-10-19T17:25:31Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1020.eqiad.wmnet 2470888 wikitext text/x-wiki 4ggk847ewk2zcxgyhxxm9ud4tym6hkf 2470889 2470888 2026-10-19T17:26:39Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P96854 and previous config saved to /var/cache/conftool/dbconfig/20261019-172638-marostegui.json 2470889 wikitext text/x-wiki aaxmdymcihmto1rhbhzqcpoii4wpsbj 2470890 2470889 2026-10-19T17:27:10Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2020.codfw.wmnet 2470890 wikitext text/x-wiki t34d23e477g9okr7cja9tfoul59aa4g 2470891 2470890 2026-10-19T17:27:20Z Stashbot 7414 dzahn@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1005.eqiad.wmnet 2470891 wikitext text/x-wiki 8ztbxio7e6ldpo4su8gwbudljvqgobh 2470892 2470891 2026-10-19T17:27:33Z Stashbot 7414 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host zuul1005.eqiad.wmnet 2470892 wikitext text/x-wiki kffh4qhyr14dpruaflxdo4zvj6zq2qr 2470893 2470892 2026-10-19T17:28:11Z Stashbot 7414 mutante: sre.host.reboot-single fails entirely if the linked task isn't public so it cant update it 2470893 wikitext text/x-wiki k6dw0juz6ipg8fv3hagus7ozt3wx9bn 2470894 2470893 2026-10-19T17:28:31Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid public cluster: Reboot Druid nodes 2470894 wikitext text/x-wiki 9egb9ufq46i6wlda8sydgp34ggr72bx 2470895 2470894 2026-10-19T17:28:49Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T441633)', diff saved to https://phabricator.wikimedia.org/P96855 and previous config saved to /var/cache/conftool/dbconfig/20261019-172847-ladsgroup.json 2470895 wikitext text/x-wiki hsafpds6fop5qcruhrcmku8m6abpfwb 2470896 2470895 2026-10-19T17:31:24Z Stashbot 7414 herron@deploy1003: Finished scap sync-world: bump rsyslog to 8.2608.0-4 T438375 (duration: 10m 45s) 2470896 wikitext text/x-wiki 5594gbl1s6sirzbpwkedzj8jhzxyzcm 2470897 2470896 2026-10-19T17:31:33Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1356887|GlobalFunctions: Remove Microsoft IIS 7.5 workaround from wfUrlencode (T441517)]] (1.47.0-wmf.24), [[gerrit:1356885|VisualEditorResourceModule: Remove unused Message::exists() check (T441253)]] (1.47.0-wmf.24) 2470897 wikitext text/x-wiki bkdbhj54a3xs1gj36twhierg0c89te7 2470899 2470897 2026-10-19T17:35:47Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1356887|GlobalFunctions: Remove Microsoft IIS 7.5 workaround from wfUrlencode (T441517)]] (1.47.0-wmf.24), [[gerrit:1356885|VisualEditorResourceModule: Remove unused Message::exists() check (T441253)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2470899 wikitext text/x-wiki bhqk2ozwi3fc8mksf3dvmweic5vlhq8 2470900 2470899 2026-10-19T17:37:56Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2470900 wikitext text/x-wiki 1l0o8713mgdeb456ypsagk8eynf8qc7 2470901 2470900 2026-10-19T17:41:42Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P96856 and previous config saved to /var/cache/conftool/dbconfig/20261019-174140-marostegui.json 2470901 wikitext text/x-wiki 0bkc8ee212ewhujaodk8smybjz1dye5 2470902 2470901 2026-10-19T17:43:23Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356887|GlobalFunctions: Remove Microsoft IIS 7.5 workaround from wfUrlencode (T441517)]] (1.47.0-wmf.24), [[gerrit:1356885|VisualEditorResourceModule: Remove unused Message::exists() check (T441253)]] (1.47.0-wmf.24) (duration: 11m 49s) 2470902 wikitext text/x-wiki 1j0zgcuzqnnvkix7ldv9ds4q1o029oh 2470903 2470902 2026-10-19T17:56:45Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T441197)', diff saved to https://phabricator.wikimedia.org/P96857 and previous config saved to /var/cache/conftool/dbconfig/20261019-175642-marostegui.json 2470903 wikitext text/x-wiki pl0bzq72v54q73qxoqnnhsaykugzgp9 2470908 2470903 2026-10-19T18:59:31Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2470908 wikitext text/x-wiki 1z15e6is70k8z9qncy4o1l71y0yizp1 2470909 2470908 2026-10-19T19:06:34Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2470909 wikitext text/x-wiki iztdzllbfxuucolayowf50rfw3jjkrs 2470910 2470909 2026-10-19T19:19:03Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2470910 wikitext text/x-wiki qny6lfkz1u0q7zp7ibs9n2t66y2itxb 2470911 2470910 2026-10-19T19:19:40Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2470911 wikitext text/x-wiki a4q7orn41zw7ja4zudqmlg4wvs5xfll 2470912 2470911 2026-10-19T19:19:42Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2470912 wikitext text/x-wiki 519erieb5mtejen2zgcjlruuyx0cupt 2470913 2470912 2026-10-19T19:20:24Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2470913 wikitext text/x-wiki h97554k6f4wb2cupxf887twkr1lzqxx 2470918 2470913 2026-10-19T20:05:42Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1356949|Ignore lints when parsing non-lintable content models (T441307)]] (1.47.0-wmf.24) 2470918 wikitext text/x-wiki rhvjffqo3gaqyb4o07bxw4xvwbdfrlx 2470919 2470918 2026-10-19T20:09:45Z Stashbot 7414 arlolra@deploy1003: arlolra: Backport for [[gerrit:1356949|Ignore lints when parsing non-lintable content models (T441307)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2470919 wikitext text/x-wiki jf8saawc3nybk0vp17n2z86ql9fq7ir 2470920 2470919 2026-10-19T20:10:20Z Stashbot 7414 arlolra@deploy1003: arlolra: Continuing with deployment 2470920 wikitext text/x-wiki p9jm7yfvj1eqmguspvext0ft1ywk3vd 2470921 2470920 2026-10-19T20:15:42Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356949|Ignore lints when parsing non-lintable content models (T441307)]] (1.47.0-wmf.24) (duration: 09m 59s) 2470921 wikitext text/x-wiki fy315lx4jpg1sjcubdon2r1uos2mf2c 2470922 2470921 2026-10-19T20:17:26Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2470922 wikitext text/x-wiki dr8v0xl2vxhk9i3rlujo0n8cqnl2dqm 2470923 2470922 2026-10-19T20:17:44Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2470923 wikitext text/x-wiki 92ijlhcogx774ex6zbo2ravchv5084u 2470924 2470923 2026-10-19T20:22:05Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1356988|OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123)]] (1.47.0-wmf.24) 2470924 wikitext text/x-wiki 41qbwejr653nd5xon5xmucp1wijp8jn 2470925 2470924 2026-10-19T20:26:09Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1356988|OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2470925 wikitext text/x-wiki ar9xhxvm7e3ynpq9ufncyowiurrijiy 2470926 2470925 2026-10-19T20:27:46Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2470926 wikitext text/x-wiki 6bajq9c0gpsy3120w247kqzpkns57cg 2470927 2470926 2026-10-19T20:28:08Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2470927 wikitext text/x-wiki a108pn3v2jikyfmcu71e9toqmphsz77 2470928 2470927 2026-10-19T20:28:51Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2470928 wikitext text/x-wiki 0ss7ahv4lbpm7v97m1v1r17u65zf9a7 2470931 2470928 2026-10-19T20:34:13Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356988|OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123)]] (1.47.0-wmf.24) (duration: 12m 07s) 2470931 wikitext text/x-wiki iyt4eu6spy3f8tljdb0jkk1wbvyoqqg 2470932 2470931 2026-10-19T20:35:32Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet 2470932 wikitext text/x-wiki q3prfuakz769ee0waceuqouuafltxos 2470933 2470932 2026-10-19T20:37:44Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema 2470933 wikitext text/x-wiki ggdoigid75gd2512ly8gqwmqyeenz87 2470934 2470933 2026-10-19T20:39:29Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1001.eqiad.wmnet 2470934 wikitext text/x-wiki 1hv711cmskme4cj77x7r3sh0m4h7x4u 2470935 2470934 2026-10-19T20:40:01Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1002.eqiad.wmnet 2470935 wikitext text/x-wiki pwc9qgxgr8rp5vvagesjzrd4eooxs6i 2470936 2470935 2026-10-19T20:43:50Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2470936 wikitext text/x-wiki 6xuc5mut9d1g7klfbbfuldte1hehpsu 2470937 2470936 2026-10-19T20:44:00Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1002.eqiad.wmnet 2470937 wikitext text/x-wiki 7eg7em07fdmov7007kpo473dp5ayazs 2470938 2470937 2026-10-19T20:44:12Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2470938 wikitext text/x-wiki eju2gbqfibh941ldmjlu7nsqloluxm4 2470939 2470938 2026-10-19T20:45:20Z Stashbot 7414 dwisehaupt@dns1005: START - running authdns-update 2470939 wikitext text/x-wiki atb38ytttdne7fcgq4wnafafxkui3n3 2470940 2470939 2026-10-19T20:47:41Z Stashbot 7414 dwisehaupt@dns1005: END - running authdns-update 2470940 wikitext text/x-wiki bhpe938alf0rm1g4uytwiec5d8m85cv 2470941 2470940 2026-10-19T20:47:45Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1003.eqiad.wmnet 2470941 wikitext text/x-wiki ltopy8xm5ly8iiuwxtyrtxqwssxpu76 2470942 2470941 2026-10-19T20:49:22Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.301.0" for 3 host(s) 2470942 wikitext text/x-wiki r6w57d9vr7zry4ejmst5f9sdrbdzjrr 2470943 2470942 2026-10-19T20:51:17Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.301.0" completed for 3 hosts 2470943 wikitext text/x-wiki 3st68xj68wsdbltqtvvuv5qa7ay5w4w 2470944 2470943 2026-10-19T20:51:39Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1003.eqiad.wmnet 2470944 wikitext text/x-wiki hezjicvx6mg8utkvnv6zr8mx4mqw74q 2470945 2470944 2026-10-19T20:52:53Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2001.codfw.wmnet 2470945 wikitext text/x-wiki d05zoyos0azwv6f15vy4oep70fbr0ak 2470946 2470945 2026-10-19T20:54:24Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema 2470946 wikitext text/x-wiki 2nmouahip4p5s74tvy3wl5atwnu38bj 2470947 2470946 2026-10-19T20:54:41Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards 2470947 wikitext text/x-wiki 3mxe8pee0ifpzqaaeb2mfpg0ubhpm2u 2470948 2470947 2026-10-19T20:55:07Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards 2470948 wikitext text/x-wiki htfl0lvz0kxo8c5ertadkg3wsefyz54 2470949 2470948 2026-10-19T20:56:43Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2001.codfw.wmnet 2470949 wikitext text/x-wiki gle8xrfist0ma660em0ur07hggk4x56 2470950 2470949 2026-10-19T20:56:57Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2002.codfw.wmnet 2470950 wikitext text/x-wiki h6a87u0ypow3z4ji837dxyb8bvie1ve 2470951 2470950 2026-10-19T20:59:13Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet 2470951 wikitext text/x-wiki 4qeve0yrcl6zc23pw5lav0iaji3hhl6 2470952 2470951 2026-10-19T21:00:24Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards 2470952 wikitext text/x-wiki 2mlmyhfqza3tdn8ug9hmeqsqzmk1a3r 2470953 2470952 2026-10-19T21:00:31Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards 2470953 wikitext text/x-wiki 16rx9wklb1yz48isnzwzul3bc81g6l1 2470954 2470953 2026-10-19T21:00:58Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2002.codfw.wmnet 2470954 wikitext text/x-wiki 7i5yftgzipawuphvcgszlgw95mtzacd 2470955 2470954 2026-10-19T21:02:22Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2003.codfw.wmnet 2470955 wikitext text/x-wiki bms1apf61k3sjpdty0n9kcnxpzvumk9 2470956 2470955 2026-10-19T21:02:53Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards 2470956 wikitext text/x-wiki tfm6pzuekx3e9kqmek7h52n4o3jf2rl 2470957 2470956 2026-10-19T21:04:02Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet 2470957 wikitext text/x-wiki 565fvt8d5isu2zjgsxibndiskbxuw15 2470958 2470957 2026-10-19T21:04:32Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl2001.codfw.wmnet 2470958 wikitext text/x-wiki tb7rymqr526hpfc8szp80p5njthodvl 2470959 2470958 2026-10-19T21:06:23Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2003.codfw.wmnet 2470959 wikitext text/x-wiki f4ae7wjbbd1kzwh2fknri7ocd21cbmc 2470960 2470959 2026-10-19T21:08:56Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet 2470960 wikitext text/x-wiki 6bfwc4ev9h1r1kvpr428lcifpbfk0yh 2470961 2470960 2026-10-19T21:09:33Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl2001.codfw.wmnet 2470961 wikitext text/x-wiki mp50a17j5w42wcc7p51u14vw76o6vqb 2470962 2470961 2026-10-19T21:09:43Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet 2470962 wikitext text/x-wiki 7xbyhafns4ls6n99gore0txvxpybf97 2470964 2470962 2026-10-19T21:12:43Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet 2470964 wikitext text/x-wiki 0iyn6xnfv4khkdbq6tduqo1l9b3zih6 2470965 2470964 2026-10-19T21:14:19Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet 2470965 wikitext text/x-wiki 1dxx0qqp7033bdzmzme43kfxdcv9vxf 2470966 2470965 2026-10-19T21:14:41Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet 2470966 wikitext text/x-wiki 8v41nehzqud61pjct8t4q8qiy2i8p2c 2470967 2470966 2026-10-19T21:18:06Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet 2470967 wikitext text/x-wiki hu4av1x3pyt4t3swoisj8lbo6yb0lqv 2470968 2470967 2026-10-19T21:23:53Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl2002.codfw.wmnet 2470968 wikitext text/x-wiki a1yylq5yucyb3chgrqgs8vuyeb3fmjg 2470969 2470968 2026-10-19T21:24:00Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet 2470969 wikitext text/x-wiki l1a1sx18uu2hywtwyxu1i2zixi9jtqr 2470970 2470969 2026-10-19T21:24:02Z Stashbot 7414 dwisehaupt@dns1005: START - running authdns-update 2470970 wikitext text/x-wiki i3xa1yltrtcr9dbnrh1zzvq9c9izs1y 2470971 2470970 2026-10-19T21:26:23Z Stashbot 7414 dwisehaupt@dns1005: END - running authdns-update 2470971 wikitext text/x-wiki 0ebw65gh339y4nhe67nfjf7ou1e2s69 2470972 2470971 2026-10-19T21:27:43Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet 2470972 wikitext text/x-wiki g9tyvdg6n5bzouk7tuso8a91h7i0j7q 2470973 2470972 2026-10-19T21:30:46Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl2002.codfw.wmnet 2470973 wikitext text/x-wiki 4lc2yuy6w02q243mayptzhj4tx5ahva 2470974 2470973 2026-10-19T21:31:28Z Stashbot 7414 maryum: Deployed security fix for T441248 2470974 wikitext text/x-wiki cborfh1nj2uch36hdofqefy7ncscc0o 2470977 2470974 2026-10-19T21:32:44Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet 2470977 wikitext text/x-wiki p2m7rr689ckuekox37eivonok4t3ul6 2470978 2470977 2026-10-19T21:33:46Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet 2470978 wikitext text/x-wiki 5xi4l0mqdnwxsn5c4mlvfl3noqzru6y 2470979 2470978 2026-10-19T21:36:34Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet 2470979 wikitext text/x-wiki sdwa3w6u9vdjgmoanuiwqv38edeh6m4 2470980 2470979 2026-10-19T21:36:47Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet 2470980 wikitext text/x-wiki 81kbbrusq3zbivtayniuabaz41iwhy9 2470981 2470980 2026-10-19T21:39:39Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet 2470981 wikitext text/x-wiki 5dplud13mk2yvuq9zc2cioey4up18c6 2470982 2470981 2026-10-19T21:40:36Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet 2470982 wikitext text/x-wiki okx05oyymgc9bgpohkb79xzudqrzuat 2470983 2470982 2026-10-19T21:43:58Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet 2470983 wikitext text/x-wiki j9rncrtuikvrjo0gs3vcmj6zvbvvr10 2470984 2470983 2026-10-19T21:47:48Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet 2470984 wikitext text/x-wiki h1962odbogdap7jsq3uio73j7gvaxbi 2470985 2470984 2026-10-19T21:50:30Z Stashbot 7414 maryum: Deployed security fix for T440498 2470985 wikitext text/x-wiki 3dxa53oqhzpnjxjcodxvv8jlyaprrpe 2470986 2470985 2026-10-19T22:08:56Z Stashbot 7414 sbassett: Deployed security fix for T440946 2470986 wikitext text/x-wiki g28oflgokwg5e0lucj2brw6anajjuyb 2470987 2470986 2026-10-19T22:27:38Z Stashbot 7414 sbassett: Deployed security fix for T441632 2470987 wikitext text/x-wiki gp4y0ihzpkxzydvp3ak5ke7oexrra1a 2470989 2470987 2026-10-19T22:42:52Z Stashbot 7414 sbassett: Deployed security fix for T441639 2470989 wikitext text/x-wiki o68kw9169x1gd4zdz19s4gpqf8f5ic5 2470990 2470989 2026-10-19T23:29:15Z Stashbot 7414 sbassett: Deployed security fix for T441038 2470990 wikitext text/x-wiki bjozo6blxs6bgm760k0rbxhtishqxhh 2470991 2470990 2026-10-19T23:30:08Z Stashbot 7414 dwisehaupt@dns1004: START - running authdns-update 2470991 wikitext text/x-wiki hwu7q46z1bz0blxlgvoqi3htpqwo3sn 2470992 2470991 2026-10-19T23:32:32Z Stashbot 7414 dwisehaupt@dns1004: END - running authdns-update 2470992 wikitext text/x-wiki 8trd4ccbi98bfvuxvfuf0qz8jnn2w6v 2470994 2470992 2026-10-20T00:00:08Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync 2470994 wikitext text/x-wiki q281iu0z9c7ch2nwz1boplxsp2f9b9p 2470995 2470994 2026-10-20T00:00:17Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync 2470995 wikitext text/x-wiki qcdrjw8a2ppun4ro7bdf9ifjx3c6wcv 2470996 2470995 2026-10-20T00:00:20Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync 2470996 wikitext text/x-wiki dw3l3zd7gtn6l3syrbt9xx4yqwjr6hq 2470997 2470996 2026-10-20T00:00:36Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync 2470997 wikitext text/x-wiki kx5fpheksz6ztkz2ze68e6idq3gj2u6 2470999 2470997 2026-10-20T00:03:03Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1348770|Drop MediaViewerMobileBeta (T436799)]] 2470999 wikitext text/x-wiki su9jk5cs8mqz1w1v26uwuo27hkyhyjp 2471000 2470999 2026-10-20T00:07:22Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1348770|Drop MediaViewerMobileBeta (T436799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2471000 wikitext text/x-wiki h5lyx0crxjcb48l7oporfw5umjlg1hj 2471001 2471000 2026-10-20T00:08:31Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2471001 wikitext text/x-wiki qymyrvp9ksg8hifmurcow6s4h2b2176 2471002 2471001 2026-10-20T00:13:53Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1348770|Drop MediaViewerMobileBeta (T436799)]] (duration: 10m 50s) 2471002 wikitext text/x-wiki lxuyxxbtbwvtg0fh7s6oaj619fjtoxu 2471003 2471002 2026-10-20T00:14:37Z Stashbot 7414 dwisehaupt@dns1004: START - running authdns-update 2471003 wikitext text/x-wiki 8k3z9wy0s566ozci3jdudx9f5aalm5y 2471005 2471003 2026-10-20T00:17:07Z Stashbot 7414 dwisehaupt@dns1004: END - running authdns-update 2471005 wikitext text/x-wiki lzrrz6j3s8dsmki445m96cm5eztkkul 2471006 2471005 2026-10-20T02:00:31Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2471006 wikitext text/x-wiki bybk681bxdzpo9h6kvu9xnvfdzbblmy 2471007 2471006 2026-10-20T02:08:08Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) 2471007 wikitext text/x-wiki 5x0y2qlosyzdypa6ww9rebyxtsfdxsi 2471008 2471007 2026-10-20T04:01:05Z Stashbot 7414 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.22 (duration: 01m 01s) 2471008 wikitext text/x-wiki 6vvrpofy9omxwx697a7t4zinacxx82j 2471013 2471008 2026-10-20T05:30:29Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance 2471013 wikitext text/x-wiki k41b8oxj0dtpvtb6k0j9hap63ndumac 2471014 2471013 2026-10-20T05:33:26Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.decommission 2471014 wikitext text/x-wiki 4zroalxzrj7taeafe3e1myymd7obor1 2471015 2471014 2026-10-20T05:33:34Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts db1156.eqiad.wmnet 2471015 wikitext text/x-wiki kv9ac4upzsku4hlrud4pvmlxtckx5uc 2471016 2471015 2026-10-20T05:39:27Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.dns.netbox 2471016 wikitext text/x-wiki a5urnwjh6w7ch8j8i17oiafslxk24ct 2471017 2471016 2026-10-20T05:44:06Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1156.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" 2471017 wikitext text/x-wiki p5hb4eg7d2prdxp4slor59vyjktbbov 2471018 2471017 2026-10-20T05:44:51Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1156.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" 2471018 wikitext text/x-wiki 48by5kivur9xbod783pqjjwpwmtcni5 2471019 2471018 2026-10-20T05:44:53Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2471019 wikitext text/x-wiki pstx0wusrfd1w1z8eb4wlaqgsre2ipk 2471020 2471019 2026-10-20T05:44:54Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1156.eqiad.wmnet 2471020 wikitext text/x-wiki 6wv35kh8zfj2xl6f582qko0ul8efrz3 2471021 2471020 2026-10-20T05:45:01Z Stashbot 7414 marostegui@cumin1004: Removing db1156 from zarcillo T441298 2471021 wikitext text/x-wiki cyupz1c8u7tw0utj5dgeo8hc0bxwn7i 2471022 2471021 2026-10-20T05:45:03Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) 2471022 wikitext text/x-wiki jhce4khfv6ho8xo5pa0yqmcplm7qn9p 2471024 2471022 2026-10-20T06:03:43Z Stashbot 7414 arnaudb: moving gitlab behind CDN T425441 2471024 wikitext text/x-wiki 4ngnw6actozy65zw0b6zy5np35x8za1 2471025 2471024 2026-10-20T06:04:51Z Stashbot 7414 arnaudb@dns1006: START - running authdns-update 2471025 wikitext text/x-wiki h9le1y5wn9oezyn7sltl74b3r429nct 2471026 2471025 2026-10-20T06:06:14Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host releases2003.codfw.wmnet 2471026 wikitext text/x-wiki gzk0cd4pcui4n8i5tlq6kpjcxar7zg2 2471027 2471026 2026-10-20T06:07:12Z Stashbot 7414 arnaudb@dns1006: END - running authdns-update 2471027 wikitext text/x-wiki g1rmdsp2z0jvzjx9zef1ynchz45d98k 2471029 2471027 2026-10-20T06:10:15Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases2003.codfw.wmnet 2471029 wikitext text/x-wiki 5iqqnhln5zi5xiqxpg6cv8pancbhecd 2471030 2471029 2026-10-20T06:29:50Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host aphlict1002.eqiad.wmnet 2471030 wikitext text/x-wiki 065ao1gvvyqqim2az5hg2tfl0w5d9yl 2471031 2471030 2026-10-20T06:30:03Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host phab1005.eqiad.wmnet 2471031 wikitext text/x-wiki otezeywk0yawsvqrqrv1md0us8248ed 2471032 2471031 2026-10-20T06:30:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet 2471032 wikitext text/x-wiki e3s3ap6em8el6r7aji5ck3t8796fukv 2471033 2471032 2026-10-20T06:30:20Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org 2471033 wikitext text/x-wiki dtigv4egrikg20gguqqc14jemqerkvq 2471034 2471033 2026-10-20T06:31:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet 2471034 wikitext text/x-wiki 6xb73pd91rzlplj3bkwf6b021dj2qhd 2471035 2471034 2026-10-20T06:32:46Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471035 wikitext text/x-wiki klfqwse6ehynt4bhz5qkmd2q0p2n73z 2471036 2471035 2026-10-20T06:33:49Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict1002.eqiad.wmnet 2471036 wikitext text/x-wiki 6ef85tpmhn1iokyly8noucdt6052rpy 2471037 2471036 2026-10-20T06:37:32Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab1005.eqiad.wmnet 2471037 wikitext text/x-wiki 3i8ks4hlq8iy1dkq1ho6xyqc2bfc0ep 2471038 2471037 2026-10-20T06:37:35Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit1003 2471038 wikitext text/x-wiki ra2izot5v16rpftiwazzhuq0v4f3oma 2471039 2471038 2026-10-20T06:37:38Z Stashbot 7414 arnaudb@cumin1004: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit1003 2471039 wikitext text/x-wiki kzieq6y02s0tsc867ijeylr24z495jp 2471040 2471039 2026-10-20T06:40:21Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org 2471040 wikitext text/x-wiki h47uskkufm6amgtzc4kyvai5esuctb6 2471041 2471040 2026-10-20T06:40:38Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet 2471041 wikitext text/x-wiki con9e5fjgnptd8ts8sl6mtirkkznpm2 2471042 2471041 2026-10-20T06:41:04Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_magru and not P{cp7002*} and not P{cp7010*} and A:cp 2471042 wikitext text/x-wiki d9y62g45zz81p7ndes2ox70fnu5i8r6 2471043 2471042 2026-10-20T06:41:45Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet 2471043 wikitext text/x-wiki 95tnkkl81tfeeb2vzy7znnbp4a12cat 2471044 2471043 2026-10-20T06:43:04Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet 2471044 wikitext text/x-wiki hawhbmca2nhv6m5qj34uyr1bgr0hffn 2471045 2471044 2026-10-20T06:47:00Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471045 wikitext text/x-wiki 02umtyj3zbxy0uwm8xfj400k8smhgxi 2471046 2471045 2026-10-20T06:47:14Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet 2471046 wikitext text/x-wiki mtx7qqkw2lr6slffmacd32qthtbtoyn 2471047 2471046 2026-10-20T06:48:39Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet 2471047 wikitext text/x-wiki o1pydc6rg1ifbybvldutb9wsdc927cx 2471048 2471047 2026-10-20T06:49:05Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet 2471048 wikitext text/x-wiki t7tk649yzq6p0x8axorrlt433w7lwo7 2471049 2471048 2026-10-20T06:49:07Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet 2471049 wikitext text/x-wiki 6a0q0r4j7zo13s4ur98oempr8ob7d5i 2471050 2471049 2026-10-20T06:49:27Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet 2471050 wikitext text/x-wiki aly51ay6wz570qidjwxy5gpquagci5f 2471051 2471050 2026-10-20T06:51:35Z Stashbot 7414 arnaudb@cumin1004: START - Cookbook sre.hosts.reboot-single for host releases1003.eqiad.wmnet 2471051 wikitext text/x-wiki tixigdqkko913paeukrnvswsvdf8958 2471052 2471051 2026-10-20T06:52:54Z Stashbot 7414 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7001.magru.wmnet 2471052 wikitext text/x-wiki 2xm5uoip3sxo3h2ap46x4wgr94vjcrl 2471053 2471052 2026-10-20T06:57:33Z Stashbot 7414 arnaudb@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases1003.eqiad.wmnet 2471053 wikitext text/x-wiki 29p6i6vblo1tnfkkokoijlc9zu8828z 2471054 2471053 2026-10-20T06:59:34Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet 2471054 wikitext text/x-wiki hes8cx4ubart24woenrjx8tf2pxd07l 2471055 2471054 2026-10-20T06:59:58Z Stashbot 7414 moritzm: failover Ganeti master in drms01 to ganeti6001 2471055 wikitext text/x-wiki dws6na0yu6uuugiowk64g7efzt6ychc 2471056 2471055 2026-10-20T07:01:11Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org 2471056 wikitext text/x-wiki cywcs811z2af7uumcv5y809fzr1br5u 2471057 2471056 2026-10-20T07:07:52Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org 2471057 wikitext text/x-wiki 7sia8zbgiaozxwzslem4bu8hfxzouab 2471058 2471057 2026-10-20T07:17:05Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2471058 wikitext text/x-wiki jjgmhkmb9cbuoy9qlusgsswb4cpablp 2471059 2471058 2026-10-20T07:20:43Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471059 wikitext text/x-wiki lrcroyyyz7j8pe03oiuu3rs06tdioha 2471060 2471059 2026-10-20T07:32:54Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2471060 wikitext text/x-wiki psnqdx38uozrnngqjxrszwarwb138f4 2471061 2471060 2026-10-20T07:34:45Z Stashbot 7414 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7003.magru.wmnet 2471061 wikitext text/x-wiki gy7tgv622d6pvg0p7j1sfhayzw9j8r4 2471062 2471061 2026-10-20T07:36:59Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471062 wikitext text/x-wiki m2afhid54vx9jxcm3hil99t6fouits0 2471063 2471062 2026-10-20T07:40:23Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2471063 wikitext text/x-wiki l3p6p7ymhw29zfjj6cofx6stk7im3un 2471064 2471063 2026-10-20T07:44:46Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet 2471064 wikitext text/x-wiki mcbuzbrjxqc1ar5zd2p0s68qjhe1wil 2471065 2471064 2026-10-20T07:45:16Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org 2471065 wikitext text/x-wiki ny279l2xjsd73kgq3cogjhirxdppwro 2471066 2471065 2026-10-20T07:46:59Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org 2471066 wikitext text/x-wiki j44duilndq76topff8glflevmln2pkw 2471067 2471066 2026-10-20T07:51:13Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet 2471067 wikitext text/x-wiki ia9q4dh9xfmsyu0cu34vy2v30l98xkc 2471068 2471067 2026-10-20T07:51:21Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet 2471068 wikitext text/x-wiki did8zudc7tlkt3q08sfpecacef16dgc 2471069 2471068 2026-10-20T07:52:02Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5005.eqsin.wmnet 2471069 wikitext text/x-wiki cmezbc1htwd1k75b31cneo2uu8po5q8 2471070 2471069 2026-10-20T07:56:09Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471070 wikitext text/x-wiki d8wnvuno9407rdpliakeoslnyf1w920 2471071 2471070 2026-10-20T07:56:51Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-master1004.eqiad.wmnet 2471071 wikitext text/x-wiki 02zovgbfd2ll7k3nz8o3uefm1k6iwac 2471072 2471071 2026-10-20T07:59:58Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet 2471072 wikitext text/x-wiki cc0wkanmf7iu3n0wyaqingkbdmyqecs 2471073 2471072 2026-10-20T08:00:28Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1004.eqiad.wmnet 2471073 wikitext text/x-wiki sqtwsdm75xcfqesbkghulznscmzajtz 2471074 2471073 2026-10-20T08:00:42Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org 2471074 wikitext text/x-wiki f8j1hukfn037d4hegf4npm9yfqr6zta 2471075 2471074 2026-10-20T08:02:13Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2471075 wikitext text/x-wiki ibd6mk7u3ss8wdv27mc5axgziecx7xp 2471076 2471075 2026-10-20T08:02:20Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471076 wikitext text/x-wiki 49euld6ly8iszqmmlvy8b59nnn5ypt1 2471077 2471076 2026-10-20T08:03:46Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2471077 wikitext text/x-wiki f28dxchpndls8l56gnqrcod8u53is6g 2471078 2471077 2026-10-20T08:04:10Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet 2471078 wikitext text/x-wiki 1lberterzrb777vp465yfw7ljgs4v5p 2471079 2471078 2026-10-20T08:04:21Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet 2471079 wikitext text/x-wiki 3884gtqz9l2ooqazwrxrdawssvbesxv 2471080 2471079 2026-10-20T08:06:09Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2471080 wikitext text/x-wiki 17itmxxsmbe23e6q4xxbr4mi0856263 2471081 2471080 2026-10-20T08:06:30Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet 2471081 wikitext text/x-wiki 4dm7gusggwf57nh666nla2yjf3zfgw4 2471082 2471081 2026-10-20T08:09:44Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org 2471082 wikitext text/x-wiki 8yj4zlbxju5jx6q6txujg4efkd6m7j0 2471083 2471082 2026-10-20T08:10:45Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet 2471083 wikitext text/x-wiki 1dv1pfynajpsmcudb08nd4uzyc9gixi 2471084 2471083 2026-10-20T08:10:53Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5005.eqsin.wmnet 2471084 wikitext text/x-wiki 1i0wpr41ol6jirdzh22wp8w0zd88b5h 2471085 2471084 2026-10-20T08:11:35Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2471085 wikitext text/x-wiki 0bndpn6w7s5m67gb6exlz89cbrsu4s2 2471086 2471085 2026-10-20T08:11:38Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet 2471086 wikitext text/x-wiki fdnli5s592gbl68gvta0ldlubqcnsu6 2471087 2471086 2026-10-20T08:11:40Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet 2471087 wikitext text/x-wiki g4g5mmna2n15w1hcoeamip621nb48z3 2471088 2471087 2026-10-20T08:16:23Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet 2471088 wikitext text/x-wiki j8bwnbb20pmeqja5jo4yqo6rwgwic6l 2471089 2471088 2026-10-20T08:16:53Z Stashbot 7414 moritzm: failover Ganeti master in eqsin to ganeti5007 2471089 wikitext text/x-wiki tc6kl66acusyk8xlyuuh9dvuiy57v75 2471090 2471089 2026-10-20T08:16:58Z Stashbot 7414 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7004.magru.wmnet 2471090 wikitext text/x-wiki 9fdh1wpvwe7ury0f78jlnvds1brt9g1 2471091 2471090 2026-10-20T08:17:57Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2471091 wikitext text/x-wiki qf2pyu1ei0cbs9i5q7w5xnghd7pvwp8 2471092 2471091 2026-10-20T08:18:00Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . 2471092 wikitext text/x-wiki ioztm0quxrsjcw76en103532essqf2r 2471093 2471092 2026-10-20T08:19:19Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org 2471093 wikitext text/x-wiki 58jdjjrui8h5p3r10sm0puuuze6b3b9 2471094 2471093 2026-10-20T08:20:06Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3071.*} 2471094 wikitext text/x-wiki bb6fzwhwnc6ey36supdcx16mmkvdsf5 2471095 2471094 2026-10-20T08:22:31Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3071.*} 2471095 wikitext text/x-wiki bo09x8k4q6el9e3cstimo9cx33kncsd 2471096 2471095 2026-10-20T08:22:37Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet 2471096 wikitext text/x-wiki gd2hngmq6jmik143jhaqyyrt0j4wpx8 2471097 2471096 2026-10-20T08:22:42Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet 2471097 wikitext text/x-wiki au1nofxw8lnjnghibz1oil0c5ygq3lf 2471098 2471097 2026-10-20T08:23:49Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-coord1004.eqiad.wmnet 2471098 wikitext text/x-wiki a2gdnm1m03cvszfdn60q2g3jmyh8z8t 2471099 2471098 2026-10-20T08:24:53Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet 2471099 wikitext text/x-wiki 6t9a2xye0r6xo97raa3epbzj2nvf5u1 2471100 2471099 2026-10-20T08:25:54Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org 2471100 wikitext text/x-wiki jospv59yzlulf9s9hr4mae1mhl8wrrw 2471101 2471100 2026-10-20T08:28:59Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw 2471101 wikitext text/x-wiki 1q8r2qp4oeemp19ys5ovhhg52yjahk7 2471102 2471101 2026-10-20T08:29:08Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org 2471102 wikitext text/x-wiki m7xjzc6qg867njm61w05uqirok5ay4m 2471103 2471102 2026-10-20T08:29:45Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-coord1004.eqiad.wmnet 2471103 wikitext text/x-wiki kbsqaoiq8auevepzozy78kcw33glzxv 2471104 2471103 2026-10-20T08:31:48Z Stashbot 7414 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet 2471104 wikitext text/x-wiki kvigezgaiffkwlzvh5kvxea7w3o5ut8 2471105 2471104 2026-10-20T08:33:24Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet 2471105 wikitext text/x-wiki 7tsvhglmo6welpalx5gl3csny8fxf8j 2471106 2471105 2026-10-20T08:33:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet 2471106 wikitext text/x-wiki 71ufq3d13tup8pd36o0ldw4meahv5ut 2471107 2471106 2026-10-20T08:35:42Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org 2471107 wikitext text/x-wiki 93q61ldl2s1ljgjos58v6ii6q9d7b1b 2471108 2471107 2026-10-20T08:37:04Z Stashbot 7414 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet 2471108 wikitext text/x-wiki tkoml0sp5jqlm8iezpoz6onj0idwlmp 2471109 2471108 2026-10-20T08:37:23Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install2005.wikimedia.org 2471109 wikitext text/x-wiki hcflws41q8uxru11d0oul20ddwzvhvw 2471111 2471109 2026-10-20T08:39:33Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet 2471111 wikitext text/x-wiki i0swjoz6qcf882bud6zi0o4s9i14az9 2471112 2471111 2026-10-20T08:39:55Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet 2471112 wikitext text/x-wiki 4dew238egisck1ddz1mpszk828we6j4 2471113 2471112 2026-10-20T08:41:31Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs4010.*,lvs5006.*,lvs6003.*} and A:liberica 2471113 wikitext text/x-wiki fvy328szudoup12uzwijnyasqfmegvk 2471114 2471113 2026-10-20T08:43:56Z Stashbot 7414 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet 2471114 wikitext text/x-wiki dl891ng1p4lj2fy3w10284zor0s7kg7 2471115 2471114 2026-10-20T08:44:03Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet 2471115 wikitext text/x-wiki cdljrdsc4p3nll29kbtxbuhhasjibfe 2471116 2471115 2026-10-20T08:44:09Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org 2471116 wikitext text/x-wiki dfe3hldy6gh4eubo155lp7htwzsc1dk 2471117 2471116 2026-10-20T08:44:25Z Stashbot 7414 moritzm: failover Ganeti master in drms02 to ganeti6002 2471117 wikitext text/x-wiki gu70f9x73za5rgwlcjcyxe7n49oz6n6 2471118 2471117 2026-10-20T08:45:19Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org 2471118 wikitext text/x-wiki 2fbvpaflalls3qm2cmsjd98jzgfeaus 2471119 2471118 2026-10-20T08:49:45Z Stashbot 7414 trueg@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: sync 2471119 wikitext text/x-wiki p5vwhowqk5icj45mag24ri02f4oibrj 2471120 2471119 2026-10-20T08:49:48Z Stashbot 7414 trueg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync 2471120 wikitext text/x-wiki tgq3c4bgpd2t6dqev43ypp2q8ptojah 2471121 2471120 2026-10-20T08:50:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet 2471121 wikitext text/x-wiki c2n5fke3kp8t4i96hsy5qs2vy8ig35b 2471122 2471121 2026-10-20T08:50:48Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet 2471122 wikitext text/x-wiki 0z13l788dtvlhj3kg45oujnyccflv4j 2471123 2471122 2026-10-20T08:50:55Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet 2471123 wikitext text/x-wiki 30fuytnnlm4k4hzi1h8n0pa7ury0d6z 2471124 2471123 2026-10-20T08:51:54Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org 2471124 wikitext text/x-wiki f6ty5agi7v518auiczwow0jtzm92f64 2471125 2471124 2026-10-20T08:52:40Z Stashbot 7414 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet 2471125 wikitext text/x-wiki gzxz24r18cfu1mwofgxrz9f6poqoapg 2471126 2471125 2026-10-20T08:53:05Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs4010.*,lvs5006.*,lvs6003.*} and A:liberica 2471126 wikitext text/x-wiki 4gnetwkuxlwq3959vj9l1loedion9h3 2471127 2471126 2026-10-20T08:54:18Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs3010.*} and A:liberica 2471127 wikitext text/x-wiki f4nu3nceekxyg6tudzynyxrjf81068e 2471128 2471127 2026-10-20T08:55:26Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet 2471128 wikitext text/x-wiki 3jdpzx2dbqcwb8u9w6n342h4ib534kq 2471129 2471128 2026-10-20T08:57:53Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet 2471129 wikitext text/x-wiki hhjmw6l7k2wjl137h7huqfeuxqqknax 2471130 2471129 2026-10-20T08:57:58Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet 2471130 wikitext text/x-wiki 0jh1yymgm9gr3g3hwq33eijcz19zl17 2471131 2471130 2026-10-20T08:58:01Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs3010.*} and A:liberica 2471131 wikitext text/x-wiki mi5zt3x9cqiwbzci9uj6wvuuh939hbz 2471132 2471131 2026-10-20T08:58:54Z Stashbot 7414 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7005.magru.wmnet 2471132 wikitext text/x-wiki f61kbjadpamh99a6blagoqewfc7m9ub 2471133 2471132 2026-10-20T09:00:00Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast4006.wikimedia.org 2471133 wikitext text/x-wiki 6f5psl6kr3s8f7h1tmdo2mpbdumwyly 2471134 2471133 2026-10-20T09:03:48Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet 2471134 wikitext text/x-wiki rwjk9pxdqk5dk3yfmzv9g6uskjz8hkw 2471135 2471134 2026-10-20T09:04:12Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw 2471135 wikitext text/x-wiki bd5m4pvsa1jd8dyoaxi9r2ryr7k1tpd 2471136 2471135 2026-10-20T09:04:16Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2002.codfw.wmnet 2471136 wikitext text/x-wiki 6fxejheunyy7ozz1qee812rym9roknv 2471137 2471136 2026-10-20T09:04:42Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet 2471137 wikitext text/x-wiki fom1tsvpuwx8jrns2drvvko4sgxhney 2471138 2471137 2026-10-20T09:04:51Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2002.codfw.wmnet 2471138 wikitext text/x-wiki hoy9200ztglqdqx6e4skoagnkroxpav 2471139 2471138 2026-10-20T09:08:03Z Stashbot 7414 herron@cumin1004: END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling reboot on A:logstash-codfw 2471139 wikitext text/x-wiki 28j27t1npu4y8icl1162udfvw7w7uol 2471140 2471139 2026-10-20T09:08:56Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2002.codfw.wmnet 2471140 wikitext text/x-wiki 6i8e4md4gti684yjx91v8exayer0zjo 2471141 2471140 2026-10-20T09:08:57Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2002.codfw.wmnet 2471141 wikitext text/x-wiki lwukdxsj0mkwq7vs46vxt7c034y8mtb 2471142 2471141 2026-10-20T09:09:03Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2003.codfw.wmnet 2471142 wikitext text/x-wiki fo68l10jytbs9m70fzrtxa6mvvbv92b 2471143 2471142 2026-10-20T09:09:35Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2003.codfw.wmnet 2471143 wikitext text/x-wiki f8hnsadpqme3welabayjcxsdkfcspeq 2471144 2471143 2026-10-20T09:10:13Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw 2471144 wikitext text/x-wiki oxu8doouhyns8eih9tnrqncrztswudo 2471145 2471144 2026-10-20T09:10:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet 2471145 wikitext text/x-wiki kd0zedu2s8hv8wr2qgmeode7wdzj3sf 2471146 2471145 2026-10-20T09:10:45Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet 2471146 wikitext text/x-wiki nsfvm69shxcdnktxjhj15n1j84y70a8 2471147 2471146 2026-10-20T09:13:39Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2003.codfw.wmnet 2471147 wikitext text/x-wiki ozwk3z7kxbk5vfmbrpt10r49spj9ny1 2471148 2471147 2026-10-20T09:13:42Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2003.codfw.wmnet 2471148 wikitext text/x-wiki 9txn84z927jt9hui9rv6dbwozzz33us 2471149 2471148 2026-10-20T09:13:47Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2004.codfw.wmnet 2471149 wikitext text/x-wiki 37j2nzeoqat2suezom9eef8n4e0phd3 2471150 2471149 2026-10-20T09:14:22Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2004.codfw.wmnet 2471150 wikitext text/x-wiki sprnfkbv9txnw39witi5a8lu33tr1vl 2471151 2471150 2026-10-20T09:15:15Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad 2471151 wikitext text/x-wiki t92r9fsd7eiqz5q7b8da823g62hdxkg 2471152 2471151 2026-10-20T09:16:32Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-web1001.eqiad.wmnet 2471152 wikitext text/x-wiki iho1vn4we19v11vle96kvzors5db2sh 2471153 2471152 2026-10-20T09:18:26Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2004.codfw.wmnet 2471153 wikitext text/x-wiki enzspvxalgbpjhbc1n9bgzdg0ft66eo 2471154 2471153 2026-10-20T09:18:29Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2004.codfw.wmnet 2471154 wikitext text/x-wiki q5w6u19e0c7ll6oluqsxnwbksf9j1nf 2471155 2471154 2026-10-20T09:18:34Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2005.codfw.wmnet 2471155 wikitext text/x-wiki mfyevggfqaaf5whs7fqri1ahd0jwzy8 2471156 2471155 2026-10-20T09:18:42Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2471156 wikitext text/x-wiki g21dgc3l2kxx9x69jczjs3gkdtmkmlg 2471157 2471156 2026-10-20T09:19:05Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2005.codfw.wmnet 2471157 wikitext text/x-wiki 176x76hupot7qhhlfhaxfxuoj6y2lyn 2471158 2471157 2026-10-20T09:19:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet 2471158 wikitext text/x-wiki d2e9fbhclwok9y2vybi2guaudinlnjg 2471159 2471158 2026-10-20T09:21:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast4006.wikimedia.org 2471159 wikitext text/x-wiki p6dsu3noobape8vof7dct3e89soi201 2471161 2471159 2026-10-20T09:22:42Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-web1001.eqiad.wmnet 2471161 wikitext text/x-wiki tnfr00blz90t9zch61m8jt6qw4atd6e 2471162 2471161 2026-10-20T09:23:10Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2005.codfw.wmnet 2471162 wikitext text/x-wiki qnmlvmbws42hp9hsmmcspm5zu7qlqtb 2471163 2471162 2026-10-20T09:23:12Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2005.codfw.wmnet 2471163 wikitext text/x-wiki 2fuxq596l1m3xomp6mrz98dm1jufvv1 2471164 2471163 2026-10-20T09:23:18Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet 2471164 wikitext text/x-wiki nzfx2cvpfc4y0bq0ftnr7gr75efaupv 2471165 2471164 2026-10-20T09:23:53Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet 2471165 wikitext text/x-wiki ms51i3xzw1o8svw1qsgd9ttrilkn5vi 2471166 2471165 2026-10-20T09:24:37Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet 2471166 wikitext text/x-wiki 8jztj80h4ziarhb014r7vrszf56pztu 2471167 2471166 2026-10-20T09:26:53Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org 2471167 wikitext text/x-wiki c4y8fmrtya5o0btj4qe05xk481bb7bk 2471168 2471167 2026-10-20T09:29:14Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet 2471168 wikitext text/x-wiki sfcg5erj4ursk2tlyk4ffe605yfwtm4 2471169 2471168 2026-10-20T09:29:16Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet 2471169 wikitext text/x-wiki 1z2d0dkz96te7w09kjpfox8m2z89vjk 2471170 2471169 2026-10-20T09:29:22Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet 2471170 wikitext text/x-wiki 68wkpzdlrvjiffvdemrfx7rsgiwskvr 2471171 2471170 2026-10-20T09:29:56Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet 2471171 wikitext text/x-wiki m5hojznls759drl6mjbopybsqvkz402 2471172 2471171 2026-10-20T09:30:14Z Stashbot 7414 hashar: restarting MediaWiki branch cutting job # T438222 T441774 2471172 wikitext text/x-wiki kajnsgtujcfqeoen014b1l1tx8yd79g 2471173 2471172 2026-10-20T09:30:18Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=1) rolling reboot on A:cephosd-eqiad 2471173 wikitext text/x-wiki id9vdaf1kiwdwrlevk0m4fxydtm2zlp 2471174 2471173 2026-10-20T09:30:27Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet 2471174 wikitext text/x-wiki hbc7bh5h67446khcnr8e2f2au9tcc37 2471175 2471174 2026-10-20T09:30:34Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet 2471175 wikitext text/x-wiki 3gylno0xorc564giuuk6jrz99x2yrjr 2471176 2471175 2026-10-20T09:30:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org 2471176 wikitext text/x-wiki esnisld2vtfk0er1gge33rieno98qku 2471177 2471176 2026-10-20T09:32:18Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance 2471177 wikitext text/x-wiki cp4xluln0sh67oyd3tsy63oco1q0ln8 2471178 2471177 2026-10-20T09:32:20Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96859 and previous config saved to /var/cache/conftool/dbconfig/20261020-093218-ladsgroup.json 2471178 wikitext text/x-wiki gdx66bt8ywr0sqeb776b2gym2v57aub 2471180 2471178 2026-10-20T09:35:17Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet 2471180 wikitext text/x-wiki lr3gzcecjpqbgl8fosh7nrhop2au9hs 2471181 2471180 2026-10-20T09:35:19Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet 2471181 wikitext text/x-wiki t1fw246l8hpk5nt0e980msacb1f368b 2471182 2471181 2026-10-20T09:35:24Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet 2471182 wikitext text/x-wiki d5g1urkuuki2qhlwbk0vewjh7el4ccp 2471184 2471182 2026-10-20T09:35:59Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet 2471184 wikitext text/x-wiki acnxg1vlalvmv73bxsp93oyr1ksem1t 2471185 2471184 2026-10-20T09:36:19Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2028.codfw.wmnet 2471185 wikitext text/x-wiki 2v3yezckzf9tup8gkjin5ap8fd55wl1 2471186 2471185 2026-10-20T09:38:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet 2471186 wikitext text/x-wiki q6wuu2e3cswr2ayqai26xg3j6qaf33g 2471187 2471186 2026-10-20T09:39:00Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96860 and previous config saved to /var/cache/conftool/dbconfig/20261020-093859-ladsgroup.json 2471187 wikitext text/x-wiki 03wa28i1yzlyrl6i2jvesrnmbi1jnyv 2471188 2471187 2026-10-20T09:40:42Z Stashbot 7414 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7006.magru.wmnet 2471188 wikitext text/x-wiki d3a7lvl45fr5ti9e0na5yfhv5p17zlv 2471189 2471188 2026-10-20T09:41:13Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor-dev2001.codfw.wmnet 2471189 wikitext text/x-wiki k4blzujv09lwvohb3jyiu0vsg06vrtu 2471190 2471189 2026-10-20T09:41:22Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet 2471190 wikitext text/x-wiki 49z2xsm2hgvau9nosmrvra18ffey58o 2471191 2471190 2026-10-20T09:41:24Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet 2471191 wikitext text/x-wiki sk9nb7g0esufx63vob4l5k3hyrjqak9 2471192 2471191 2026-10-20T09:41:29Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet 2471192 wikitext text/x-wiki g44jizxupigdwdxo4fqntnct9ygz5ga 2471193 2471192 2026-10-20T09:43:43Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet 2471193 wikitext text/x-wiki j62j59nzckt68jm7ypulyseicthoo29 2471194 2471193 2026-10-20T09:44:18Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet 2471194 wikitext text/x-wiki 241yzault66du2vtjfb82dfel1ljtth 2471195 2471194 2026-10-20T09:44:41Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet 2471195 wikitext text/x-wiki n9gotcr0vj1rrjk6xh7t89seprwmikz 2471196 2471195 2026-10-20T09:45:14Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor-dev2001.codfw.wmnet 2471196 wikitext text/x-wiki 1066xfu3mo1fpksjvd6r85zxej1tg34 2471197 2471196 2026-10-20T09:46:12Z Stashbot 7414 jelto@deploy1003: Started scap deploy-service miscweb: demo deployment for MVP walk through 2471197 wikitext text/x-wiki 0vjcs3mhgqtpbsuohio5souxeam2xuf 2471198 2471197 2026-10-20T09:46:14Z Stashbot 7414 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb 2471198 wikitext text/x-wiki 9klk9ovjdfdo845mxbb2g5y83360x8o 2471199 2471198 2026-10-20T09:46:31Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet 2471199 wikitext text/x-wiki 1tv81wsphdg3qdbbos8v33zzndvwwyx 2471200 2471199 2026-10-20T09:47:06Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet 2471200 wikitext text/x-wiki 8eblkn7ovwyw4803unmda862n57iw0m 2471201 2471200 2026-10-20T09:47:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet 2471201 wikitext text/x-wiki ky1ymx7hzwxbl2ohsm01nvkf7fz0o0p 2471202 2471201 2026-10-20T09:47:53Z Stashbot 7414 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb 2471202 wikitext text/x-wiki qitd6ta1eck62sddxd3mng253iozifi 2471203 2471202 2026-10-20T09:49:02Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P96862 and previous config saved to /var/cache/conftool/dbconfig/20261020-094901-ladsgroup.json 2471203 wikitext text/x-wiki dxhhwipur1lpq5ibkozs7mqza5pawrh 2471204 2471203 2026-10-20T09:49:40Z Stashbot 7414 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb 2471204 wikitext text/x-wiki sn6fnj449ens67g081hgtrjdxp9298c 2471205 2471204 2026-10-20T09:49:43Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1010.eqiad.wmnet 2471205 wikitext text/x-wiki kda3ftdoihqsmfmkuh6zv193yfhx5sw 2471206 2471205 2026-10-20T09:49:56Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7002.*} and A:liberica 2471206 wikitext text/x-wiki s7qr6nugen6h8qbd3xwktktozkqaj5s 2471207 2471206 2026-10-20T09:50:12Z Stashbot 7414 jelto@deploy1003: Finished scap deploy-service miscweb: demo deployment for MVP walk through (duration: 03m 59s) 2471207 wikitext text/x-wiki 2dtoehar781wtkv9qjhqv2dimq92t5n 2471208 2471207 2026-10-20T09:50:16Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet 2471208 wikitext text/x-wiki e8vdp2xl2w9q3x3en3pjtpyzoson5fm 2471209 2471208 2026-10-20T09:50:29Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2010.codfw.wmnet 2471209 wikitext text/x-wiki q1flut075ffw9w9xingy8rrpgi6514f 2471210 2471209 2026-10-20T09:51:05Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet 2471210 wikitext text/x-wiki nwn8c34jvpg71l1e6zdhsnqfgwahyns 2471211 2471210 2026-10-20T09:51:30Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'configure' for AS: 3292 2471211 wikitext text/x-wiki j35akww3s7pcuyqxjsihts6t8lnklds 2471212 2471211 2026-10-20T09:51:59Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet 2471212 wikitext text/x-wiki 3oih90uqttr2hibqr62ob8ofhhjbgrs 2471213 2471212 2026-10-20T09:52:02Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet 2471213 wikitext text/x-wiki c8szgvkjpxh3notia5qhxtvmclm77dy 2471214 2471213 2026-10-20T09:52:04Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw 2471214 wikitext text/x-wiki rxrtlz9y3r08yc9rowxiazk0v2ksk9a 2471215 2471214 2026-10-20T09:52:24Z Stashbot 7414 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 3292 2471215 wikitext text/x-wiki 8i0xu428vqpf636jjg9t61lcm6pm830 2471216 2471215 2026-10-20T09:53:53Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7002.*} and A:liberica 2471216 wikitext text/x-wiki lex3nx5rg2lx7d61qebxuoqq5pss870 2471217 2471216 2026-10-20T09:54:46Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet 2471217 wikitext text/x-wiki imt9v9n81bw1ijpxaz86xs0adbwvo7q 2471218 2471217 2026-10-20T09:55:11Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet 2471218 wikitext text/x-wiki 0kvapdxat0a0jbqnst2w6qv7b58n4n4 2471219 2471218 2026-10-20T09:55:19Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1010.eqiad.wmnet 2471219 wikitext text/x-wiki qihv3thm00157izzm4ep1vsaq267z42 2471220 2471219 2026-10-20T09:55:22Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1011.eqiad.wmnet 2471220 wikitext text/x-wiki m8rvdsl27jd9rlqvgpxp0z90jztfx9s 2471221 2471220 2026-10-20T09:55:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet 2471221 wikitext text/x-wiki ce0wedmqf1mfomqlqorqhewf92j5jd7 2471222 2471221 2026-10-20T09:55:32Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet 2471222 wikitext text/x-wiki 0xfut3648mcbm47lqiap3egq8jeo9zh 2471223 2471222 2026-10-20T09:55:39Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet 2471223 wikitext text/x-wiki dhw4vg49eh215sofk8siwebf7q8s3bt 2471224 2471223 2026-10-20T09:57:11Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad 2471224 wikitext text/x-wiki ay0btm5ma6jq9dj5fzok3xkitgjdbru 2471225 2471224 2026-10-20T09:57:14Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1002.eqiad.wmnet 2471225 wikitext text/x-wiki m7giptra0y0jpnikwp6h3pqf305ozx5 2471226 2471225 2026-10-20T09:57:51Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1002.eqiad.wmnet 2471226 wikitext text/x-wiki g6tza0uz8z60098rca6onq0mkv2s2sr 2471227 2471226 2026-10-20T09:58:06Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet 2471227 wikitext text/x-wiki ouc431lnxqx5ya0wumys68gk99p4gr6 2471228 2471227 2026-10-20T09:58:48Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2010.codfw.wmnet 2471228 wikitext text/x-wiki nnta9crjp8onbmv8qruffjbnoo828ge 2471229 2471228 2026-10-20T09:58:52Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2011.codfw.wmnet 2471229 wikitext text/x-wiki 9j0mboeoguc4cmo3js4y184y6k3f5rh 2471230 2471229 2026-10-20T09:59:04Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P96863 and previous config saved to /var/cache/conftool/dbconfig/20261020-095902-ladsgroup.json 2471230 wikitext text/x-wiki snm97hoxlnwg1tq37dxffi5w6v6kplm 2471231 2471230 2026-10-20T09:59:09Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet 2471231 wikitext text/x-wiki a7hvq1jazhx0fhqp9gap5rxx4f07et4 2471232 2471231 2026-10-20T10:01:59Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1002.eqiad.wmnet 2471232 wikitext text/x-wiki gw13hxj40j2l0ex05gpjxcmgwti07m5 2471233 2471232 2026-10-20T10:02:02Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1002.eqiad.wmnet 2471233 wikitext text/x-wiki 1p4xyi1ta3rqqr0dgkbzpd4ahvrz7ao 2471234 2471233 2026-10-20T10:02:09Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1003.eqiad.wmnet 2471234 wikitext text/x-wiki nt4l2sxld3c56gh7zugyj8421dxby1f 2471235 2471234 2026-10-20T10:02:30Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1011.eqiad.wmnet 2471235 wikitext text/x-wiki oge4qagmk03sezupl7fqmiq6iictkyr 2471236 2471235 2026-10-20T10:02:41Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1003.eqiad.wmnet 2471236 wikitext text/x-wiki pzq40t2nwd0je5rc8lu70xzq6f44wam 2471237 2471236 2026-10-20T10:02:50Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet 2471237 wikitext text/x-wiki pkaf422a8tj35nvq7slxo2ae7t5q7x6 2471238 2471237 2026-10-20T10:03:16Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet 2471238 wikitext text/x-wiki 0osizl1t83l9pdr721xl91yayuda1p7 2471239 2471238 2026-10-20T10:03:26Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet 2471239 wikitext text/x-wiki 1gkdd2sibaiil3n3wt7e53gk6jv0ta7 2471240 2471239 2026-10-20T10:04:45Z Stashbot 7414 moritzm: powercycle ganeti2029 2471240 wikitext text/x-wiki byq9qtqsoj4lm1sytwtjuw79pgi3ruw 2471241 2471240 2026-10-20T10:05:50Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2011.codfw.wmnet 2471241 wikitext text/x-wiki cunzixw6gt16f41rkmu5sz54n2gmhz6 2471242 2471241 2026-10-20T10:06:02Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet 2471242 wikitext text/x-wiki njtad9d0s5zf6tb7dpddlkxs1xqgaan 2471243 2471242 2026-10-20T10:06:49Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1003.eqiad.wmnet 2471243 wikitext text/x-wiki nvohp9d4dm9lsfzm2kub8z7tlqo3x21 2471244 2471243 2026-10-20T10:06:52Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1003.eqiad.wmnet 2471244 wikitext text/x-wiki 5yyhxhu7y24m9yrevcq4dwx3effex4w 2471245 2471244 2026-10-20T10:06:57Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1004.eqiad.wmnet 2471245 wikitext text/x-wiki 8qw2opdpnd555y2lr6w1mxp24dh3uz6 2471246 2471245 2026-10-20T10:07:22Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet 2471246 wikitext text/x-wiki m7r12jnm4gboit98jmro9ki4f0xjh7p 2471247 2471246 2026-10-20T10:07:30Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1004.eqiad.wmnet 2471247 wikitext text/x-wiki gzfcz9gznfhqxse44xu4qhjiszhhu4d 2471248 2471247 2026-10-20T10:09:04Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backupmon1001.eqiad.wmnet 2471248 wikitext text/x-wiki 04sn609tad3qau2iev0dsftzlqoz3nq 2471249 2471248 2026-10-20T10:09:59Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96864 and previous config saved to /var/cache/conftool/dbconfig/20261020-100904-ladsgroup.json 2471249 wikitext text/x-wiki 9q7hgel0d8j35mg9cyoiusjylgzebwv 2471251 2471249 2026-10-20T10:10:30Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance 2471251 wikitext text/x-wiki rd9i9y6fbkfno311zucqv1k8vtaekll 2471252 2471251 2026-10-20T10:11:19Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2180 (T441633)', diff saved to https://phabricator.wikimedia.org/P96865 and previous config saved to /var/cache/conftool/dbconfig/20261020-100926-ladsgroup.json 2471252 wikitext text/x-wiki 7ct1ecivfl3ngh4kgtajtc9m3gybvho 2471253 2471252 2026-10-20T10:11:53Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet 2471253 wikitext text/x-wiki mdmgjiq9re3sy5czb1pzd9pvmz2oe1y 2471254 2471253 2026-10-20T10:12:35Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet 2471254 wikitext text/x-wiki 517srlhew8a7219wqxqep9lh50vhkjk 2471255 2471254 2026-10-20T10:13:09Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1004.eqiad.wmnet 2471255 wikitext text/x-wiki b0tgxejy9pfczufxi0ilkib5yoiuftl 2471256 2471255 2026-10-20T10:13:52Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1004.eqiad.wmnet 2471256 wikitext text/x-wiki jjddnnkxklxskjkb4gq09qhk0389vll 2471258 2471256 2026-10-20T10:14:23Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1005.eqiad.wmnet 2471258 wikitext text/x-wiki bwxs3pjkci1vcubj9gw7ggafrntoaia 2471259 2471258 2026-10-20T10:14:44Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet 2471259 wikitext text/x-wiki gz3dhtvexdi5rf4ylj3ortwrrdxfx7p 2471261 2471259 2026-10-20T10:15:30Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet 2471261 wikitext text/x-wiki 1mg42g4l9ovoou6fikp069fna0207mh 2471262 2471261 2026-10-20T10:16:17Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backupmon1001.eqiad.wmnet 2471262 wikitext text/x-wiki ilcx3e1g36y2qcdma3httfmt30hr4jk 2471263 2471262 2026-10-20T10:17:03Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet 2471263 wikitext text/x-wiki jdzhvytanioihw00b5wib0jd25umaqb 2471264 2471263 2026-10-20T10:20:37Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1005.eqiad.wmnet 2471264 wikitext text/x-wiki mp8xh8u63eqi32ug1r291hu85ifeb20 2471265 2471264 2026-10-20T10:21:02Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1005.eqiad.wmnet 2471265 wikitext text/x-wiki lgwwqcz0728c7s6e0sjclcmjroyc2lz 2471266 2471265 2026-10-20T10:21:24Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet 2471266 wikitext text/x-wiki mn64ub12moetg9kjvzj8w11ao2oep3c 2471268 2471266 2026-10-20T10:21:42Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet 2471268 wikitext text/x-wiki qqiom7y5o9tnh5y7pgel2x25yf316n4 2471269 2471268 2026-10-20T10:22:17Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2471269 wikitext text/x-wiki rb53kkfbo2vrdyal58oskjfqn4889pg 2471270 2471269 2026-10-20T10:22:38Z Stashbot 7414 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7007.magru.wmnet 2471270 wikitext text/x-wiki 93gxpgp384qdb3g2z1yhfnt7wdrdwxx 2471271 2471270 2026-10-20T10:23:07Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7001.*} and A:liberica 2471271 wikitext text/x-wiki 65yifeakdrw4hehpxetvh0x2tsolcdx 2471272 2471271 2026-10-20T10:23:30Z Stashbot 7414 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) rebooting P{lvs7001.*} and A:liberica 2471272 wikitext text/x-wiki aj3tyd7b7systmh8rrz2g3q76g98tho 2471273 2471272 2026-10-20T10:24:20Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs7001.magru.wmnet 2471273 wikitext text/x-wiki nsljumjvieodo2sah7sokyenzmm3urn 2471274 2471273 2026-10-20T10:24:40Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2471274 wikitext text/x-wiki e2693q6bnl4h416buj9sbd3d55vy1l0 2471275 2471274 2026-10-20T10:26:17Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad 2471275 wikitext text/x-wiki 2uakwpjkxj8ffwhyolu1bjnrjrxkwq1 2471276 2471275 2026-10-20T10:26:34Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P96867 and previous config saved to /var/cache/conftool/dbconfig/20261020-102619-ladsgroup.json 2471276 wikitext text/x-wiki 3rk6n47mwngpyicb0ipadn1jj1xy2ms 2471277 2471276 2026-10-20T10:26:35Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet 2471277 wikitext text/x-wiki m8zj81ktasfdvekizpiklvcz024ht8o 2471278 2471277 2026-10-20T10:26:37Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet 2471278 wikitext text/x-wiki 981qb2z7dcdlaqlyjia3qliu0wa6abc 2471279 2471278 2026-10-20T10:26:44Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet 2471279 wikitext text/x-wiki e9mzg2kphv3khygyjrlbqp7e039d88c 2471280 2471279 2026-10-20T10:27:21Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet 2471280 wikitext text/x-wiki 3xs024pi5ivrz9cow26r5x2o88sj2k9 2471281 2471280 2026-10-20T10:27:48Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs7001.magru.wmnet 2471281 wikitext text/x-wiki rn3s2tl8kbchn5vrm8rk0zcl85xtkwe 2471282 2471281 2026-10-20T10:28:25Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7001.*} and A:liberica 2471282 wikitext text/x-wiki 54awszn9shllb64kty9ijbr393auuqk 2471283 2471282 2026-10-20T10:28:59Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7001.*} and A:liberica 2471283 wikitext text/x-wiki m3lhafkc2wdgg0ekq3p7lq29c3l3aeb 2471284 2471283 2026-10-20T10:31:34Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host ceph-admin2001.codfw.wmnet 2471284 wikitext text/x-wiki 9danas9uje53u1sfyywluxgyaunm87r 2471285 2471284 2026-10-20T10:32:36Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet 2471285 wikitext text/x-wiki 0cphywvc7u4hlh8ccv1jfrl1qocnrak 2471286 2471285 2026-10-20T10:32:38Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet 2471286 wikitext text/x-wiki rnfwo337vh4c5ugmgk92h45lf4iujsm 2471287 2471286 2026-10-20T10:32:44Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet 2471287 wikitext text/x-wiki 23gdsd4x4ba1rqyx7jsa01k2xt9ex6x 2471288 2471287 2026-10-20T10:33:19Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet 2471288 wikitext text/x-wiki serat163qp1xaqwy8w6z6kiph0eytbv 2471289 2471288 2026-10-20T10:35:35Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ceph-admin2001.codfw.wmnet 2471289 wikitext text/x-wiki rylbd86b44ifq4sadhfmr036hn8j5jz 2471290 2471289 2026-10-20T10:36:06Z Stashbot 7414 jiji@cumin1004: START - Cookbook sre.discovery.service-route check mw-api-int-ro: maintenance 2471290 wikitext text/x-wiki lzi9buukvr6zxpiuk10vbtti6rgfhpt 2471291 2471290 2026-10-20T10:36:08Z Stashbot 7414 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-int-ro: maintenance 2471291 wikitext text/x-wiki docaa2rzqjkxgf0xake1meb0gja8lq7 2471292 2471291 2026-10-20T10:36:23Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P96868 and previous config saved to /var/cache/conftool/dbconfig/20261020-103621-ladsgroup.json 2471292 wikitext text/x-wiki m2awwiha7582mbxtycynsgyj0m51d11 2471293 2471292 2026-10-20T10:36:28Z Stashbot 7414 jiji@cumin1004: START - Cookbook sre.discovery.service-route check mw-api-ext-ro: maintenance 2471293 wikitext text/x-wiki mrmu08jfr721xlfhm8spd0fl3jcocbq 2471294 2471293 2026-10-20T10:36:30Z Stashbot 7414 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-ext-ro: maintenance 2471294 wikitext text/x-wiki 5hyumz3luzug0gignn4i3c18i74gzik 2471295 2471294 2026-10-20T10:36:44Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host matomo1003.eqiad.wmnet 2471295 wikitext text/x-wiki rfag945g38g55p43r2nsm6cdpalo2he 2471296 2471295 2026-10-20T10:38:29Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet 2471296 wikitext text/x-wiki 92cjnj2bqknk15ojhfcwm0lkndwupon 2471297 2471296 2026-10-20T10:38:31Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet 2471297 wikitext text/x-wiki f9dyw6uo3crcxz9oumeta2yqbddsx06 2471298 2471297 2026-10-20T10:38:37Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet 2471298 wikitext text/x-wiki rrzxfzda84jk8flvt6n8edf7rmg6u7f 2471299 2471298 2026-10-20T10:39:15Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet 2471299 wikitext text/x-wiki h1st8epansh8dc86japeax6apka7s4o 2471300 2471299 2026-10-20T10:39:57Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet 2471300 wikitext text/x-wiki t70cgsjcw12gw0kk4nnpydb7cc7je03 2471301 2471300 2026-10-20T10:40:38Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1003.eqiad.wmnet 2471301 wikitext text/x-wiki ndfqpqge2wfkxlkrkf8418y4wzpgml7 2471302 2471301 2026-10-20T10:42:11Z Stashbot 7414 arnaudb@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2471302 wikitext text/x-wiki 9aq0gxd5boln9dkw7vg9alwz4hnvybd 2471303 2471302 2026-10-20T10:42:14Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host krb1005.eqiad.wmnet 2471303 wikitext text/x-wiki jfh8gdns54v6dvwe9j0io7qc6allem5 2471304 2471303 2026-10-20T10:42:38Z Stashbot 7414 arnaudb@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2471304 wikitext text/x-wiki mkeru56g03yer6u15elz810ctfn99sv 2471305 2471304 2026-10-20T10:43:42Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet 2471305 wikitext text/x-wiki gau60yjkgpsd3chyqzdg0um31br9hda 2471306 2471305 2026-10-20T10:43:55Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet 2471306 wikitext text/x-wiki 7r69bnmijbmrbjtjwwluzztzk9o0d14 2471307 2471306 2026-10-20T10:44:39Z Stashbot 7414 root@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet 2471307 wikitext text/x-wiki 08xxr00ln6xgyzqhc4wryixhmlymooq 2471308 2471307 2026-10-20T10:44:41Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet 2471308 wikitext text/x-wiki gz3hellhniaxf39f9z57fevmsodulbz 2471309 2471308 2026-10-20T10:44:42Z Stashbot 7414 root@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad 2471309 wikitext text/x-wiki 5sbsnou4vvr6i0exkxkjetfam124bne 2471310 2471309 2026-10-20T10:46:10Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1005.eqiad.wmnet 2471310 wikitext text/x-wiki eds4wco91myggppim62et3zz3q9s1xo 2471311 2471310 2026-10-20T10:46:25Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T441633)', diff saved to https://phabricator.wikimedia.org/P96869 and previous config saved to /var/cache/conftool/dbconfig/20261020-104623-ladsgroup.json 2471311 wikitext text/x-wiki hz72ig2pz0zlphkdgj4dlcm3xa296ht 2471312 2471311 2026-10-20T10:46:34Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance 2471312 wikitext text/x-wiki shu41a90u4oc1fhx9uqhs1sgwbr4swl 2471313 2471312 2026-10-20T10:46:36Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96870 and previous config saved to /var/cache/conftool/dbconfig/20261020-104634-ladsgroup.json 2471313 wikitext text/x-wiki 8dteusrwzqntxfy62okfv3b524ygqet 2471314 2471313 2026-10-20T10:46:45Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host archiva1002.wikimedia.org 2471314 wikitext text/x-wiki ny68d2thsui4hdqmzk5vu0jckoonyqm 2471315 2471314 2026-10-20T10:47:46Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet 2471315 wikitext text/x-wiki bcnkfjgsp0vtgr8c3hzc1s7fosdjihl 2471316 2471315 2026-10-20T10:48:00Z Stashbot 7414 arnaudb@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2471316 wikitext text/x-wiki sbcinz2q3ulzw09axdbmcmpolwroazk 2471318 2471316 2026-10-20T10:48:52Z Stashbot 7414 arnaudb@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2471318 wikitext text/x-wiki l8qer7d7ofw87saecee5e9vqa25ypjo 2471319 2471318 2026-10-20T10:50:05Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471319 wikitext text/x-wiki 05zqqzagni24wf2rx7xujemi6dmsyle 2471320 2471319 2026-10-20T10:50:42Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host archiva1002.wikimedia.org 2471320 wikitext text/x-wiki 4z8pnq72joeixplac6nx36s9haatjqc 2471321 2471320 2026-10-20T10:53:30Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96871 and previous config saved to /var/cache/conftool/dbconfig/20261020-105328-ladsgroup.json 2471321 wikitext text/x-wiki r8rmae7u9zjf8wft0fsn1ir9f8y3o4h 2471322 2471321 2026-10-20T10:57:46Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2471322 wikitext text/x-wiki aia5un9oq1ns1vnjfviifr0e9t7b0z4 2471323 2471322 2026-10-20T10:58:28Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2471323 wikitext text/x-wiki pf1ld0nf37c9m2itddb08dvhxdkelkp 2471324 2471323 2026-10-20T10:58:47Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2471324 wikitext text/x-wiki 0emrpkfsm9l6uak09m7j3why8nxl4so 2471325 2471324 2026-10-20T10:59:28Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2471325 wikitext text/x-wiki fp18wcx9gjewdgod6n8p0mmybr7q1c6 2471326 2471325 2026-10-20T10:59:47Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet 2471326 wikitext text/x-wiki 5m7xoqdiwi9gov15y1amr47n0r5h061 2471327 2471326 2026-10-20T10:59:55Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet 2471327 wikitext text/x-wiki fffla8252eitt3rc2rof6qozi0qp54n 2471328 2471327 2026-10-20T11:00:08Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet 2471328 wikitext text/x-wiki g5zmz7ksm1kajz60r4if76ses17tn76 2471329 2471328 2026-10-20T11:01:19Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting P{lvs5005.*,lvs4009.*} and A:liberica 2471329 wikitext text/x-wiki 6f4x5ogcmbwg0y46pjhxuvduhs2ymah 2471330 2471329 2026-10-20T11:03:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet 2471330 wikitext text/x-wiki q1kszc73dv0rg9t4xiawx4np4rcoeqk 2471331 2471330 2026-10-20T11:03:32Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P96872 and previous config saved to /var/cache/conftool/dbconfig/20261020-110330-ladsgroup.json 2471331 wikitext text/x-wiki 96su8v15e17pkdf3vkiv5mu8xvhjid5 2471332 2471331 2026-10-20T11:03:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet 2471332 wikitext text/x-wiki ikicwicg979hxw2j2p3ud35p3xgvtyl 2471333 2471332 2026-10-20T11:04:21Z Stashbot 7414 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7008.magru.wmnet 2471333 wikitext text/x-wiki 5c08lero75wf0o3g3s7nqanmjbij8wv 2471334 2471333 2026-10-20T11:04:54Z Stashbot 7414 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool mw-api-int-ro in eqiad: T440896 2471334 wikitext text/x-wiki l1a42i27cg8hzg3vttvvh52lrz926dj 2471335 2471334 2026-10-20T11:04:58Z Stashbot 7414 jiji@cumin1004: START - Cookbook sre.dns.wipe-cache mw-api-int-ro.discovery.wmnet on all recursors 2471335 wikitext text/x-wiki 7awzcwfuipv39coxsok3zfbkszrfwvd 2471336 2471335 2026-10-20T11:04:59Z Stashbot 7414 jiji@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mw-api-int-ro.discovery.wmnet on all recursors 2471336 wikitext text/x-wiki lpk3zsltwj2grq7kite8ian26lxz83b 2471337 2471336 2026-10-20T11:05:57Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet 2471337 wikitext text/x-wiki m5ul324jmw33b4g8piq69x0wfsehi0d 2471338 2471337 2026-10-20T11:08:17Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs5005.*,lvs4009.*} and A:liberica 2471338 wikitext text/x-wiki ewofcknrqc4ekf5lkexs4ucwp3m93ey 2471339 2471338 2026-10-20T11:09:31Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet 2471339 wikitext text/x-wiki evbem3bt25ih1vwk0odkx3ib48kg7bn 2471340 2471339 2026-10-20T11:09:54Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet 2471340 wikitext text/x-wiki n26j2ahgadvcmh7alxky58nkjr2l8tr 2471341 2471340 2026-10-20T11:09:58Z Stashbot 7414 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-int-ro in eqiad: T440896 2471341 wikitext text/x-wiki t66jbvhkt2abhqciqnr6dwquvgv69lw 2471342 2471341 2026-10-20T11:10:00Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet 2471342 wikitext text/x-wiki 1bp4zcb9i41d2f0rlrkecqxv9feax0b 2471343 2471342 2026-10-20T11:10:16Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet 2471343 wikitext text/x-wiki lxedtqe8rh1za6juo5tnus55eeks9af 2471344 2471343 2026-10-20T11:13:33Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P96873 and previous config saved to /var/cache/conftool/dbconfig/20261020-111332-ladsgroup.json 2471344 wikitext text/x-wiki o1nezdtocn0vprh3h6hx6yf2zj5r40n 2471346 2471344 2026-10-20T11:21:43Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host acmechief1002.eqiad.wmnet 2471346 wikitext text/x-wiki ramilblcsby0zid4euj7992gi1ew5uf 2471347 2471346 2026-10-20T11:21:47Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reboot-single for host acmechief2002.codfw.wmnet 2471347 wikitext text/x-wiki gr1cssz04zy87yzycjp9ej1d86m978k 2471348 2471347 2026-10-20T11:23:35Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96874 and previous config saved to /var/cache/conftool/dbconfig/20261020-112333-ladsgroup.json 2471348 wikitext text/x-wiki c0c2c7auguyxb6xc6ery08adi4movh7 2471349 2471348 2026-10-20T11:23:55Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance 2471349 wikitext text/x-wiki n9j6km7qnyqprf80il7zh0glfzkur1n 2471350 2471349 2026-10-20T11:25:28Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1002.eqiad.wmnet 2471350 wikitext text/x-wiki ho96xaaejo6cbrrudxv7n5gls3zcmix 2471351 2471350 2026-10-20T11:25:36Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2002.codfw.wmnet 2471351 wikitext text/x-wiki o8ipqe6tr47htjrcxtoskftm882h1vh 2471353 2471351 2026-10-20T11:28:34Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2471353 wikitext text/x-wiki 4tui9ocu7qi5r6jrafibb6ocg9s05aj 2471354 2471353 2026-10-20T11:28:36Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2471354 wikitext text/x-wiki t60669lkzww18u94kjrlabde6czibd1 2471355 2471354 2026-10-20T11:28:48Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2471355 wikitext text/x-wiki r2396xdf27fzdh54r7h3choihiwjrfx 2471356 2471355 2026-10-20T11:29:39Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: Maintenance 2471356 wikitext text/x-wiki 91lcxriwvfzabeeakvm2hnjwg2sz2v9 2471357 2471356 2026-10-20T11:29:42Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96875 and previous config saved to /var/cache/conftool/dbconfig/20261020-112940-ladsgroup.json 2471357 wikitext text/x-wiki mc2mywx4zspctioywzcsa8nij8tmomh 2471359 2471357 2026-10-20T11:35:32Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2471359 wikitext text/x-wiki lxlead1nico0kmxn4bjge00932m48uf 2471360 2471359 2026-10-20T11:35:41Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2471360 wikitext text/x-wiki j2uqkqfwftve9xkjmw6413pcem4hzi7 2471361 2471360 2026-10-20T11:35:45Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . 2471361 wikitext text/x-wiki 2wyjogrszm5vjw8i3yujh74yeyl70lx 2471362 2471361 2026-10-20T11:36:36Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96876 and previous config saved to /var/cache/conftool/dbconfig/20261020-113634-ladsgroup.json 2471362 wikitext text/x-wiki bmcqlp1exljt2lbz7siqinuofzb9ba2 2471363 2471362 2026-10-20T11:37:49Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash rolling reboot on A:apifeatureusage 2471363 wikitext text/x-wiki 29mw3gukiyzjnqj1mz04z4p5sqtppja 2471364 2471363 2026-10-20T11:37:59Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2471364 wikitext text/x-wiki 23u49iahu91afy22zujzoiiqx0rcen2 2471365 2471364 2026-10-20T11:38:03Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . 2471365 wikitext text/x-wiki 8xcbqf5sfwlkmxidnvdh2vr3yqflprh 2471368 2471365 2026-10-20T11:43:41Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad 2471368 wikitext text/x-wiki jec35pfjmb69timcicqzkjarqoycchu 2471369 2471368 2026-10-20T11:44:40Z Stashbot 7414 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on backup1014.eqiad.wmnet with reason: Maintenance 2471369 wikitext text/x-wiki l1crhxski42v0sew93zfabybatobr4u 2471370 2471369 2026-10-20T11:45:27Z Stashbot 7414 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 8 hosts with reason: Maintenance 2471370 wikitext text/x-wiki jilf19mbgbg731ilmkbsmzua68ajtkg 2471371 2471370 2026-10-20T11:45:43Z Stashbot 7414 slyngshede@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7009.magru.wmnet 2471371 wikitext text/x-wiki fghn6vojrt1iugft4wpgnp4p2qm3pap 2471372 2471371 2026-10-20T11:45:45Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_magru and not P{cp7002*} and not P{cp7010*} and A:cp 2471372 wikitext text/x-wiki qzxkfumb6of0vl8hkaudtpudhfia74l 2471373 2471372 2026-10-20T11:46:38Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P96877 and previous config saved to /var/cache/conftool/dbconfig/20261020-114636-ladsgroup.json 2471373 wikitext text/x-wiki ptbfk26s3qmnypl4voxsrtgeeuzkz4g 2471374 2471373 2026-10-20T11:47:35Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1001.eqiad.wmnet 2471374 wikitext text/x-wiki fnq36x115gm5odeoh5mgjmuhtwhfy8m 2471375 2471374 2026-10-20T11:49:34Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling reboot on A:apifeatureusage 2471375 wikitext text/x-wiki rw9cs5rvcna4wtkdlw5nfzro36zriyh 2471377 2471375 2026-10-20T11:54:15Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1001.eqiad.wmnet 2471377 wikitext text/x-wiki dkwsoh5hls9cu0o2czrubvvhh30em9e 2471378 2471377 2026-10-20T11:56:37Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1008.eqiad.wmnet 2471378 wikitext text/x-wiki 2zyckfddjqpvajo52kipf3r4222cngd 2471379 2471378 2026-10-20T11:56:39Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P96878 and previous config saved to /var/cache/conftool/dbconfig/20261020-115638-ladsgroup.json 2471379 wikitext text/x-wiki opd4gk6om72c3mr2izq3e7e9gwjqj5i 2471381 2471379 2026-10-20T11:58:37Z Stashbot 7414 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: T440896 2471381 wikitext text/x-wiki stj6yp3qlxnzo95xy8tej1kupd2xn9w 0 17290 2470809 2469294 2026-10-19T15:43:23Z Stashbot 7414 phedenskog: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1353720 2470809 wikitext text/x-wiki rjsk5m1phmm6fqpx7ap1gj7f4iwq1j5 12 18678 2470851 2441956 2026-10-19T16:39:40Z DCaro (WMF) 18271 /* Credentials */ Added credential self-rotation procedure 2470851 wikitext text/x-wiki 096vlgm4amswigt7tjr4fnibyscynpa 0 22139 2470810 2463235 2026-10-19T15:43:34Z Neil Shah-Quinn (WMF) 36601 Link to how to kill a job 2470810 wikitext text/x-wiki o8r1cyynei9nzjbcjuco2y4527ibzea 498 30942 2470975 2470237 2026-10-19T21:32:36Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431374) 2470975 wikitext text/x-wiki 63faiemw0gxxhxtlu9ueat9xgps9qxp 2470976 2470975 2026-10-19T21:32:40Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431374) 2470976 wikitext text/x-wiki 85zhu5vzvrg74esfjoql123hot83994 2471160 2470976 2026-10-20T09:22:06Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'P{O:wmcs::openstack::eqiad1::control}' 2471160 wikitext text/x-wiki dqiz41qcu5qives36p0xs5q5ek8hn9n 2471183 2471160 2026-10-20T09:35:50Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::control}' 2471183 wikitext text/x-wiki 85eua9ds0fu1z8iktoc926jl82asnmi 2471345 2471183 2026-10-20T11:21:31Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'A:cloudlb AND A:eqiad' 2471345 wikitext text/x-wiki khg1k10qhcbmln1nk6a4uufbgcfdzjl 2471352 2471345 2026-10-20T11:27:02Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=0) on hosts matched by 'A:cloudlb AND A:eqiad' 2471352 wikitext text/x-wiki b2108pgicqgvq80klmqj81n435ix6ef 2471366 2471352 2026-10-20T11:41:30Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirtlocal1001} AND NOT P{F:kernelversion = 6.6.6}' 2471366 wikitext text/x-wiki eplav8tizcpzqblxp4mn1x4ujeonhta 2471367 2471366 2026-10-20T11:41:59Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirtlocal1001} AND NOT P{F:kernelversion = 6.6.6}' 2471367 wikitext text/x-wiki 977sr3bn1l95pg1oq4e4nqzd4bmi1qr 498 55577 2470806 2470175 2026-10-19T15:42:36Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf 2470806 wikitext text/x-wiki 5uil6bwnzn82bjvmgzhmsbylb2nsbji 2470807 2470806 2026-10-19T15:42:51Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf 2470807 wikitext text/x-wiki ltpnvjdzqin2u9nu7kcqlaenpmx2m3m 0 440900 2471376 2451324 2026-10-20T11:52:00Z Filippo Giunchedi 2939 /* Live Migrating Virtual Machines */ proper monospaced formatting 2471376 wikitext text/x-wiki tm5vqf81y41sh461ds53680xkac62eo 498 443946 2470905 2465847 2026-10-19T18:10:25Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 51c233f..8cbfa07: l10n updates (skr-arab) 2470905 wikitext text/x-wiki p6945nhsig3l3ie8icsiyd7bu5yrqsm 498 443947 2470904 2465845 2026-10-19T18:08:48Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 9ac1608..f23674a: l10n updates (skr-arab) 2470904 wikitext text/x-wiki d8ertsi76p6mll3mib086nd2hp5dgwj 0 446472 2470849 2353798 2026-10-19T16:36:35Z SNwachukwu (WMF) 39521 Added link to the section mentioned. 2470849 wikitext text/x-wiki gopx7yka9yaha65n23gi8j6nw0y4lp9 498 447034 2470906 2461426 2026-10-19T18:12:24Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 1161d3e..40f18f1: l10n updates (skr-arab) 2470906 wikitext text/x-wiki 7piiw179loipn7s9tni1pavlj7iod60 117 447050 2470988 2470159 2026-10-19T22:30:07Z BryanDavis 1604 /* XMPP bridge */ Reply 2470988 wikitext text/x-wiki p4l5nvfujkxe5c8z10uf4nl2lr4yrw4 0 449160 2470588 2470203 2026-10-19T13:43:14Z Dexbot 30554 Bot: Updating the report 2470588 wikitext text/x-wiki 8r5c1op1m3qahn3gytuiob7sjlkxnfb 2470998 2470588 2026-10-20T00:02:41Z Dexbot 30554 Bot: Updating the report 2470998 wikitext text/x-wiki do4pizs624cp4devme7c0xq0u5gebf6 2471179 2470998 2026-10-20T09:33:57Z Dexbot 30554 Bot: Updating the report 2471179 wikitext text/x-wiki m8ftxyeafsgl1otv29ekdmrozm1xw4i 0 451075 2470929 2463851 2026-10-19T20:30:34Z DAdedoyin (WMF) 39150 /* Historical */ 2470929 wikitext text/x-wiki ey7n9jbn4f98lfq7h4g3ko24mvn8df5 2470930 2470929 2026-10-19T20:33:08Z DAdedoyin (WMF) 39150 /* Current */ 2470930 wikitext text/x-wiki ityqy868vbhpdrvdvo1yy7thhy1uocp 0 451256 2471023 2435453 2026-10-20T05:56:19Z WFan (WMF) 38087 2471023 wikitext text/x-wiki om3yxzx8bt8nywujcfe9vk2m5g41cio 2471028 2471023 2026-10-20T06:09:27Z WFan (WMF) 38087 2471028 wikitext text/x-wiki kwy5nxmlr92qgnuux4ieg1ch9h3r800 0 453382 2471260 2455989 2026-10-20T10:14:55Z CMooney (WMF) 24288 /* Troubleshooting */ 2471260 wikitext text/x-wiki 38uyv2j083kxy85a06h6jc210z81qde 116 453906 2471267 2470182 2026-10-20T10:21:26Z Gitlabaccountapprovalbot 37332 boyazas was rejected. 2471267 wikitext text/x-wiki l7095pzq45q44ssv1stuc0fg3vyf2em 0 455474 2471012 2272181 2026-10-20T05:14:21Z PHedenskog (WMF) 5986 Correct link to get the API key 2471012 wikitext text/x-wiki jf13wibms7hnzuwq9y15c0t83dzg4lq 0 458676 2470595 2470215 2026-10-19T13:47:50Z MVernon (WMF) 25893 Add link to the gitlab-rw quota increase ticket 2470595 wikitext text/x-wiki cfkos7e7auxbokooemn9he7ly26hggb 0 459041 2470898 2464051 2026-10-19T17:32:26Z KSoučková-WMF 35993 /* Preparation for WMF */ add reminder about rsyslog change 2470898 wikitext text/x-wiki 4txe3g26px583pfh8g2rauejt08lte5 2471250 2470898 2026-10-20T10:10:02Z Krinkle 44 /* Out of scope */ 2471250 wikitext text/x-wiki doennhhnfn9edghplx5i4d4tr3od2qe 2471257 2471250 2026-10-20T10:14:01Z Krinkle 44 /* Out of scope */ 2471257 wikitext text/x-wiki 5yf89m0gkc1d0h3x6j1wbp0f4q5asba 0 459290 2470907 2470265 2026-10-19T18:54:25Z SNwachukwu (WMF) 39521 Add an item to the Next deployment train to be deployed 2470907 wikitext text/x-wiki mvwsk3te5l78wa8fvg20p7gkins9164 0 459560 2470963 2463860 2026-10-19T21:10:59Z DAdedoyin (WMF) 39150 Test 2470963 wikitext text/x-wiki c9bxt10jyuki34n3y2xju1nlalcm7r9 0 459699 2471009 2470029 2026-10-20T04:02:07Z Stashbot 7414 eileen: config revision changed from 14485224 to 60162562 2471009 wikitext text/x-wiki 7p9gk3uiefakn7at7sq9z6tzaqym74g 2471010 2471009 2026-10-20T04:21:27Z Stashbot 7414 eileen: * civicrm upgraded from 0d24ecd6 to 348449da 2471010 wikitext text/x-wiki ltdm03vdqcjz0sliztr29zt58nxbd24 2471011 2471010 2026-10-20T05:01:32Z Stashbot 7414 eileen: config revision changed from 60162562 to fb7d4101 2471011 wikitext text/x-wiki mmadsi03y88f6ogl96s55ypow1ycpok 0 460417 2470795 2470230 2026-10-19T15:33:02Z Cheaterblocker 60771 /* KServe Predict Endpoint */ 2470795 wikitext text/x-wiki k46wgovr33bud4y75eqgxj6k60ch84y 0 460548 2471317 2446654 2026-10-20T10:48:42Z JMonton-WMF 47887 adding more info 2471317 wikitext text/x-wiki 1x1jfdzh5r3wo41gfjena7nj7rp7l4t 0 460779 2470914 2466254 2026-10-19T19:29:44Z EEvans (WMF) 5035 add SLIs 2470914 wikitext text/x-wiki 3i3yx2rl44bp5abxz4dymlu4gom2vgf 2470915 2470914 2026-10-19T19:31:13Z EEvans (WMF) 5035 remove some instructions link 2470915 wikitext text/x-wiki q58ah7fhzfktx8zhdtnvvzoqkasvyuv 2470916 2470915 2026-10-19T19:32:42Z EEvans (WMF) 5035 /* Service Level Indicators (SLIs) */ expound a bit 2470916 wikitext text/x-wiki 4m7u7tvvka59v75my62fc7g90flks77 2470993 2470916 2026-10-19T23:42:22Z EEvans (WMF) 5035 operational section 2470993 wikitext text/x-wiki a1edtil0mcbe6wifqu6ydyfiazqk86z 2471004 2470993 2026-10-20T00:16:11Z EEvans (WMF) 5035 a stab at some ideal SLO targets 2471004 wikitext text/x-wiki s2fesh2p9xemvbai2xww40fltw6vt8v 0 460789 2470492 2464726 2026-10-19T13:03:45Z GKyziridis-WMF 39160 Update on the qwen models. Add table for each model on realistic targets 2470492 wikitext text/x-wiki 4dze5ed8htmocc26umq5o6h8nityd54 2470503 2470492 2026-10-19T13:04:35Z GKyziridis-WMF 39160 2470503 wikitext text/x-wiki n7bcnsfv2ryx71c1bzolpou0zvfa7r1 2470504 2470503 2026-10-19T13:05:10Z GKyziridis-WMF 39160 Remove empty lines between sections 2470504 wikitext text/x-wiki mhiv68oks124mvpuf1tzert9vokpy75 0 460803 2471358 2469290 2026-10-20T11:34:53Z ESanders (WMF) 953 2471358 wikitext text/x-wiki ljpktv1dala9i5uwcw4051ffb1bsqjw 0 460830 2470658 2026-10-19T14:26:51Z APuthin-WMF 52425 Adding first version of page 2470658 wikitext text/x-wiki mjx104he1fdvlfmmi4t2h7sd4dj92et 2470745 2470658 2026-10-19T15:07:50Z APuthin-WMF 52425 Add link to cloud-announce message 2470745 wikitext text/x-wiki rqh134j392aslmo1zqv0kw9hovly8ih 2470853 2470745 2026-10-19T16:43:12Z DCaro (WMF) 18271 /* ToolsDB / Replicas */ Updated to point to the self-service procedure and re-organize accordingly 2470853 wikitext text/x-wiki t4p38c6nc6bbqy1072bpeuj6ms0pye7 0 460831 2471110 2026-10-20T08:38:53Z DPogorzelski-WMF 48508 Created page with "'''Chart name''': gateway-api-crds '''Chart version''': v1.4.1, standard channel (WMF chart version 0.0.1) '''Phab task''': {{Phabricator/en|T436650}} '''Reviewers''': <name #1>, <name #2> '''Link to repo''': Upstream publishes no Helm chart or OCI artifact, only plain manifests as release assets: https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/standard-install.yaml '''Link to source code''': https://github.com/kubernetes-sigs/gateway-api/tre..." 2471110 wikitext text/x-wiki 4wc7zur2km4n4wxv6hxwna0i12k1yq6 0 460832 2471380 2026-10-20T11:57:42Z OKarakaya-WMF 40765 initial commit with todos in draft mode 2471380 wikitext text/x-wiki otshmcogl5xix4on9c42fi4ewtreqhl