Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.3 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2416318 2415554 2026-05-20T00:34:26Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2416318 wikitext text/x-wiki avilc2ylhk78ola6oob34vbmi8o5rpy 2416325 2416318 2026-05-20T00:47:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2416325 wikitext text/x-wiki 73xz7curss4ikpal3ldaw9cxmf1k6iq 2416332 2416325 2026-05-20T00:57:20Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2416332 wikitext text/x-wiki gnb1qf5vufzvcs8buwugndcm9l2dsks 2416333 2416332 2026-05-20T00:58:51Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2416333 wikitext text/x-wiki 8e3995tqjri00vptdtl5fng42ia9quu 0 4108 2415673 2415216 2026-05-19T12:27:22Z ScheduleDeploymentBot 37566 Add [[gerrit:1289328]] to Tuesday, May 19 UTC afternoon backport window 2415673 wikitext text/x-wiki 05vbzppneewro0f793u3y4tvlmqjx9d 2415862 2415673 2026-05-19T14:32:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1288953]] to Tuesday, May 19 UTC late backport window 2415862 wikitext text/x-wiki sc5mbpkdqfv8mtuuanzglibeus43arb 2416093 2415862 2026-05-19T17:27:02Z ScheduleDeploymentBot 37566 Add [[gerrit:1288983]] to Tuesday, May 19 UTC late backport window 2416093 wikitext text/x-wiki 8s2wor4gamd5260yg4rfjff4ycwjkto 2416111 2416093 2026-05-19T17:51:00Z ScheduleDeploymentBot 37566 Add [[gerrit:1288999]] to Tuesday, May 19 UTC late backport window 2416111 wikitext text/x-wiki hbyohpxs0g4qe5g0bcqkx7ouvp6a9b3 2416204 2416111 2026-05-19T19:50:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1289352]] to Tuesday, May 19 UTC late backport window 2416204 wikitext text/x-wiki r3jkh2rydtiy1g93id6918ekw7pegq7 2416314 2416204 2026-05-20T00:06:58Z ScheduleDeploymentBot 37566 Add [[gerrit:1289446]] to Wednesday, May 20 UTC afternoon backport window 2416314 wikitext text/x-wiki 7j9ojp0dnp0fh6twv03omowmmx9g6bn 2416381 2416314 2026-05-20T06:52:10Z ScheduleDeploymentBot 37566 Add [[gerrit:1289743]] to Wednesday, May 20 UTC morning backport window 2416381 wikitext text/x-wiki oqgkomj4io8liweczdievb3r65t6ma2 2416382 2416381 2026-05-20T06:52:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1289744]] to Wednesday, May 20 UTC morning backport window 2416382 wikitext text/x-wiki 6smayutl5wgvpaj7jew7psapvo5et2s 2416537 2416382 2026-05-20T09:52:59Z ScheduleDeploymentBot 37566 Add [[gerrit:1289736]] to Wednesday, May 20 UTC afternoon backport window 2416537 wikitext text/x-wiki nxcy7usn50p55tq3zp13vqnc0cq00dh 2416573 2416537 2026-05-20T10:21:54Z ScheduleDeploymentBot 37566 Add [[gerrit:1287433]] to Wednesday, May 20 UTC afternoon backport window 2416573 wikitext text/x-wiki 6kqqfioudqcallt2l6in7mc5fza0mhw 498 6726 2416309 2415551 2026-05-19T23:44:24Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2416309 wikitext text/x-wiki bwld7gwfjlfvps9hrjxinzej5wfep2o 2416310 2416309 2026-05-19T23:54:16Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2416310 wikitext text/x-wiki 66omzsrqp4rrmzdpc059k9j5g8qys7v 2416311 2416310 2026-05-20T00:00:41Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2416311 wikitext text/x-wiki b04ws37tqyu2x96xfk5eulupbcntxsi 2416315 2416311 2026-05-20T00:10:11Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2416315 wikitext text/x-wiki 83tmkglsczn9bbggwj66ukapvs589ud 2416316 2416315 2026-05-20T00:23:25Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2416316 wikitext text/x-wiki nrvfexgklafiz4a9xdwa72mw6z6vrru 2416317 2416316 2026-05-20T00:34:16Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2416317 wikitext text/x-wiki d2rx911xe5kb2a57768ax89b3we108w 2416323 2416317 2026-05-20T00:46:35Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2416323 wikitext text/x-wiki req2jefs1ocdtklrpowyj5of2wubtjg 2416327 2416323 2026-05-20T00:47:44Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2416327 wikitext text/x-wiki ogmvh22vkntj4f9fyve9xtkmwm3qg44 2416455 2416327 2026-05-20T08:26:12Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2416455 wikitext text/x-wiki 52wvcs9mo6igj2ggfy886dp71pv8txb 2416456 2416455 2026-05-20T08:26:14Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli 2416456 wikitext text/x-wiki 0svk5c56iejlaj37qrmia9opv553i9x 2416457 2416456 2026-05-20T08:26:23Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2416457 wikitext text/x-wiki 9ctktri6iu8an6l5tkaidq2z0at29tf 2416459 2416457 2026-05-20T08:27:39Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli 2416459 wikitext text/x-wiki d3x0ny7e95c4r83hyy9qh3pd1l28okz 0 7919 2415622 2415621 2026-05-19T12:00:13Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1283.eqiad.wmnet with reason: host reimage 2415622 wikitext text/x-wiki jwva2et5ys3b64wk6ykkq287392094l 2415623 2415622 2026-05-19T12:00:33Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2006.codfw.wmnet 2415623 wikitext text/x-wiki hre4f5c58dg3t97zohuil7wcjcrdkrq 2415624 2415623 2026-05-19T12:01:31Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host apus-be2005.codfw.wmnet 2415624 wikitext text/x-wiki gjsyxziulnv7ineqn6udlia2sbklydd 2415625 2415624 2026-05-19T12:03:45Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1281.eqiad.wmnet with reason: host reimage 2415625 wikitext text/x-wiki l8dn6amu78evt6xps0c9lbswmmesz7y 2415626 2415625 2026-05-19T12:04:47Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1277.eqiad.wmnet with OS trixie 2415626 wikitext text/x-wiki h57v1npdtucjhhrg21kmk3bq7amb564 2415627 2415626 2026-05-19T12:06:02Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2007.codfw.wmnet 2415627 wikitext text/x-wiki cu3fqeyp1oz7fa8hkbk7ubxt1a2g16n 2415628 2415627 2026-05-19T12:06:07Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus6002.drmrs.wmnet 2415628 wikitext text/x-wiki 9u2l99a7rfwi8qkddu3xx35t8qga5k5 2415629 2415628 2026-05-19T12:06:11Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1284.eqiad.wmnet with OS trixie 2415629 wikitext text/x-wiki 5yitq9dp9arv347oeulq02s5hh0e8vg 2415630 2415629 2026-05-19T12:06:31Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet 2415630 wikitext text/x-wiki p04342rx0fl6lsmnll5outzojucbbyr 2415631 2415630 2026-05-19T12:06:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti7004.magru.wmnet 2415631 wikitext text/x-wiki k6fyfit3dqkxs1462kqw2ji7122h2rr 2415632 2415631 2026-05-19T12:07:10Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2005.codfw.wmnet 2415632 wikitext text/x-wiki r8jr21r65yl8x6ql1cj00uadvjqczak 2415633 2415632 2026-05-19T12:07:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1288.eqiad.wmnet with reason: host reimage 2415633 wikitext text/x-wiki e8zmd6y56wu11e3ayrnxfav8u7vtcst 2415634 2415633 2026-05-19T12:07:48Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2415634 wikitext text/x-wiki t6sf5pgcgmsp2ada3ydszpacenhg919 2415635 2415634 2026-05-19T12:08:09Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host apus-be2004.codfw.wmnet 2415635 wikitext text/x-wiki kp1zxti8mt1wvrmerfezekfzcx5i8qt 2415636 2415635 2026-05-19T12:08:21Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2415636 wikitext text/x-wiki ssdae5f6rvjt7rv89fmy6rqqx5im04l 2415637 2415636 2026-05-19T12:08:46Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2415637 wikitext text/x-wiki 3ax0gb8vawkvdzq2avol55zbejfsur7 2415638 2415637 2026-05-19T12:09:38Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2415638 wikitext text/x-wiki 1x6q44hreqaz27vg1gtib49fcp2fr8r 2415639 2415638 2026-05-19T12:10:20Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1282.eqiad.wmnet with OS trixie 2415639 wikitext text/x-wiki 3ufccm9zbh1bi3hs0xyr3oco61s4sgx 2415641 2415639 2026-05-19T12:11:27Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1278.eqiad.wmnet with reason: host reimage 2415641 wikitext text/x-wiki kmy5qxflij2uhpmae2jsuziony1rf1s 2415642 2415641 2026-05-19T12:12:02Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6002.drmrs.wmnet 2415642 wikitext text/x-wiki 6lbxuedglpd30i6o87uwum0r215v4ih 2415643 2415642 2026-05-19T12:12:07Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus7002.magru.wmnet 2415643 wikitext text/x-wiki 36kfcra1pm9os2jh0eugyguk1vueer1 2415644 2415643 2026-05-19T12:12:47Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2004.codfw.wmnet 2415644 wikitext text/x-wiki npjece967t0ltfsu1liooenrl9mco4t 2415645 2415644 2026-05-19T12:14:10Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db1279.eqiad.wmnet with reason: host reimage 2415645 wikitext text/x-wiki fh85hwhhon1g8ttcp1vwn6dvk6bb7qj 2415646 2415645 2026-05-19T12:14:17Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1283.eqiad.wmnet with OS trixie 2415646 wikitext text/x-wiki ozzs8pz8yc7x58vrtolrevow24k2sky 2415647 2415646 2026-05-19T12:14:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7004.magru.wmnet 2415647 wikitext text/x-wiki 40gpyz6m50584sfg500zgiyfbfy1tj9 2415648 2415647 2026-05-19T12:14:54Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db1280.eqiad.wmnet with reason: host reimage 2415648 wikitext text/x-wiki 32qsh9uaybhjwe2bfvf4bihrx7zp98f 2415649 2415648 2026-05-19T12:15:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet 2415649 wikitext text/x-wiki ex1h3l2jdpa28t9uwdzglbffb2ijj8x 2415650 2415649 2026-05-19T12:15:11Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7004.magru.wmnet 2415650 wikitext text/x-wiki 8tvfsqoxopp0bi5v3pfal35frh1ojkj 2415651 2415650 2026-05-19T12:15:29Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet 2415651 wikitext text/x-wiki 6dkyld9nmjovf3ciphicqan3t9gzux2 2415652 2415651 2026-05-19T12:15:34Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1286.eqiad.wmnet with reason: host reimage 2415652 wikitext text/x-wiki bo3odura84epunknhebgliaq7cu7rjh 2415653 2415652 2026-05-19T12:17:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1281.eqiad.wmnet with OS trixie 2415653 wikitext text/x-wiki nferwgkpuaz4ispktkb2bfdwkwq18cm 2415656 2415653 2026-05-19T12:18:03Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus7002.magru.wmnet 2415656 wikitext text/x-wiki mhys2vmtradmt34x067yag8q35n26tq 2415657 2415656 2026-05-19T12:18:08Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus2006.codfw.wmnet 2415657 wikitext text/x-wiki 4pujjnrnc30bdbx2f2cpr9wnlow40wb 2415658 2415657 2026-05-19T12:18:56Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1064.eqiad.wmnet 2415658 wikitext text/x-wiki 5ds0ao2lv1yv0fk3lhzo61vcr70wwz0 2415659 2415658 2026-05-19T12:19:04Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2062.codfw.wmnet 2415659 wikitext text/x-wiki fep72fjh1dndgx3jnaf48ij0zlje6ch 2415661 2415659 2026-05-19T12:19:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3005.esams.wmnet 2415661 wikitext text/x-wiki 9e0v4wswgfyimu2qvqa7f5iwp8t667q 2415662 2415661 2026-05-19T12:20:55Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1279.eqiad.wmnet with OS trixie 2415662 wikitext text/x-wiki 3a0sf5kqpgismdy5hodmo9jwyue798p 2415663 2415662 2026-05-19T12:21:25Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1288.eqiad.wmnet with OS trixie 2415663 wikitext text/x-wiki t36s2j01n83nsth4l4qmxyg61si7ax0 2415664 2415663 2026-05-19T12:22:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3005.esams.wmnet 2415664 wikitext text/x-wiki 1m4ukod5tyq8gnjb7z1m90kwt99k76n 2415666 2415664 2026-05-19T12:23:38Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1279.eqiad.wmnet with OS trixie 2415666 wikitext text/x-wiki myy8yt80eqr5bfmvbw12z5wifhi8wmq 2415667 2415666 2026-05-19T12:24:14Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1287.eqiad.wmnet with reason: host reimage 2415667 wikitext text/x-wiki n68s0irgn4mrhzi4yfc749at0pxjwwx 2415668 2415667 2026-05-19T12:25:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1278.eqiad.wmnet with OS trixie 2415668 wikitext text/x-wiki 50taznphxg9ucvubbypje9krrxn2lwd 2415669 2415668 2026-05-19T12:26:05Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2062.codfw.wmnet 2415669 wikitext text/x-wiki 4k6pbktujnwcwvlx70al7yy4dur0uau 2415670 2415669 2026-05-19T12:26:08Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1064.eqiad.wmnet 2415670 wikitext text/x-wiki 9pu8ngez5nxlzam60ft2hptixws5f54 2415671 2415670 2026-05-19T12:26:54Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1065.eqiad.wmnet 2415671 wikitext text/x-wiki oz440hwkqpbr7958c88p057egepcj3t 2415672 2415671 2026-05-19T12:27:03Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2063.codfw.wmnet 2415672 wikitext text/x-wiki akm9istw2ex22p8x38g33d7eivlfceu 2415674 2415672 2026-05-19T12:27:24Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan2001.codfw.wmnet 2415674 wikitext text/x-wiki mygnuwoeoqixgft7972vgi6wtveb3pt 2415675 2415674 2026-05-19T12:28:04Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2006.codfw.wmnet 2415675 wikitext text/x-wiki 1uqwtakx0h6i3jd14hny1uommr9rqwn 2415676 2415675 2026-05-19T12:28:08Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus2008.codfw.wmnet 2415676 wikitext text/x-wiki 09bjljra0nekdphbakw84yldp7k3han 2415677 2415676 2026-05-19T12:29:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1280.eqiad.wmnet with OS trixie 2415677 wikitext text/x-wiki 7riam8vqe8lglufe9d6j80qen5heiir 2415679 2415677 2026-05-19T12:32:02Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1286.eqiad.wmnet with OS trixie 2415679 wikitext text/x-wiki 7fnn721j4pfwwwiyt3re1swsvpspu4h 2415680 2415679 2026-05-19T12:32:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3005.esams.wmnet 2415680 wikitext text/x-wiki 6f311efbofuzirfu0drt7rcfla7gobs 2415682 2415680 2026-05-19T12:32:23Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3005.esams.wmnet 2415682 wikitext text/x-wiki axbayfr8z83hvg09rlxvlvr55bglkdr 2415683 2415682 2026-05-19T12:33:06Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2063.codfw.wmnet 2415683 wikitext text/x-wiki jwosi217den8mybj1pzcz2csk3djz5i 2415684 2415683 2026-05-19T12:33:55Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2064.codfw.wmnet 2415684 wikitext text/x-wiki rb5gw63rflx9078o22nfyqu8xumxs21 2415685 2415684 2026-05-19T12:34:10Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1065.eqiad.wmnet 2415685 wikitext text/x-wiki c1q1uhnk8hgp4uw0wkb8qn35b8y2e10 2415686 2415685 2026-05-19T12:34:21Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1066.eqiad.wmnet 2415686 wikitext text/x-wiki e5dyiuepyk5z4i8kuhu21ikv1ospc37 2415687 2415686 2026-05-19T12:35:18Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1289.eqiad.wmnet with OS trixie 2415687 wikitext text/x-wiki 9zsok9mwbpmz8z0lrav5vfa3jw8hw27 2415688 2415687 2026-05-19T12:35:28Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan2001.codfw.wmnet 2415688 wikitext text/x-wiki pk073x1jdbq7hxbdy3pfzwils61og87 2415689 2415688 2026-05-19T12:35:32Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1290.eqiad.wmnet with OS trixie 2415689 wikitext text/x-wiki krrtx56xl0grb5cq7rljh7i0adqxvm4 2415690 2415689 2026-05-19T12:36:01Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2008.codfw.wmnet 2415690 wikitext text/x-wiki qhc7pxl99nqand71wahszkzjlui86m5 2415691 2415690 2026-05-19T12:36:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1279.eqiad.wmnet with reason: host reimage 2415691 wikitext text/x-wiki 0sxa5h2v4pgn0no66ofo0j2xhwr8xqk 2415692 2415691 2026-05-19T12:36:32Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db1279.eqiad.wmnet with reason: host reimage 2415692 wikitext text/x-wiki 6iz5m6w9jguml56421pu6flb0wp9q7j 2415693 2415692 2026-05-19T12:36:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1287.eqiad.wmnet with OS trixie 2415693 wikitext text/x-wiki g0mpgf6cb6jnrvxer1milblfova0umt 2415694 2415693 2026-05-19T12:39:41Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2064.codfw.wmnet 2415694 wikitext text/x-wiki atd172mewzeuhy45ac94pwtccy660b5 2415695 2415694 2026-05-19T12:39:49Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2065.codfw.wmnet 2415695 wikitext text/x-wiki swsvljev03yk0fy9niaupjxjjlwllph 2415696 2415695 2026-05-19T12:41:19Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1066.eqiad.wmnet 2415696 wikitext text/x-wiki el72zx0lzyatm4ra3p2kfhvu9aesolv 2415697 2415696 2026-05-19T12:41:41Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1067.eqiad.wmnet 2415697 wikitext text/x-wiki ey7emy2p324pt0vo95fhrmytemvu6ox 2415698 2415697 2026-05-19T12:44:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet 2415698 wikitext text/x-wiki 80tuz1qxo654wo9wnkhdxtvwrq2rx60 2415701 2415698 2026-05-19T12:45:41Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2065.codfw.wmnet 2415701 wikitext text/x-wiki 8of0z2myrrht63xfcb12ry2fozsvhnx 2415703 2415701 2026-05-19T12:46:46Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet 2415703 wikitext text/x-wiki bwf98xhhtefmgpwfrhqylf8k7ckf8ta 2415704 2415703 2026-05-19T12:47:33Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.decommission for hosts db2149.codfw.wmnet 2415704 wikitext text/x-wiki tpk176i5bd0ytec2kmyiqipvuel9p65 2415705 2415704 2026-05-19T12:48:04Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1067.eqiad.wmnet 2415705 wikitext text/x-wiki 7d9wscscprr79jc7ssilaam4fd0qzbs 2415706 2415705 2026-05-19T12:48:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1289.eqiad.wmnet with reason: host reimage 2415706 wikitext text/x-wiki dx90m1lji02dul1k2rh9q9hwhksmiye 2415707 2415706 2026-05-19T12:48:49Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1068.eqiad.wmnet 2415707 wikitext text/x-wiki 7ovugux3ktlnvy4uw306bkmdfsxkqp1 2415708 2415707 2026-05-19T12:48:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1290.eqiad.wmnet with reason: host reimage 2415708 wikitext text/x-wiki 3mwxivi1fl3zdddiylqbuj64dexxx98 2415709 2415708 2026-05-19T12:49:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3006.esams.wmnet 2415709 wikitext text/x-wiki oyu7jcsuzwd2h2zrftzh2ct07fmbodk 2415710 2415709 2026-05-19T12:49:55Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet 2415710 wikitext text/x-wiki 5gvft3tht1d85yixurxnrx0khysogw5 2415711 2415710 2026-05-19T12:50:43Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2066.codfw.wmnet 2415711 wikitext text/x-wiki rh1sk11t2nhrvzatf86qbiekq3r60pa 2415712 2415711 2026-05-19T12:50:51Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1279.eqiad.wmnet with OS trixie 2415712 wikitext text/x-wiki p9fbwwk26vhhkjf2pgpq6atz4my03yk 2415713 2415712 2026-05-19T12:52:08Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.dns.netbox 2415713 wikitext text/x-wiki 0k52ge439wi0lbii4l6u58hmba64anx 2415716 2415713 2026-05-19T13:08:45Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.dns.netbox 2415716 wikitext text/x-wiki c8pbvyt3paimswjnms93stwidtb5377 2415717 2415716 2026-05-19T13:09:25Z Stashbot 7414 dbrant@deploy1003: Started scap sync-world: Backport for [[gerrit:1289328|Add "get_login_creds" permission to Android app for auth domain. (T426010)]] 2415717 wikitext text/x-wiki pqgrf5kf78vurlfj9r227p2maz8nn5j 2415718 2415717 2026-05-19T13:09:35Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2068.codfw.wmnet 2415718 wikitext text/x-wiki bwmcr0g453w6u5ystx8vmuymsqmasf9 2415719 2415718 2026-05-19T13:09:40Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2069.codfw.wmnet 2415719 wikitext text/x-wiki q9tmg2u96o5tezx2tq2r72htmoo0j37 2415720 2415719 2026-05-19T13:10:12Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1289.eqiad.wmnet with OS trixie 2415720 wikitext text/x-wiki isuldjr6ykyz5gtkuuxbzjupydlggeb 2415721 2415720 2026-05-19T13:11:14Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3007.esams.wmnet 2415721 wikitext text/x-wiki souk6nljjc8m1xjcnaipousng6i0xsv 2415722 2415721 2026-05-19T13:11:35Z Stashbot 7414 dbrant@deploy1003: dbrant: Backport for [[gerrit:1289328|Add "get_login_creds" permission to Android app for auth domain. (T426010)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2415722 wikitext text/x-wiki tj89gtdesasf5hi08s22pg9ikljxh2r 2415723 2415722 2026-05-19T13:11:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3007.esams.wmnet 2415723 wikitext text/x-wiki pnfelshpibwv9q2d31th1bnrbks9hq2 2415724 2415723 2026-05-19T13:11:58Z Stashbot 7414 moritzm: failover Ganeti cluster in esams to ganeti3005 2415724 wikitext text/x-wiki funjuewumpvkr104vjofv6f1v3p6jm2 2415725 2415724 2026-05-19T13:12:10Z Stashbot 7414 dbrant@deploy1003: dbrant: Continuing with deployment 2415725 wikitext text/x-wiki rk89st99vooo95uen45v2w5gtn3e0yr 2415726 2415725 2026-05-19T13:12:20Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet 2415726 wikitext text/x-wiki p1mn92pey990r2biu90dshoj90nn6ik 2415727 2415726 2026-05-19T13:13:04Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1290.eqiad.wmnet with OS trixie 2415727 wikitext text/x-wiki mq79s2yhckus5ujnuykja05bk8msn2d 2415728 2415727 2026-05-19T13:13:15Z Stashbot 7414 cezmunsta: Removing db2143 from zarcillo T424171 2415728 wikitext text/x-wiki 9htpgccfaqk8gg0j9nctf12iqw51y4f 2415729 2415728 2026-05-19T13:13:26Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2143.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwilliams@cumin1003" 2415729 wikitext text/x-wiki d59ytzftpc5cf2pwzy4aqikpeemxnp4 2415730 2415729 2026-05-19T13:13:53Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2143.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwilliams@cumin1003" 2415730 wikitext text/x-wiki 8db09eyfah05bxwefcdfr8aps1neqxs 2415731 2415730 2026-05-19T13:13:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2415731 wikitext text/x-wiki 0nf9gliqjb3c3py5cpqh0cwpj6hwbk0 2415732 2415731 2026-05-19T13:13:56Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2143.codfw.wmnet 2415732 wikitext text/x-wiki eq8np817ur0zv9i1i2173q9n2sgs047 2415733 2415732 2026-05-19T13:14:22Z Stashbot 7414 cezmunsta: Removing db2143 from orchestrator T424171 2415733 wikitext text/x-wiki k5sfknomzcmg0s6487vwdq6n2gsa4no 2415734 2415733 2026-05-19T13:15:26Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1070.eqiad.wmnet 2415734 wikitext text/x-wiki 5269dnj0ph8dapi8yqawih2eculkivf 2415735 2415734 2026-05-19T13:15:30Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2069.codfw.wmnet 2415735 wikitext text/x-wiki o4htzqz03fge5840vwyk8qxxiaog644 2415736 2415735 2026-05-19T13:15:34Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2070.codfw.wmnet 2415736 wikitext text/x-wiki 1uqu0xs5h9idpgj5l27tjnxvztfl2jd 2415737 2415736 2026-05-19T13:15:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3008.esams.wmnet 2415737 wikitext text/x-wiki kyfhq7qm7i2ubk81leleicrpcuq76hr 2415738 2415737 2026-05-19T13:15:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2045.codfw.wmnet 2415738 wikitext text/x-wiki ruq2ha2jzhlj0qs4h1klds5lccqa6cw 2415739 2415738 2026-05-19T13:16:26Z Stashbot 7414 dbrant@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289328|Add "get_login_creds" permission to Android app for auth domain. (T426010)]] (duration: 07m 00s) 2415739 wikitext text/x-wiki r5n7edtf5ixdwpyn4wetz4m2z9gkwy4 2415740 2415739 2026-05-19T13:17:43Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1071.eqiad.wmnet 2415740 wikitext text/x-wiki a1uu7q2ruhlaevtt3a5fuqymswluny3 2415741 2415740 2026-05-19T13:18:24Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti3008.esams.wmnet 2415741 wikitext text/x-wiki 2ttoixpt86nr6pkgi0knggts8pxmud0 2415742 2415741 2026-05-19T13:21:11Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2045.codfw.wmnet 2415742 wikitext text/x-wiki 3cy40w44c7ob987546ssd55qr1xtxrc 2415743 2415742 2026-05-19T13:21:18Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet 2415743 wikitext text/x-wiki hxxjva7kfvxmcja3mtdsu1gtz8nb7sp 2415744 2415743 2026-05-19T13:21:56Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.reimage for host db1204.eqiad.wmnet with OS trixie 2415744 wikitext text/x-wiki l4m5bsad9h91w50z2htro3w8r221h4j 2415745 2415744 2026-05-19T13:22:47Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2070.codfw.wmnet 2415745 wikitext text/x-wiki ok8fhfh44rmbypmd2j7uoprxdiuoqpn 2415746 2415745 2026-05-19T13:22:51Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2071.codfw.wmnet 2415746 wikitext text/x-wiki qb2r4fvr2e7lmifgniwi6usa31bl9iq 2415747 2415746 2026-05-19T13:23:13Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1071.eqiad.wmnet 2415747 wikitext text/x-wiki 1j5oz5ifpzglvbx57vgrl4lwdeksi7h 2415748 2415747 2026-05-19T13:23:23Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1289070|Forward-compatibility for serialization of ContentHolder in ParserOutput (T423701)]], [[gerrit:1289071|ParsoidLanguageConverter: don't convert TOC if __NOCONTENTCONVERT__ (T424773)]] 2415748 wikitext text/x-wiki oju76bnx11ssk6yehtvz5zvbnmlegls 2415749 2415748 2026-05-19T13:23:46Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1072.eqiad.wmnet 2415749 wikitext text/x-wiki svaqhy997mgoyewop9cxaxjq8f8dja0 2415750 2415749 2026-05-19T13:25:28Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1289070|Forward-compatibility for serialization of ContentHolder in ParserOutput (T423701)]], [[gerrit:1289071|ParsoidLanguageConverter: don't convert TOC if __NOCONTENTCONVERT__ (T424773)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2415750 wikitext text/x-wiki 3mfnjf6bxm0vylkrtw8vojqtjbnawre 2415751 2415750 2026-05-19T13:25:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2046.codfw.wmnet 2415751 wikitext text/x-wiki jr1trltps4p08lbt0scg40ynsmba6is 2415752 2415751 2026-05-19T13:25:52Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.decommission for hosts pki1001.eqiad.wmnet 2415752 wikitext text/x-wiki 2xpodc4dti91izla6bfwgas7cb6esh6 2415753 2415752 2026-05-19T13:27:00Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2415753 wikitext text/x-wiki 2k29owa9y5henepuc9cewrtjyrl47ex 2415754 2415753 2026-05-19T13:27:02Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2046.codfw.wmnet 2415754 wikitext text/x-wiki tlx9mwnmusomd60jxoinp9dnb2m833s 2415755 2415754 2026-05-19T13:27:24Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough 2415755 wikitext text/x-wiki nf96qrp19yzauwdzqsb6uaxsbj6lpsy 2415756 2415755 2026-05-19T13:27:38Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and A:durum 2415756 wikitext text/x-wiki 7xbunqb2c15ghve3oig1uhq3znfrf2o 2415757 2415756 2026-05-19T13:28:06Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy 2415757 wikitext text/x-wiki 3y4gh9e9ptdpjuw3wjnc2zn6p8tliyc 2415758 2415757 2026-05-19T13:28:48Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox and A:magru and (A:dnsbox) 2415758 wikitext text/x-wiki bpmz9weheg9lfuuhsl7qf0d999s2o45 2415759 2415758 2026-05-19T13:28:50Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns7001.wikimedia.org 2415759 wikitext text/x-wiki sifkzas82dz85lswm8ed4kox6yl3ett 2415761 2415759 2026-05-19T13:29:28Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2071.codfw.wmnet 2415761 wikitext text/x-wiki 1ipbdd5eqp74y7uhakktex852ryii4g 2415762 2415761 2026-05-19T13:29:31Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2072.codfw.wmnet 2415762 wikitext text/x-wiki hgzvobznsz1y4gpnkbwwqxfe3r9xi2l 2415764 2415762 2026-05-19T13:30:57Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1072.eqiad.wmnet 2415764 wikitext text/x-wiki d2ggxbmon3piz939zdrxx9mza4jyjxo 2415765 2415764 2026-05-19T13:31:02Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1073.eqiad.wmnet 2415765 wikitext text/x-wiki e897z1f0f7eyinzc1lmcil6vc6bytkt 2415766 2415765 2026-05-19T13:31:15Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3008.esams.wmnet 2415766 wikitext text/x-wiki 2o4homupmtjhcfb9mtwo7f2ruuutybh 2415767 2415766 2026-05-19T13:31:21Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3008.esams.wmnet 2415767 wikitext text/x-wiki sfbs4idetu9gqx0acu6weristvk87wl 2415768 2415767 2026-05-19T13:31:28Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.netbox 2415768 wikitext text/x-wiki 47rm1snoozkjfkka97mawmq5rwxc4zo 2415769 2415768 2026-05-19T13:32:04Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2415769 wikitext text/x-wiki 7cc98umqalvuim3iajjpfwcwsbflcs0 2415770 2415769 2026-05-19T13:32:11Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2046.codfw.wmnet 2415770 wikitext text/x-wiki 73g83c53oyfquwmbp2x95q10pem0vab 2415771 2415770 2026-05-19T13:32:17Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2046.codfw.wmnet 2415771 wikitext text/x-wiki 7d0plig7a219dwj57vwjkyeqwykfses 2415772 2415771 2026-05-19T13:32:20Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2415772 wikitext text/x-wiki 0joove4b4sekmwpcgthe0skabutkewm 2415773 2415772 2026-05-19T13:33:31Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2415773 wikitext text/x-wiki olcivjg32wsb333jprbwch7bmh1rcve 2415774 2415773 2026-05-19T13:34:17Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet 2415774 wikitext text/x-wiki nwcdhbhqehcpg95y14sjfbq2oufd8l8 2415775 2415774 2026-05-19T13:35:27Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" 2415775 wikitext text/x-wiki 73kmini2a4hppwtvqszzbwrk03xd4ve 2415776 2415775 2026-05-19T13:36:05Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" 2415776 wikitext text/x-wiki qnykx4ejuvnghevn5ahhell34y831tn 2415777 2415776 2026-05-19T13:36:07Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2415777 wikitext text/x-wiki ed7trxfg43t99dowjc9xlnw95zjs37u 2415778 2415777 2026-05-19T13:36:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki1001.eqiad.wmnet 2415778 wikitext text/x-wiki h6j80j832k7jgo7knttx12i4d5g0igc 2415779 2415778 2026-05-19T13:36:20Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289070|Forward-compatibility for serialization of ContentHolder in ParserOutput (T423701)]], [[gerrit:1289071|ParsoidLanguageConverter: don't convert TOC if __NOCONTENTCONVERT__ (T424773)]] (duration: 12m 56s) 2415779 wikitext text/x-wiki jpc35j5fve7xduc8za13djr8e7u2ndy 2415780 2415779 2026-05-19T13:36:39Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1204.eqiad.wmnet with reason: host reimage 2415780 wikitext text/x-wiki 0on8btw97iq78bxw6e0ebdhrzu4nn2q 2415781 2415780 2026-05-19T13:37:41Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2415781 wikitext text/x-wiki dbviyl6f6fmbo24abpa9xo1urmkdxfq 2415782 2415781 2026-05-19T13:38:34Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2072.codfw.wmnet 2415782 wikitext text/x-wiki g2i9gwa817d6wfsw05o6s7elwa8fqql 2415783 2415782 2026-05-19T13:38:38Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2073.codfw.wmnet 2415783 wikitext text/x-wiki 7frr3yoriuy55dlqubuya39cpjjzhyk 2415784 2415783 2026-05-19T13:39:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet 2415784 wikitext text/x-wiki mqhz6ra1541d2ck1w75j85dk40tuwqu 2415785 2415784 2026-05-19T13:40:48Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1073.eqiad.wmnet 2415785 wikitext text/x-wiki 0f6d2wn0ag0tio2nvz2jk95ey401qac 2415786 2415785 2026-05-19T13:40:52Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1074.eqiad.wmnet 2415786 wikitext text/x-wiki 45jopjod3fpwklyqzzwbo75lfic6dda 2415787 2415786 2026-05-19T13:42:47Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns7001.wikimedia.org 2415787 wikitext text/x-wiki qxutdx5jq2e882amfa3d9te63rwvwaj 2415788 2415787 2026-05-19T13:45:26Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2073.codfw.wmnet 2415788 wikitext text/x-wiki leoy42d6hdryouv5umbaafs5hd3xe5p 2415789 2415788 2026-05-19T13:45:30Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2074.codfw.wmnet 2415789 wikitext text/x-wiki 6mcc10va2je100w5k16xgh935x4pyvg 2415790 2415789 2026-05-19T13:46:39Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db1204.eqiad.wmnet with reason: host reimage 2415790 wikitext text/x-wiki i6sslc9txn96jpmo83mxjbgwo2ftoof 2415791 2415790 2026-05-19T13:48:31Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1074.eqiad.wmnet 2415791 wikitext text/x-wiki chwpsmh2n4i4lrcrotccfmec9fdpf68 2415792 2415791 2026-05-19T13:48:36Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1075.eqiad.wmnet 2415792 wikitext text/x-wiki hakrg9ooziz1kd22qlfut8fmxru1b5x 2415793 2415792 2026-05-19T13:50:34Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet 2415793 wikitext text/x-wiki chj3uplxspj88majd7jnnextmfkmcbq 2415794 2415793 2026-05-19T13:50:40Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet 2415794 wikitext text/x-wiki 5ey38i6o65i7m4r52t62mvmkc2xscy2 2415795 2415794 2026-05-19T13:50:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet 2415795 wikitext text/x-wiki fnqfxb395ux5kmpcn2yuri0zwdltdr7 2415796 2415795 2026-05-19T13:51:09Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet 2415796 wikitext text/x-wiki fv6th6fpw2aja9mmbw1kehs51wt5yrm 2415797 2415796 2026-05-19T13:53:44Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet 2415797 wikitext text/x-wiki 3suil859yl5wo1dywh0vjwbo97hjgzf 2415798 2415797 2026-05-19T13:55:05Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2074.codfw.wmnet 2415798 wikitext text/x-wiki imwkp6b0oyt6ykpbjh47mzo96eh6uj4 2415799 2415798 2026-05-19T13:55:09Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2075.codfw.wmnet 2415799 wikitext text/x-wiki rjkb97bdiru9lcm87oiz5p4390mkxby 2415800 2415799 2026-05-19T13:55:55Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2415800 wikitext text/x-wiki f7e4ko4w3f55v2zjrublg7iswc7i3g5 2415801 2415800 2026-05-19T13:57:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw 2415801 wikitext text/x-wiki ngacakjksykwk9qznbxxpyfyclba64r 2415802 2415801 2026-05-19T13:57:24Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2002.codfw.wmnet 2415802 wikitext text/x-wiki jv529erj83rzwmgnpphyifkohfy75hp 2415803 2415802 2026-05-19T13:57:47Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns7002.wikimedia.org 2415803 wikitext text/x-wiki kbie3dletbrwi5tlyi2cv2e0rkl31vt 2415804 2415803 2026-05-19T13:57:51Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1075.eqiad.wmnet 2415804 wikitext text/x-wiki mbabo4u3awj48td9vohvkwyzxak691v 2415805 2415804 2026-05-19T13:57:55Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1076.eqiad.wmnet 2415805 wikitext text/x-wiki bm8znyj0z1tcbea8iunyo05o5z2tqyn 2415806 2415805 2026-05-19T13:58:00Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2002.codfw.wmnet 2415806 wikitext text/x-wiki h6t4qwn7thwu0zyyy55iif7pbwm1eo1 2415807 2415806 2026-05-19T13:59:09Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[1204-1205].eqiad.wmnet with reason: restart/reimage 2415807 wikitext text/x-wiki sp3toysq9cof0lu70h7qjn4qbn7je8t 2415808 2415807 2026-05-19T14:01:49Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2415808 wikitext text/x-wiki eqbfvcc3xi1hazl3gwpeomtj5chqqze 2415809 2415808 2026-05-19T14:01:54Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2415809 wikitext text/x-wiki 29rbor6x5xhscgmgka54xpktnwd7mcu 2415810 2415809 2026-05-19T14:02:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2002.codfw.wmnet 2415810 wikitext text/x-wiki 0x6z1dwf558wodfssqqr4md6rsw0z3n 2415811 2415810 2026-05-19T14:02:07Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2002.codfw.wmnet 2415811 wikitext text/x-wiki pw6op1ayp0joahm8w6eundcbc1lp6s7 2415812 2415811 2026-05-19T14:02:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2003.codfw.wmnet 2415812 wikitext text/x-wiki meyirp1m9qot3xhygbgwtsr9nz386do 2415813 2415812 2026-05-19T14:02:20Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2415813 wikitext text/x-wiki 5e7kgoecvjcjj12ee7xel475yh9ueaz 2415814 2415813 2026-05-19T14:02:44Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2075.codfw.wmnet 2415814 wikitext text/x-wiki 17ljueda1culvoyt5vf3g972khuwe9e 2415815 2415814 2026-05-19T14:02:46Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2003.codfw.wmnet 2415815 wikitext text/x-wiki pxchikr37oyqxlo84flzy2dustadgyy 2415816 2415815 2026-05-19T14:02:48Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2076.codfw.wmnet 2415816 wikitext text/x-wiki q26owkvz731djjkk2p7znn7ma27ub3k 2415817 2415816 2026-05-19T14:04:15Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet 2415817 wikitext text/x-wiki 3u4v1rtutqqknaqa9fpvt4kdsdgsn6l 2415818 2415817 2026-05-19T14:04:20Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet 2415818 wikitext text/x-wiki rpfeo1igexe39nh37766da88n9k3e5m 2415819 2415818 2026-05-19T14:06:39Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2003.codfw.wmnet 2415819 wikitext text/x-wiki fbz3tgo284esab8n8xg8xtv5yzlncp5 2415820 2415819 2026-05-19T14:06:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2003.codfw.wmnet 2415820 wikitext text/x-wiki 6m2b4bgsybij3c15zq8894ud9ireeti 2415821 2415820 2026-05-19T14:06:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2004.codfw.wmnet 2415821 wikitext text/x-wiki su1bee9pls7tewe6dud5lb6kpmatkzh 2415822 2415821 2026-05-19T14:07:14Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns7002.wikimedia.org 2415822 wikitext text/x-wiki cqsh1ejpo0a3udiitosbxen9jzj5ubo 2415823 2415822 2026-05-19T14:07:16Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:dnsbox and A:magru and (A:dnsbox) 2415823 wikitext text/x-wiki fdxazdszxbdgpw3mac9xvusivfcbg27 2415824 2415823 2026-05-19T14:07:18Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2004.codfw.wmnet 2415824 wikitext text/x-wiki t3fjsc03civg4vnrqi3zeh3iy36hvx4 2415825 2415824 2026-05-19T14:07:20Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1076.eqiad.wmnet 2415825 wikitext text/x-wiki lbtvugp22111740mj3xidowc0q1k684 2415826 2415825 2026-05-19T14:07:24Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1077.eqiad.wmnet 2415826 wikitext text/x-wiki 8kzywz96hc09nneia2vc22duk29jwnj 2415828 2415826 2026-05-19T14:10:24Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2076.codfw.wmnet 2415828 wikitext text/x-wiki inhcbrd34ol9vi0kqw5p1uy3uovfjf7 2415829 2415828 2026-05-19T14:10:29Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2077.codfw.wmnet 2415829 wikitext text/x-wiki 16c7kpeu4rvrl20wzfz2e333l8pxwwj 2415830 2415829 2026-05-19T14:11:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2004.codfw.wmnet 2415830 wikitext text/x-wiki 64ugatj1lul1uoa1e424n2k48b8j5w1 2415831 2415830 2026-05-19T14:11:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2004.codfw.wmnet 2415831 wikitext text/x-wiki alqrip2jh9wnr9fw8iw2bm20c9tlvjn 2415832 2415831 2026-05-19T14:11:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2005.codfw.wmnet 2415832 wikitext text/x-wiki shxdhykc0l2sc8xugszhnkitn3nkbx4 2415833 2415832 2026-05-19T14:11:57Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2005.codfw.wmnet 2415833 wikitext text/x-wiki q6zqp0i4dpn3dlt2aqa40ije4ov2khc 2415835 2415833 2026-05-19T14:14:20Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1077.eqiad.wmnet 2415835 wikitext text/x-wiki gvzfnjpcy4x7a9c09gqsjf42g03k4wp 2415836 2415835 2026-05-19T14:14:25Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1078.eqiad.wmnet 2415836 wikitext text/x-wiki qu1ci8g3yjv5llaumwiip8dlxhkm92l 2415837 2415836 2026-05-19T14:16:09Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2005.codfw.wmnet 2415837 wikitext text/x-wiki mf7dkhpluj0ftera12hf5oj1ldkkcll 2415838 2415837 2026-05-19T14:16:12Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2005.codfw.wmnet 2415838 wikitext text/x-wiki 5bvoemxmqu9qrtdyge5ko9xjrlcoluo 2415839 2415838 2026-05-19T14:16:17Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet 2415839 wikitext text/x-wiki qs32ycsv9uqaiqwd5ebzpqff2ni1fam 2415840 2415839 2026-05-19T14:16:24Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2415840 wikitext text/x-wiki lzbl12kscqkh90l5w47oodcj2gr0mza 2415841 2415840 2026-05-19T14:16:46Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2243: Upgrading db2243.codfw.wmnet 2415841 wikitext text/x-wiki q7ki2iq0kraksrce5ik7vycpkcrenc8 2415842 2415841 2026-05-19T14:17:26Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2243: Upgrading db2243.codfw.wmnet 2415842 wikitext text/x-wiki j60z9derkauyg5b8japxfkn8089ozt8 2415843 2415842 2026-05-19T14:18:03Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2077.codfw.wmnet 2415843 wikitext text/x-wiki bqkgkmbrcx1n3m8kme1tcutniqihsb5 2415844 2415843 2026-05-19T14:18:07Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2078.codfw.wmnet 2415844 wikitext text/x-wiki 844t6cmkwe00xkrt2bw18ww6rc5mbsb 2415845 2415844 2026-05-19T14:19:46Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1204.eqiad.wmnet with OS trixie 2415845 wikitext text/x-wiki n8xuiz2hke4e6k3l2ab9l4xge2ru49j 2415846 2415845 2026-05-19T14:20:54Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2243.codfw.wmnet with OS trixie 2415846 wikitext text/x-wiki 0o5mqlcrd9g5d3z8q4ywtkejfpwmpcu 2415847 2415846 2026-05-19T14:21:21Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet 2415847 wikitext text/x-wiki ka4tw9aktbs0wkw98xdl49hxu12ehrx 2415848 2415847 2026-05-19T14:22:02Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1078.eqiad.wmnet 2415848 wikitext text/x-wiki snuwu7dxh0cow0mg46yjggio6lbh8g5 2415849 2415848 2026-05-19T14:22:07Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1079.eqiad.wmnet 2415849 wikitext text/x-wiki 48wazxoikctejsi8nm5lunu10hd7mdm 2415850 2415849 2026-05-19T14:25:26Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2078.codfw.wmnet 2415850 wikitext text/x-wiki 39ujrx52zp7wl7avd5f84bt8iogxc36 2415851 2415850 2026-05-19T14:25:30Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2079.codfw.wmnet 2415851 wikitext text/x-wiki htb05np4s7h4tmlr6k8p4do2qw3twyu 2415852 2415851 2026-05-19T14:26:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet 2415852 wikitext text/x-wiki 8o8u05zjfs5n2o67hl784f2d1nejfd4 2415853 2415852 2026-05-19T14:26:44Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet 2415853 wikitext text/x-wiki m0tceb4hgva76n3a7sqkvx9vdusn66c 2415854 2415853 2026-05-19T14:26:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet 2415854 wikitext text/x-wiki tlypy8yxe7t8pfumi0kj4jjoyufn0oj 2415855 2415854 2026-05-19T14:27:26Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet 2415855 wikitext text/x-wiki ahyklphxnoc9hk3llvfq0jywt035pqh 2415856 2415855 2026-05-19T14:29:15Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1079.eqiad.wmnet 2415856 wikitext text/x-wiki qj0lu4mua2y8lhn47d66epy1z78siwj 2415857 2415856 2026-05-19T14:29:20Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1080.eqiad.wmnet 2415857 wikitext text/x-wiki 9etx1tylvsk57e3p2mgmw2sj1miupb3 2415858 2415857 2026-05-19T14:29:56Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2415858 wikitext text/x-wiki is6vxnkyncy1jagzui5b7x7ins0zj9n 2415859 2415858 2026-05-19T14:30:40Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2415859 wikitext text/x-wiki nc1nv3gt5vxqw3b0lhiz3w2cavtsrig 2415860 2415859 2026-05-19T14:30:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2415860 wikitext text/x-wiki qee9jf2ab4wa5di5i33hwcbkoknn5uu 2415861 2415860 2026-05-19T14:31:25Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2415861 wikitext text/x-wiki 6zqt28kohucsu72hna1sf5391kgifcu 2415863 2415861 2026-05-19T14:32:30Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2079.codfw.wmnet 2415863 wikitext text/x-wiki mkfxpnenxixhrxdqo9m19rhefa3v0i1 2415864 2415863 2026-05-19T14:32:35Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2080.codfw.wmnet 2415864 wikitext text/x-wiki 6i6fapfh43d6i8xrzlhsg4ze2fov8aj 2415865 2415864 2026-05-19T14:32:37Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet 2415865 wikitext text/x-wiki m1s2inidkaoxcdg89019mg41k8k7kyb 2415866 2415865 2026-05-19T14:32:38Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet 2415866 wikitext text/x-wiki 51uo02youmfhgifxvgaco1nqankm4dj 2415867 2415866 2026-05-19T14:32:44Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet 2415867 wikitext text/x-wiki nx83r35r5hcmn9z96ke4ei44xg7s1yg 2415868 2415867 2026-05-19T14:33:17Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet 2415868 wikitext text/x-wiki te9zn1pktzqy1g5slzwn90axtyejfe8 2415869 2415868 2026-05-19T14:33:33Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2002.codfw.wmnet 2415869 wikitext text/x-wiki crpydep11shx46ogkj4ybcadnoygp9d 2415870 2415869 2026-05-19T14:33:37Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1289348|hCaptcha: Enable for group1 wikis (except itwiki, metawiki) (T425354)]], [[gerrit:1289340|Drop unused $wgIPInfoIpoidUrl definition]], [[gerrit:1289339|Drop unused $wgWikimediaEventsIPoidUrl definition]], [[gerrit:1289337|Drop wgCheckUserDisplayClientHints definition]] 2415870 wikitext text/x-wiki 9x33ck3woak3c0vjx71rnwbp9qvyupi 2415871 2415870 2026-05-19T14:33:48Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host deploy2002.codfw.wmnet 2415871 wikitext text/x-wiki klt07p8y74k3tezdf518a1d5olb7mhk 2415872 2415871 2026-05-19T14:34:49Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling reboot on A:tcpproxy and A:tcpproxy 2415872 wikitext text/x-wiki 9sbez1akj7e5nb8qpl811twwb5tp9rl 2415873 2415872 2026-05-19T14:35:10Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2002.codfw.wmnet 2415873 wikitext text/x-wiki 6x7lkm8pirxgtby40mizu82atldlyuh 2415874 2415873 2026-05-19T14:35:52Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Make pc1021 master of pc1 in eqiad T418973', diff saved to https://phabricator.wikimedia.org/P92605 and previous config saved to /var/cache/conftool/dbconfig/20260519-143549-marostegui.json 2415874 wikitext text/x-wiki 67nmbk3eo3b7l2hprtltgt2llqa2kx2 2415875 2415874 2026-05-19T14:35:58Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir and A:ncredir 2415875 wikitext text/x-wiki 77fqw13q9q455fki6e16opeza500ms2 2415876 2415875 2026-05-19T14:36:35Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc1 T418973', diff saved to https://phabricator.wikimedia.org/P92606 and previous config saved to /var/cache/conftool/dbconfig/20260519-143632-marostegui.json 2415876 wikitext text/x-wiki jyl7o28q0brxjafp81v2gn2sb3jaynf 2415877 2415876 2026-05-19T14:36:37Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2243.codfw.wmnet with reason: host reimage 2415877 wikitext text/x-wiki 4z0twjf44md2r6lmlr6l147nakr2f41 2415879 2415877 2026-05-19T14:37:07Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1080.eqiad.wmnet 2415879 wikitext text/x-wiki soy52itnyxatfts128jim7j099paotb 2415880 2415879 2026-05-19T14:37:12Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1081.eqiad.wmnet 2415880 wikitext text/x-wiki efdf2nsdw51o1rj4x3v5021prxkjkvf 2415881 2415880 2026-05-19T14:37:46Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, kharlan: Backport for [[gerrit:1289348|hCaptcha: Enable for group1 wikis (except itwiki, metawiki) (T425354)]], [[gerrit:1289340|Drop unused $wgIPInfoIpoidUrl definition]], [[gerrit:1289339|Drop unused $wgWikimediaEventsIPoidUrl definition]], [[gerrit:1289337|Drop wgCheckUserDisplayClientHints definition]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). C 2415881 wikitext text/x-wiki i0a7cdrcbgpnjwvza5sbmju7o1oq0sg 2415882 2415881 2026-05-19T14:38:13Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7003.magru.wmnet} and A:liberica 2415882 wikitext text/x-wiki 41byixtx83o5bftibis39afh87fi54g 2415883 2415882 2026-05-19T14:38:38Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet 2415883 wikitext text/x-wiki ib33jamp91neqkfyqoegw3qiucffa9f 2415884 2415883 2026-05-19T14:38:40Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet 2415884 wikitext text/x-wiki j5vj54g2mlds96yypue0n7f6fajig2g 2415885 2415884 2026-05-19T14:38:45Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet 2415885 wikitext text/x-wiki ah1ttfk6xqfu14j4vd48itxdqyisx5s 2415886 2415885 2026-05-19T14:39:19Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet 2415886 wikitext text/x-wiki 6vy1nve1decw9j7z8ytdstgmmztcaos 2415887 2415886 2026-05-19T14:39:24Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2002.codfw.wmnet 2415887 wikitext text/x-wiki hwt4vfz8ictim62gg442fbwkgcvmvg8 2415888 2415887 2026-05-19T14:39:48Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2080.codfw.wmnet 2415888 wikitext text/x-wiki tbqkqbp3kqzu6jyt8qty7c49x4v4qe9 2415889 2415888 2026-05-19T14:39:52Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2081.codfw.wmnet 2415889 wikitext text/x-wiki 7y1yf5epgiyhl090uby9pv8b3mnfal3 2415890 2415889 2026-05-19T14:40:17Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy 2415890 wikitext text/x-wiki i6d3dihewb77hhbdcuopcgnjr7sb3qh 2415891 2415890 2026-05-19T14:40:18Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, kharlan: Rolling back deployment 2415891 wikitext text/x-wiki 1qn91bkxpp1h2fjnpux1mi1ngbm1ooe 2415892 2415891 2026-05-19T14:40:58Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289348|hCaptcha: Enable for group1 wikis (except itwiki, metawiki) (T425354)]], [[gerrit:1289340|Drop unused $wgIPInfoIpoidUrl definition]], [[gerrit:1289339|Drop unused $wgWikimediaEventsIPoidUrl definition]], [[gerrit:1289337|Drop wgCheckUserDisplayClientHints definition]] (duration: 07m 16s) 2415892 wikitext text/x-wiki a6xub9jhn96twklqhxmpx5sx5rmf2ls 2415893 2415892 2026-05-19T14:41:15Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2415893 wikitext text/x-wiki 8b6d5cdh9lxpnd4f6ut6rmjhx10j2lf 2415894 2415893 2026-05-19T14:41:43Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1289348|hCaptcha: Enable for group1 wikis (except itwiki, metawiki) (T425354)]], [[gerrit:1289340|Drop unused $wgIPInfoIpoidUrl definition]], [[gerrit:1289339|Drop unused $wgWikimediaEventsIPoidUrl definition]], [[gerrit:1289337|Drop wgCheckUserDisplayClientHints definition]] 2415894 wikitext text/x-wiki ja7nfbl9hsyjhhlc6szzfqdl3accs6c 2415895 2415894 2026-05-19T14:42:02Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7003.magru.wmnet} and A:liberica 2415895 wikitext text/x-wiki f664kuy6pg9w2az2g7mu5is4spomora 2415896 2415895 2026-05-19T14:42:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS trixie 2415896 wikitext text/x-wiki r38a4uujb0mwxcho11q4axf8jwk65k6 2415897 2415896 2026-05-19T14:43:10Z Stashbot 7414 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1289348|hCaptcha: Enable for group1 wikis (except itwiki, metawiki) (T425354)]], [[gerrit:1289340|Drop unused $wgIPInfoIpoidUrl definition]], [[gerrit:1289339|Drop unused $wgWikimediaEventsIPoidUrl definition]], [[gerrit:1289337|Drop wgCheckUserDisplayClientHints definition]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). C 2415897 wikitext text/x-wiki 689bjcch50egzanssz59s4zsfr04bov 2415898 2415897 2026-05-19T14:43:38Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum and A:durum 2415898 wikitext text/x-wiki 2jpuszfqkgfjug8fhrtp9pk3bznqcp6 2415899 2415898 2026-05-19T14:44:04Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2243.codfw.wmnet with reason: host reimage 2415899 wikitext text/x-wiki 4t83y2kddombls8uu4od32jus122w52 2415900 2415899 2026-05-19T14:44:38Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough 2415900 wikitext text/x-wiki 88jkm355uc8dsnat0krn1p568aduuiu 2415901 2415900 2026-05-19T14:44:40Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet 2415901 wikitext text/x-wiki pbno0sngj9z925j2mmbt7zz305si7yg 2415902 2415901 2026-05-19T14:44:42Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet 2415902 wikitext text/x-wiki pdh4i0t98clnxjh8psdktz6lgaufyji 2415903 2415902 2026-05-19T14:44:44Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw 2415903 wikitext text/x-wiki 6xtv1inftc70x4kt3h0zos8z2hvymz5 2415904 2415903 2026-05-19T14:44:46Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1081.eqiad.wmnet 2415904 wikitext text/x-wiki 0gb1z5dad1zp8ycc7rwce42xqmyatbc 2415905 2415904 2026-05-19T14:44:47Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1082.eqiad.wmnet 2415905 wikitext text/x-wiki iu57g1mre26kokgvyo38sb239plimdn 2415906 2415905 2026-05-19T14:45:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw 2415906 wikitext text/x-wiki 0h2l3m968umi794n8taj0i0q53cx1wg 2415907 2415906 2026-05-19T14:45:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet 2415907 wikitext text/x-wiki i4yjtuycygtfw4z47oue8se6a1kiac6 2415908 2415907 2026-05-19T14:45:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet 2415908 wikitext text/x-wiki 8pdwwb9hvgujz6fj4m11v25wj6n8g1n 2415909 2415908 2026-05-19T14:47:05Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2081.codfw.wmnet 2415909 wikitext text/x-wiki dpjifhn0wr21jmzm1reexvk554dyll8 2415910 2415909 2026-05-19T14:47:09Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2082.codfw.wmnet 2415910 wikitext text/x-wiki kezlduooadzygcqskx7bu1um2m8guvl 2415911 2415910 2026-05-19T14:47:11Z Stashbot 7414 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment 2415911 wikitext text/x-wiki c1tgn7n07uk2o3yoygqxpue5iwnm083 2415912 2415911 2026-05-19T14:50:01Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet 2415912 wikitext text/x-wiki 0ykfmafksxjp7p1ewu2qor3hekuyl5y 2415913 2415912 2026-05-19T14:50:03Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet 2415913 wikitext text/x-wiki p7baqd9vmiazdtygz4baryjc2bu4m2b 2415914 2415913 2026-05-19T14:50:08Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet 2415914 wikitext text/x-wiki 22te2igwbga5b0x1cd94w14exdja5pw 2415915 2415914 2026-05-19T14:50:10Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet 2415915 wikitext text/x-wiki 9gfguptzwm03bvieqwcx7sjmub6tf28 2415916 2415915 2026-05-19T14:51:06Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2415916 wikitext text/x-wiki 4osa6ep96gi3o7tl8tjqxf1isje73j2 2415917 2415916 2026-05-19T14:51:20Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289348|hCaptcha: Enable for group1 wikis (except itwiki, metawiki) (T425354)]], [[gerrit:1289340|Drop unused $wgIPInfoIpoidUrl definition]], [[gerrit:1289339|Drop unused $wgWikimediaEventsIPoidUrl definition]], [[gerrit:1289337|Drop wgCheckUserDisplayClientHints definition]] (duration: 09m 37s) 2415917 wikitext text/x-wiki kmao7ocybcjwrccrs786p1k4gp8wu3p 2415918 2415917 2026-05-19T14:51:42Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1082.eqiad.wmnet 2415918 wikitext text/x-wiki nmyki5jpkgs7x5jraxvg62sfcwl6sot 2415919 2415918 2026-05-19T14:51:47Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1083.eqiad.wmnet 2415919 wikitext text/x-wiki 79hiaw4hhq749ifcj9iq7192gon3yzh 2415920 2415919 2026-05-19T14:51:55Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2415920 wikitext text/x-wiki b78mlks9udvhc4r85149jaxnqshu6gj 2415921 2415920 2026-05-19T14:54:01Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2082.codfw.wmnet 2415921 wikitext text/x-wiki 6x6xzlhen93bj0vrt9imifmp2vf60g9 2415922 2415921 2026-05-19T14:54:05Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2083.codfw.wmnet 2415922 wikitext text/x-wiki 8y5eqpcf0wdh9721cslp0z0raclx1a3 2415923 2415922 2026-05-19T14:54:37Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2415923 wikitext text/x-wiki m89av8eveybew4ju00k58ce3gbuzo6q 2415924 2415923 2026-05-19T14:55:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet 2415924 wikitext text/x-wiki n8m5trf86zkbcqisj9zambdgmf9fjrj 2415925 2415924 2026-05-19T14:55:08Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet 2415925 wikitext text/x-wiki dyuyy8xk6fl5fguh8sr9p7r4cu1mb34 2415926 2415925 2026-05-19T14:55:10Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw 2415926 wikitext text/x-wiki k5rp1b9u8b3nyncwtm4xrzcc0cgsd8r 2415927 2415926 2026-05-19T14:55:21Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2415927 wikitext text/x-wiki n09hyu49m9jtr1djgbctnwuavj1bdi8 2415928 2415927 2026-05-19T14:56:33Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2415928 wikitext text/x-wiki 2ijb4qcfzlu0n60v9gb0a3dxps6jmne 2415929 2415928 2026-05-19T14:57:40Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2415929 wikitext text/x-wiki 6175xo7aeyrgpf3rnr42y0blr1tbe12 2415930 2415929 2026-05-19T14:57:58Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1289368|Enable hCaptcha for wikitext editor on group1 minus meta and itwiki (T425354)]] 2415930 wikitext text/x-wiki 6x9hy4701iqm08wk57rp1o5tcvzmljw 2415931 2415930 2026-05-19T14:58:59Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1083.eqiad.wmnet 2415931 wikitext text/x-wiki ge1iuby8es9oxsza0t4tgrfy2h5lx9c 2415932 2415931 2026-05-19T14:59:04Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1084.eqiad.wmnet 2415932 wikitext text/x-wiki cx352giwpuv5of2reoubdiqtjasuqj6 2415933 2415932 2026-05-19T14:59:23Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad 2415933 wikitext text/x-wiki 1knaf709pgnm0uxj123txr9js7n8hxn 2415934 2415933 2026-05-19T14:59:26Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet 2415934 wikitext text/x-wiki kmz5xbmtvulgdnngi53lwrnpij5ngy3 2415935 2415934 2026-05-19T14:59:28Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet 2415935 wikitext text/x-wiki l2hqykmgvfiivi50zjk1ftksjycp2x9 2415936 2415935 2026-05-19T15:00:00Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1289368|Enable hCaptcha for wikitext editor on group1 minus meta and itwiki (T425354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2415936 wikitext text/x-wiki b6jhenenkhu5o6o7qnsbrn7x39ohkub 2415937 2415936 2026-05-19T15:00:50Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2083.codfw.wmnet 2415937 wikitext text/x-wiki 26uvkiusjmq6ifs0ue0q570t690loc7 2415938 2415937 2026-05-19T15:00:51Z Stashbot 7414 moritzm: failover Ganeti cluster in drmrs01 to ganeti6001 2415938 wikitext text/x-wiki sj6o3pc4o7awr7a40hgl697c5xtm02u 2415939 2415938 2026-05-19T15:00:54Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2084.codfw.wmnet 2415939 wikitext text/x-wiki 21w80xe7wieliny2i0gm29uhgbkx5xq 2415940 2415939 2026-05-19T15:01:48Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2415940 wikitext text/x-wiki k90pge4a0tjwxqrbetyof8iw9m5w91w 2415941 2415940 2026-05-19T15:02:02Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2243.codfw.wmnet with OS trixie 2415941 wikitext text/x-wiki qj54zvcak06p3ksoo6huov34o5ko3j0 2415942 2415941 2026-05-19T15:02:22Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1001.eqiad.wmnet 2415942 wikitext text/x-wiki i1sz3s39j9m5eryhm6g5mttof3tobz7 2415943 2415942 2026-05-19T15:04:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2243: Migration of db2243.codfw.wmnet completed 2415943 wikitext text/x-wiki m8xpusfo166xa8koequyio1jtgg86jv 2415944 2415943 2026-05-19T15:04:18Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet 2415944 wikitext text/x-wiki m2ea21rvenketcniajlcoho96q9rki5 2415945 2415944 2026-05-19T15:04:20Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet 2415945 wikitext text/x-wiki lygujo8j7rsn2dsj0fl63t46y6v18gt 2415946 2415945 2026-05-19T15:04:24Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet 2415946 wikitext text/x-wiki f538b0mfcdohf3d9x98bmxahvvpy0w8 2415947 2415946 2026-05-19T15:04:26Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet 2415947 wikitext text/x-wiki lm5xqwud7amy8k8sn5aq13t26nzh9fm 2415948 2415947 2026-05-19T15:05:52Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1084.eqiad.wmnet 2415948 wikitext text/x-wiki 5a7rw2pamfmuucittrlklbm0ag2ft9u 2415949 2415948 2026-05-19T15:05:57Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1085.eqiad.wmnet 2415949 wikitext text/x-wiki 1bjjxcnq2p9tce7k9qnc1ckj967r9dt 2415950 2415949 2026-05-19T15:06:01Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289368|Enable hCaptcha for wikitext editor on group1 minus meta and itwiki (T425354)]] (duration: 08m 03s) 2415950 wikitext text/x-wiki kj9vgwyljjyimhbyepla7n0cyici45h 2415951 2415950 2026-05-19T15:06:15Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reboot-single for host phab1004.eqiad.wmnet 2415951 wikitext text/x-wiki 9hukjto5873d0p4pittg4pgep3ixddi 2415952 2415951 2026-05-19T15:06:19Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1001.eqiad.wmnet 2415952 wikitext text/x-wiki jupbrh11b7vv3fzspg4e01wclsdvz52 2415953 2415952 2026-05-19T15:06:24Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1002.eqiad.wmnet 2415953 wikitext text/x-wiki 1lwb3czjg94nedkevfez1wade4b9kwv 2415954 2415953 2026-05-19T15:07:53Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2084.codfw.wmnet 2415954 wikitext text/x-wiki h2edhzbda20qcli0wxylt60n13s9lmi 2415955 2415954 2026-05-19T15:07:57Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2085.codfw.wmnet 2415955 wikitext text/x-wiki o6j7bpghtrj5cdwc7tm5m7w29fa0djm 2415956 2415955 2026-05-19T15:08:45Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1002.eqiad.wmnet 2415956 wikitext text/x-wiki 453fn3yz33mool9earinf2rbj0cth2o 2415957 2415956 2026-05-19T15:08:51Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1003.eqiad.wmnet 2415957 wikitext text/x-wiki 5y4ci47i1unu6xqyekcli9own7rv79a 2415958 2415957 2026-05-19T15:09:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet 2415958 wikitext text/x-wiki 1wejihg8xsjeftx1vc28qw357xdu27h 2415959 2415958 2026-05-19T15:09:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet 2415959 wikitext text/x-wiki 21sdeuu61q0m5o1m2lblw8gt9aemws2 2415960 2415959 2026-05-19T15:09:18Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad 2415960 wikitext text/x-wiki ozu48avads38zio022hgdljnh7dge4m 2415961 2415960 2026-05-19T15:11:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1003.eqiad.wmnet 2415961 wikitext text/x-wiki 2hoig87hckfrwp5x2re86d3t3ii77wr 2415962 2415961 2026-05-19T15:12:20Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2001.codfw.wmnet 2415962 wikitext text/x-wiki aqnau9lhx8puou1k6e9vyts8oqfyv5x 2415963 2415962 2026-05-19T15:13:04Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1085.eqiad.wmnet 2415963 wikitext text/x-wiki fh80eqog85xfjkizv4mtzgd5v2bymfx 2415964 2415963 2026-05-19T15:13:09Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1086.eqiad.wmnet 2415964 wikitext text/x-wiki fjiq9gja4uwwsza9x49ipbby5jobvbq 2415965 2415964 2026-05-19T15:13:11Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetboard2003.codfw.wmnet 2415965 wikitext text/x-wiki d5l58hfx9p4lzqch2w6t25ji8vp95tt 2415966 2415965 2026-05-19T15:13:18Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet 2415966 wikitext text/x-wiki 8bi73cdwaino44zpn903rj25bu8nho4 2415967 2415966 2026-05-19T15:14:02Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@463a948]: deploy phab2002 for T426754 2415967 wikitext text/x-wiki 6cndoduocq7bi45va4bztzsn5fdtslr 2415968 2415967 2026-05-19T15:14:05Z Stashbot 7414 aokoth@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phorge Deploy 2415968 wikitext text/x-wiki oqf4st6hjfngl14028mrwvoadgp3nao 2415969 2415968 2026-05-19T15:14:18Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1245.eqiad.wmnet with reason: restart 2415969 wikitext text/x-wiki msmkpxlw1cth2k4tbgh7ychfnnc2ctq 2415970 2415969 2026-05-19T15:14:30Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab1004.eqiad.wmnet 2415970 wikitext text/x-wiki 6rnt89te4811v1cfur4wos169b3rsqb 2415971 2415970 2026-05-19T15:14:52Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@463a948]: deploy phab2002 for T426754 (duration: 00m 49s) 2415971 wikitext text/x-wiki 329kc9iulr9iw96548fepc97hbn3ekp 2415972 2415971 2026-05-19T15:14:55Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2085.codfw.wmnet 2415972 wikitext text/x-wiki dknfhzlxr2xultcbnai60763ood0hlf 2415973 2415972 2026-05-19T15:14:59Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2086.codfw.wmnet 2415973 wikitext text/x-wiki ne6jgkljzhgfk128tepwu3dw24amhy2 2415974 2415973 2026-05-19T15:16:04Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@463a948]: deploy phab1004 for T426754 2415974 wikitext text/x-wiki 7l87b01zk3qqvgdi8mkb5zx5t43imxw 2415975 2415974 2026-05-19T15:16:12Z Stashbot 7414 aokoth@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phorge Deploy 2415975 wikitext text/x-wiki 0h0b55ie26ptdjyadyy1qnq3muyfvs3 2415976 2415975 2026-05-19T15:16:17Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2001.codfw.wmnet 2415976 wikitext text/x-wiki 3mjwb7na132w7k9819ap7w9z3ti2ipd 2415977 2415976 2026-05-19T15:16:53Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@463a948]: deploy phab1004 for T426754 (duration: 00m 46s) 2415977 wikitext text/x-wiki dd7gdx2od9qo1i4djsrycbgzj5dna6p 2415978 2415977 2026-05-19T15:17:10Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet 2415978 wikitext text/x-wiki 83kt223penjrw0f7bxplvua3i78ti31 2415979 2415978 2026-05-19T15:17:12Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2003.codfw.wmnet 2415979 wikitext text/x-wiki t8u2rt0lpl1dc84kzfequ5efji94utx 2415980 2415979 2026-05-19T15:19:28Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet 2415980 wikitext text/x-wiki kmv0ez2affn35f3o0tyxe5ngvlon6bi 2415981 2415980 2026-05-19T15:19:30Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet 2415981 wikitext text/x-wiki lfxjil0maxktapszykybic3dwdhcho0 2415982 2415981 2026-05-19T15:19:38Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2003.codfw.wmnet 2415982 wikitext text/x-wiki miozaxdtko4yo5tm5zbndklxfxvk0dr 2415983 2415982 2026-05-19T15:19:41Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1086.eqiad.wmnet 2415983 wikitext text/x-wiki s19or582eevejm9xcjbi6gwnq0sqc38 2415984 2415983 2026-05-19T15:19:45Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1087.eqiad.wmnet 2415984 wikitext text/x-wiki cyi6hi85lo7berhmfg4gl30daaaoi3d 2415985 2415984 2026-05-19T15:20:27Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2002.codfw.wmnet 2415985 wikitext text/x-wiki ne5l3pdihwtc3f1fbzsyod90d7rrw9d 2415986 2415985 2026-05-19T15:20:41Z Stashbot 7414 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf11u2 into component/php83 2415986 wikitext text/x-wiki hgq0ieakv47eshyymq3061vx1y7rgd1 2415987 2415986 2026-05-19T15:21:15Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2415987 wikitext text/x-wiki 84c7moir8omp67qa1sgc6lv5tk4vg5g 2415988 2415987 2026-05-19T15:21:33Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2415988 wikitext text/x-wiki hhulz09hdy5w0kn1c9qmt1mmq469lwh 2415989 2415988 2026-05-19T15:21:45Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2415989 wikitext text/x-wiki n8n22zx5b05aivg9lwqwzhlpo4pa3s4 2415990 2415989 2026-05-19T15:21:58Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2086.codfw.wmnet 2415990 wikitext text/x-wiki b8uzk77aybda7riez2upx2kq7i5gmgi 2415991 2415990 2026-05-19T15:22:03Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2087.codfw.wmnet 2415991 wikitext text/x-wiki jbcf0430exctrsajv6djkj4g1ph8656 2415992 2415991 2026-05-19T15:22:19Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2415992 wikitext text/x-wiki gmcmawk6e8nagyzyopk87nctczog6la 2415993 2415992 2026-05-19T15:22:31Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2415993 wikitext text/x-wiki 74pgo0060jvjnk8yp9l97z6lz6naepj 2415994 2415993 2026-05-19T15:22:50Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2002.codfw.wmnet 2415994 wikitext text/x-wiki 6pu4mgyfp4rdvzwbk0lti8emwdg13xw 2415995 2415994 2026-05-19T15:23:12Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2415995 wikitext text/x-wiki hsxjnyi8kkpssp6d650w7fsabor5kr0 2415996 2415995 2026-05-19T15:23:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet 2415996 wikitext text/x-wiki cwzhcset5a2k9i3o4f88x6umb0fllbo 2415997 2415996 2026-05-19T15:23:29Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2001.codfw.wmnet 2415997 wikitext text/x-wiki agnp4z81hbizqtj0owuzweiwl5aenpu 2415998 2415997 2026-05-19T15:25:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet 2415998 wikitext text/x-wiki j3xxjig9ar4256582aey9ymgmxaqp7r 2415999 2415998 2026-05-19T15:25:22Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet 2415999 wikitext text/x-wiki cduybxzr6o6ysc5jwk1ttysvafmk1o8 2416000 2415999 2026-05-19T15:25:24Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet 2416000 wikitext text/x-wiki 7fytxhcmy03lhkyfoqw2e19fctqmrz3 2416001 2416000 2026-05-19T15:26:55Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2001.codfw.wmnet 2416001 wikitext text/x-wiki g1y7ix4tmbzszpv1z3wvaweu1wdi7rb 2416002 2416001 2026-05-19T15:26:59Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1087.eqiad.wmnet 2416002 wikitext text/x-wiki 7saz9axqflvnnibsvmjrguquva4r8fe 2416003 2416002 2026-05-19T15:27:03Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1088.eqiad.wmnet 2416003 wikitext text/x-wiki toog08it4dtmf2iqz845jg3w4xt36uu 2416004 2416003 2026-05-19T15:27:06Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2002.codfw.wmnet 2416004 wikitext text/x-wiki 7wno886y1nze58nt14git9hnyj56vom 2416005 2416004 2026-05-19T15:27:10Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{wikikube-worker[1006-1007,1015-1016,1021,1034-1057,1064-1081,1084-1087,1093-1095,1113-1165,1240-1289,1291-1327,1375-1384].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2416005 wikitext text/x-wiki ff1th6spmz7mjcvmy38w6pq1g02l1xu 2416006 2416005 2026-05-19T15:27:19Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1006-1007,1015-1016].eqiad.wmnet 2416006 wikitext text/x-wiki mdx2jxyypgvnwx2p9jsdo3kb28unjz4 2416007 2416006 2026-05-19T15:28:42Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-codfw 2416007 wikitext text/x-wiki nrezq5nszhpxh9e5822x3jx132tdozz 2416008 2416007 2026-05-19T15:28:58Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet 2416008 wikitext text/x-wiki mbfkm6tpyqwitonrl7iiz2t8ya45n31 2416009 2416008 2026-05-19T15:29:02Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2087.codfw.wmnet 2416009 wikitext text/x-wiki 2hkf6xv6an82hvg78ueo5ocj12sucgo 2416010 2416009 2026-05-19T15:29:06Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2088.codfw.wmnet 2416010 wikitext text/x-wiki cqifjq4hge8qvscrh6yxelig54qnfki 2416011 2416010 2026-05-19T15:29:37Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1006-1007,1015-1016].eqiad.wmnet 2416011 wikitext text/x-wiki fiiw0v4anixu8bg6vrti7ieckfc3ahr 2416012 2416011 2026-05-19T15:29:42Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet 2416012 wikitext text/x-wiki 15g77u9e9brtdvj9apqermuc3gb276r 2416013 2416012 2026-05-19T15:30:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet 2416013 wikitext text/x-wiki d776l5lwvjov442b7gc7y2rulzbw6sl 2416014 2416013 2026-05-19T15:30:29Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2002.codfw.wmnet 2416014 wikitext text/x-wiki mlux991po5w1qka2p2148zak1fo3zys 2416015 2416014 2026-05-19T15:31:10Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet 2416015 wikitext text/x-wiki jncyrgieafw99id94zjpzoe5gx7rqov 2416016 2416015 2026-05-19T15:33:55Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1088.eqiad.wmnet 2416016 wikitext text/x-wiki spw6hbn7cq10lcr6nfp7fuzwm8j52na 2416017 2416016 2026-05-19T15:33:59Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1089.eqiad.wmnet 2416017 wikitext text/x-wiki m4dmr1mkktqnn7h5fw0avhb283nrk9g 2416018 2416017 2026-05-19T15:35:52Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2088.codfw.wmnet 2416018 wikitext text/x-wiki 28g0a14fwqczcvvyrm5t4tnv8vwt0fq 2416019 2416018 2026-05-19T15:35:57Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet 2416019 wikitext text/x-wiki kgc8ilrj7bfj1k2s2eddh5rnd96gehy 2416020 2416019 2026-05-19T15:36:30Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1006-1007,1015-1016].eqiad.wmnet 2416020 wikitext text/x-wiki nqa42q4mv5tub1i2j4hr79lv0ypmvx8 2416021 2416020 2026-05-19T15:36:32Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1006-1007,1015-1016].eqiad.wmnet 2416021 wikitext text/x-wiki pbwd2v1e1kmp4ufmarym0uj8hqbe0xs 2416022 2416021 2026-05-19T15:36:43Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1021,1034-1036].eqiad.wmnet 2416022 wikitext text/x-wiki 4ihuo0k57nn3fy4r2cciwvzwd9v2eli 2416023 2416022 2026-05-19T15:37:22Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet 2416023 wikitext text/x-wiki 7y93r41a6kteuatp9syy3bn5phyh7fb 2416024 2416023 2026-05-19T15:39:05Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1021,1034-1036].eqiad.wmnet 2416024 wikitext text/x-wiki qsqyj8znlvlum10sichuvi6tv7jf1bb 2416025 2416024 2026-05-19T15:39:59Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling reboot on A:tcpproxy and A:tcpproxy 2416025 wikitext text/x-wiki sbci5dku1ifajt442xh5s5a9vsta7d7 2416026 2416025 2026-05-19T15:41:06Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1089.eqiad.wmnet 2416026 wikitext text/x-wiki eo70zwqmsvxf2k1x47maqq80coosle6 2416027 2416026 2026-05-19T15:41:11Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1090.eqiad.wmnet 2416027 wikitext text/x-wiki e808bpi01ueqx6rkerlhv36mu3cb9ev 2416028 2416027 2026-05-19T15:43:02Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2089.codfw.wmnet 2416028 wikitext text/x-wiki ixg3h839wzfu98nrme8kh96lgxi2hpz 2416029 2416028 2026-05-19T15:43:06Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2090.codfw.wmnet 2416029 wikitext text/x-wiki tccqw3nnh7jf2oo34tw6d3phhvvd7p2 2416030 2416029 2026-05-19T15:43:27Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet 2416030 wikitext text/x-wiki aw55pvqbpf2tfud1nvs3aqaje87mfl1 2416031 2416030 2026-05-19T15:43:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet 2416031 wikitext text/x-wiki 9bzjneavsz4q3aacsmns5a78oehku66 2416032 2416031 2026-05-19T15:45:44Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1021,1034-1036].eqiad.wmnet 2416032 wikitext text/x-wiki d8nkxjl4aalu637kk1j78z3oqeemnn1 2416033 2416032 2026-05-19T15:45:46Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1021,1034-1036].eqiad.wmnet 2416033 wikitext text/x-wiki lennshrt1okme3uf5nbnozr7kxrwtli 2416034 2416033 2026-05-19T15:45:56Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1037-1040].eqiad.wmnet 2416034 wikitext text/x-wiki 60ywjduib14eamnizmv8otbryn8g1vi 2416035 2416034 2026-05-19T15:48:12Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1090.eqiad.wmnet 2416035 wikitext text/x-wiki slel4mtzsd392zhw5ewl9axmcnqvmvl 2416036 2416035 2026-05-19T15:48:17Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1091.eqiad.wmnet 2416036 wikitext text/x-wiki t7299zac78ahn27iwzd3zukl9ffqelm 2416037 2416036 2026-05-19T15:49:27Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir and A:ncredir 2416037 wikitext text/x-wiki 4c5xu9i27f24nl6u2fk7ava1ju9u043 2416038 2416037 2026-05-19T15:49:37Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2243: Migration of db2243.codfw.wmnet completed 2416038 wikitext text/x-wiki 15gmczfknsmatwtjlfc0v9jq62lp2yl 2416039 2416038 2026-05-19T15:49:39Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2416039 wikitext text/x-wiki mjulzkgbcvx03mjqge0pdgcwmaj9lsp 2416040 2416039 2026-05-19T15:49:48Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2090.codfw.wmnet 2416040 wikitext text/x-wiki cl2la0yzq4abbad58wdkxjsgd3qh0rz 2416041 2416040 2026-05-19T15:49:52Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2091.codfw.wmnet 2416041 wikitext text/x-wiki ryh2o99w17z8at0ha9x2h8p92k9rj14 2416042 2416041 2026-05-19T15:51:15Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet 2416042 wikitext text/x-wiki 1u1k8i1a3g4kepxu42h8pmsr3pl48zw 2416043 2416042 2026-05-19T15:52:37Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1037-1040].eqiad.wmnet 2416043 wikitext text/x-wiki 308zy8dbtbrnrohmxkisip4a7n3zi07 2416044 2416043 2026-05-19T15:53:10Z Stashbot 7414 moritzm: temporarily drop ganeti2029 from the codfw cluster T426199 2416044 wikitext text/x-wiki sjic1qq5r86sw79a6lruypcn2tgh8zx 2416045 2416044 2026-05-19T15:55:19Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1091.eqiad.wmnet 2416045 wikitext text/x-wiki j2s1cq2xog5t2sydcgrup8icjrb3u43 2416046 2416045 2026-05-19T15:55:54Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2091.codfw.wmnet 2416046 wikitext text/x-wiki gjd8ty1ql5bkwlnxe5u1fdfzcud031j 2416047 2416046 2026-05-19T15:59:34Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1037-1040].eqiad.wmnet 2416047 wikitext text/x-wiki bdt2q8vxy0udkl7gj2dyhut82n0s1mr 2416048 2416047 2026-05-19T15:59:37Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1037-1040].eqiad.wmnet 2416048 wikitext text/x-wiki nltwx6lthlvhk2e4d6cnn0w70k9iow9 2416049 2416048 2026-05-19T15:59:47Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1041-1044].eqiad.wmnet 2416049 wikitext text/x-wiki t3f7894yl2lduf0742wihaw77dw1xbh 2416050 2416049 2026-05-19T16:02:11Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1041-1044].eqiad.wmnet 2416050 wikitext text/x-wiki tw3x8emapzpv8xu91b4jhkf0hg3t17o 2416051 2416050 2026-05-19T16:02:44Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS trixie 2416051 wikitext text/x-wiki s9ybh8mitv3ajxwgrqh2l2s6v08tc63 2416052 2416051 2026-05-19T16:07:14Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs-test1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2416052 wikitext text/x-wiki 5iv4022ryeul1bp1v2c83ybp3fpxcsb 2416053 2416052 2026-05-19T16:09:16Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1041-1044].eqiad.wmnet 2416053 wikitext text/x-wiki bnxkqyplhon2dbvj5vt4w67nh5cykoo 2416054 2416053 2026-05-19T16:09:19Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1041-1044].eqiad.wmnet 2416054 wikitext text/x-wiki 617comoqup27jkf40iet5y07esx41tb 2416055 2416054 2026-05-19T16:09:29Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1045-1048].eqiad.wmnet 2416055 wikitext text/x-wiki i6st8g72rshmmfv8c2nfk0gvdllts7d 2416056 2416055 2026-05-19T16:11:41Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1045-1048].eqiad.wmnet 2416056 wikitext text/x-wiki fqi3t2d4vlhvek234hj1135f6dwtzxo 2416057 2416056 2026-05-19T16:13:46Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs-test1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2416057 wikitext text/x-wiki hrhxzym70r7z842q8sip0365bt26v7a 2416058 2416057 2026-05-19T16:21:42Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1045-1048].eqiad.wmnet 2416058 wikitext text/x-wiki 4m1lm00dixeq0lokeicci8l5gsa1u9l 2416059 2416058 2026-05-19T16:21:45Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1045-1048].eqiad.wmnet 2416059 wikitext text/x-wiki sy4uw1573czwraxphhx2zjp79reee6y 2416060 2416059 2026-05-19T16:21:54Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1049-1052].eqiad.wmnet 2416060 wikitext text/x-wiki izjmkschhrxk4sdepjblzmqkzyyzho3 2416061 2416060 2026-05-19T16:24:12Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1049-1052].eqiad.wmnet 2416061 wikitext text/x-wiki 6ed7bi5of1a98268iplr1owb67nt7fj 2416062 2416061 2026-05-19T16:28:10Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbprov[1004-1007].eqiad.wmnet with reason: restart 2416062 wikitext text/x-wiki dw0oa62fa77w04rr1oauwn053o9s2r7 2416063 2416062 2026-05-19T16:32:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1049-1052].eqiad.wmnet 2416063 wikitext text/x-wiki t5l6sbouvvhf81cugd71smxbh7rfe1i 2416064 2416063 2026-05-19T16:32:22Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1049-1052].eqiad.wmnet 2416064 wikitext text/x-wiki majw3dlal3z8o141r3o67mj5ukbpzdi 2416065 2416064 2026-05-19T16:32:36Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1053-1056].eqiad.wmnet 2416065 wikitext text/x-wiki 5du89vonbwdfjjikkxgh26gesd9vvun 2416066 2416065 2026-05-19T16:34:48Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1053-1056].eqiad.wmnet 2416066 wikitext text/x-wiki czxpqxo4tv84qc3nbpxxg9aso68c0lp 2416067 2416066 2026-05-19T16:43:08Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1053-1056].eqiad.wmnet 2416067 wikitext text/x-wiki 8s3xkbyqr7ot2cu76a4uhoxnbbg8khd 2416068 2416067 2026-05-19T16:43:10Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1053-1056].eqiad.wmnet 2416068 wikitext text/x-wiki kbliki75jtjiityzj1sce9274rugort 2416069 2416068 2026-05-19T16:43:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1057,1064-1066].eqiad.wmnet 2416069 wikitext text/x-wiki ogbc6x12jfdeag7c03wcy0glqtn0k31 2416070 2416069 2026-05-19T16:46:09Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1057,1064-1066].eqiad.wmnet 2416070 wikitext text/x-wiki 08q5erqf5ic87r1e1xkadsk7ti86qpn 2416071 2416070 2026-05-19T16:53:14Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1057,1064-1066].eqiad.wmnet 2416071 wikitext text/x-wiki 7jb8agt8hzcvgk75x1nh53qtfaywn9c 2416072 2416071 2026-05-19T16:53:16Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1057,1064-1066].eqiad.wmnet 2416072 wikitext text/x-wiki b92tmmt5sk8mblenvmai63r295j8dk3 2416073 2416072 2026-05-19T16:53:26Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1067-1070].eqiad.wmnet 2416073 wikitext text/x-wiki qcehpyimkeip33iqknnvnmkgxg8cxws 2416074 2416073 2026-05-19T16:54:25Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backupmon1001.eqiad.wmnet with reason: restart 2416074 wikitext text/x-wiki nvv6s7j863ex1bulipvohgzdgamhu4q 2416075 2416074 2026-05-19T16:55:43Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1067-1070].eqiad.wmnet 2416075 wikitext text/x-wiki k2lrto80uy06t161nafqlx2jpdp30zy 2416076 2416075 2026-05-19T17:02:17Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbprov[2004-2007].codfw.wmnet with reason: restart 2416076 wikitext text/x-wiki ao5vg64d2zlgt52lyq38f5e222hiai7 2416077 2416076 2026-05-19T17:02:57Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1067-1070].eqiad.wmnet 2416077 wikitext text/x-wiki i2mcj5cwyvadenh45j0tuvu804qtwaj 2416078 2416077 2026-05-19T17:02:59Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1067-1070].eqiad.wmnet 2416078 wikitext text/x-wiki dlkn0dhl00t2sgyd7xl1uvyzr3ria1t 2416079 2416078 2026-05-19T17:03:09Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1071-1074].eqiad.wmnet 2416079 wikitext text/x-wiki izd5zdc3biv8dwtesgrf66721q1uxxn 2416080 2416079 2026-05-19T17:03:15Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet 2416080 wikitext text/x-wiki jjyulvxc7j84aq7gll0v3ky7ihlqx11 2416081 2416080 2026-05-19T17:05:21Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1071-1074].eqiad.wmnet 2416081 wikitext text/x-wiki 7yuqjomcvgx5mgbfuqr2g5dnt0u77gv 2416082 2416081 2026-05-19T17:12:31Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1071-1074].eqiad.wmnet 2416082 wikitext text/x-wiki idetrsodghrcz1yghhd6j2wamq364rh 2416083 2416082 2026-05-19T17:12:33Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1071-1074].eqiad.wmnet 2416083 wikitext text/x-wiki t36cjgvwtoyu10wf7b3ekivxulzszqu 2416084 2416083 2026-05-19T17:12:47Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1075-1078].eqiad.wmnet 2416084 wikitext text/x-wiki cewyelxgjobgzu7875b4w0h4cepko9m 2416085 2416084 2026-05-19T17:13:04Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet 2416085 wikitext text/x-wiki 084sdyarsp5u8v3jz9fgxtabh9bx7qq 2416086 2416085 2026-05-19T17:14:59Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1075-1078].eqiad.wmnet 2416086 wikitext text/x-wiki ej9ogz1cc679nzmru38nbakf4xvo3x3 2416087 2416086 2026-05-19T17:22:13Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts2002.codfw.wmnet 2416087 wikitext text/x-wiki kkdt19jdsv1al7w3hens5pcue5yy4d9 2416088 2416087 2026-05-19T17:23:27Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1075-1078].eqiad.wmnet 2416088 wikitext text/x-wiki ngfmwdnx8lrtvtn6hpfyely0u246h8i 2416089 2416088 2026-05-19T17:23:29Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1075-1078].eqiad.wmnet 2416089 wikitext text/x-wiki qdwjorc6wu0r8arvgfok5z9bjzhstgr 2416090 2416089 2026-05-19T17:23:40Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1079-1081,1084].eqiad.wmnet 2416090 wikitext text/x-wiki 65pph5u72riudrgwdzw14sh576fgmjd 2416091 2416090 2026-05-19T17:26:03Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1079-1081,1084].eqiad.wmnet 2416091 wikitext text/x-wiki tsyjgya6j1mdudj7c6aplzfoyc8sahv 2416092 2416091 2026-05-19T17:26:57Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2416092 wikitext text/x-wiki c2md4hqoln1ermm6dr2s6xw8sy00ufb 2416094 2416092 2026-05-19T17:27:07Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2416094 wikitext text/x-wiki ommjsljetlewixzq40yqnywoqt7cagd 2416095 2416094 2026-05-19T17:27:19Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2416095 wikitext text/x-wiki nk0s8ulcggk0hoakln648v1jzhzvyil 2416096 2416095 2026-05-19T17:27:52Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2416096 wikitext text/x-wiki erg51ev6jj61h42l8csewlufjcjvtbm 2416097 2416096 2026-05-19T17:28:00Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2416097 wikitext text/x-wiki 3by8d7p73zfonzj19sgs3ocyhn4rzni 2416098 2416097 2026-05-19T17:28:35Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2416098 wikitext text/x-wiki 8b7zavtlwwvdgrinsffk4m4ouxv5i53 2416099 2416098 2026-05-19T17:28:52Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts2002.codfw.wmnet 2416099 wikitext text/x-wiki k923xubtq3z9lijsovpf4b4wgjy1f49 2416100 2416099 2026-05-19T17:30:04Z Stashbot 7414 tchin@deploy1003: Started deploy [analytics/refinery@eeef7f3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@eeef7f3d] 2416100 wikitext text/x-wiki r7vxhwqs3qw7lcccg3iu8thslsrk2sv 2416101 2416100 2026-05-19T17:30:51Z Stashbot 7414 tchin@deploy1003: Finished deploy [analytics/refinery@eeef7f3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@eeef7f3d] (duration: 00m 46s) 2416101 wikitext text/x-wiki t3yxl3db7vz9qp14px8hqq43zuhxahj 2416102 2416101 2026-05-19T17:33:57Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1079-1081,1084].eqiad.wmnet 2416102 wikitext text/x-wiki db1psajv2szn3cghi51jo43dfne2sda 2416103 2416102 2026-05-19T17:33:59Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1079-1081,1084].eqiad.wmnet 2416103 wikitext text/x-wiki p6k13enq44sc36tj5xy7h5zqujq7phk 2416104 2416103 2026-05-19T17:34:09Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1085-1087,1093].eqiad.wmnet 2416104 wikitext text/x-wiki c4tmjrxqdd3ketadlkt8kmcf9cwbsxq 2416105 2416104 2026-05-19T17:35:05Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 14 hosts with reason: restart 2416105 wikitext text/x-wiki sl7x6lza1up2gybg1qxpn9ysj4ocj8z 2416106 2416105 2026-05-19T17:36:29Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1085-1087,1093].eqiad.wmnet 2416106 wikitext text/x-wiki fbin5up727djxmvl01u2ac57on1g4sa 2416107 2416106 2026-05-19T17:43:44Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1085-1087,1093].eqiad.wmnet 2416107 wikitext text/x-wiki fxjkf5x09xdqxe77hdbwyb7ez9oapnr 2416108 2416107 2026-05-19T17:43:46Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1085-1087,1093].eqiad.wmnet 2416108 wikitext text/x-wiki 7l9dz433k44ah1cyu89m1a4r1a4tfjn 2416109 2416108 2026-05-19T17:43:56Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1094-1095,1113-1114].eqiad.wmnet 2416109 wikitext text/x-wiki sxzz7xvoexzmvbzdd689rlxbydkdmte 2416110 2416109 2026-05-19T17:46:13Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1094-1095,1113-1114].eqiad.wmnet 2416110 wikitext text/x-wiki aoj60obz48odu3guuldvbtl18grhdmk 2416112 2416110 2026-05-19T17:51:03Z Stashbot 7414 joal@deploy1003: Started deploy [analytics/refinery@eeef7f3] (hadoop-test): Hotfix Hadoop-test [analytics/refinery@eeef7f3d] 2416112 wikitext text/x-wiki 84wobt0va43s1rss4xl2937pjklxur2 2416114 2416112 2026-05-19T17:53:17Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1094-1095,1113-1114].eqiad.wmnet 2416114 wikitext text/x-wiki pryqez2v31rjb7twxcs4ar8hao0ooki 2416115 2416114 2026-05-19T17:53:20Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1094-1095,1113-1114].eqiad.wmnet 2416115 wikitext text/x-wiki 754uqowvsn3ss1lvg0l5667ylopaces 2416116 2416115 2026-05-19T17:53:29Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1115-1118].eqiad.wmnet 2416116 wikitext text/x-wiki 0w3n56ij8mwroa11ilonuj4zrx6ktcx 2416117 2416116 2026-05-19T17:56:17Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1115-1118].eqiad.wmnet 2416117 wikitext text/x-wiki oym01fik2p2nvtee9oaz3dmf4kgwuu4 2416122 2416117 2026-05-19T17:59:38Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host ncmonitor1001.eqiad.wmnet 2416122 wikitext text/x-wiki jpb32u0w5f4cbl7zb876h9mux33jfgx 2416123 2416122 2026-05-19T18:03:32Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ncmonitor1001.eqiad.wmnet 2416123 wikitext text/x-wiki jmjw3zx9vu1xfrdaqrtf7baaktz9up2 2416124 2416123 2026-05-19T18:05:35Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1115-1118].eqiad.wmnet 2416124 wikitext text/x-wiki 4ejmgkhfbj1raqjpr2oj3d728l8uhca 2416125 2416124 2026-05-19T18:05:37Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1115-1118].eqiad.wmnet 2416125 wikitext text/x-wiki bgbd3w0alg26ad8ap9eo1o1kmhpybtu 2416126 2416125 2026-05-19T18:05:52Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1119-1122].eqiad.wmnet 2416126 wikitext text/x-wiki m3o1rajxm46a1avej92yvz5tbt844ga 2416127 2416126 2026-05-19T18:07:36Z Stashbot 7414 cjd91: import gdnsd_3.99.0-alpha3~deb13u1 into trixie-wikimedia-T401832 2416127 wikitext text/x-wiki fkvgkm84xdy53o5rucglvxalrx5unih 2416129 2416127 2026-05-19T18:08:11Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1119-1122].eqiad.wmnet 2416129 wikitext text/x-wiki 4jyzkw0xrgu72fpqd2acfbmyn2e9z6l 2416130 2416129 2026-05-19T18:12:25Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp7001.magru.wmnet,cp7009.magru.wmnet} and A:cp 2416130 wikitext text/x-wiki a7gebdy7bogo2pta3p103huj7bcozoc 2416131 2416130 2026-05-19T18:17:42Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1119-1122].eqiad.wmnet 2416131 wikitext text/x-wiki l2jw518ks6xiabp1tzkv8ctqca20aut 2416132 2416131 2026-05-19T18:17:44Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1119-1122].eqiad.wmnet 2416132 wikitext text/x-wiki fdxzegxyztdh07rmva4vvsyqjfmp5v6 2416133 2416132 2026-05-19T18:17:54Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1123-1126].eqiad.wmnet 2416133 wikitext text/x-wiki i09teyurz87f7gw04xhype3qfmfuvsn 2416134 2416133 2026-05-19T18:20:10Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1123-1126].eqiad.wmnet 2416134 wikitext text/x-wiki p483bwasei68dy08amx8z6qknkblfb4 2416135 2416134 2026-05-19T18:24:01Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp7001.magru.wmnet 2416135 wikitext text/x-wiki 5t51dx8l9tjx428pl79thfw29eo1r51 2416136 2416135 2026-05-19T18:26:50Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1123-1126].eqiad.wmnet 2416136 wikitext text/x-wiki 2nteh5mvxyrmww3aytyt1i7mjb1rx4p 2416137 2416136 2026-05-19T18:26:52Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1123-1126].eqiad.wmnet 2416137 wikitext text/x-wiki m0tk2jvf9pjvia2j4i30rt4fnv5zpkp 2416138 2416137 2026-05-19T18:27:01Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1127-1130].eqiad.wmnet 2416138 wikitext text/x-wiki rhd28x96kz5kb9f8yi7lb65kht8rbxg 2416139 2416138 2026-05-19T18:33:42Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1127-1130].eqiad.wmnet 2416139 wikitext text/x-wiki nvplexzd9afs27acdwz2ysp0funx7d9 2416140 2416139 2026-05-19T18:40:25Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1127-1130].eqiad.wmnet 2416140 wikitext text/x-wiki fsb60ey8fa3ym91u1dn3zudksgxerrm 2416141 2416140 2026-05-19T18:40:27Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1127-1130].eqiad.wmnet 2416141 wikitext text/x-wiki k9w9mzjkldvy2unw76n3plixeyipez3 2416142 2416141 2026-05-19T18:40:37Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1131-1134].eqiad.wmnet 2416142 wikitext text/x-wiki 0tqpxe6w53l7zqj11rn7key8223u8c7 2416144 2416142 2026-05-19T18:43:30Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1131-1134].eqiad.wmnet 2416144 wikitext text/x-wiki ge9z0zffekh5j8az3rpuub61qwli2s2 2416145 2416144 2026-05-19T18:45:06Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir-magru and A:ncredir 2416145 wikitext text/x-wiki 0ehypshv2x8wfoh1619fg375swzqr5k 2416147 2416145 2026-05-19T18:50:25Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1131-1134].eqiad.wmnet 2416147 wikitext text/x-wiki a5y7mkitp7lchnw0385kgl7o6gox0fz 2416148 2416147 2026-05-19T18:50:27Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1131-1134].eqiad.wmnet 2416148 wikitext text/x-wiki qwk2ip4dgdjr3ygodbwaleznf8i7q8l 2416149 2416148 2026-05-19T18:50:37Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1135-1138].eqiad.wmnet 2416149 wikitext text/x-wiki s2pw7ug4hw1foxc84r57t0b6aspyc5t 2416150 2416149 2026-05-19T18:50:42Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-codfw 2416150 wikitext text/x-wiki herlwvnzomogmk3t0745mtvkkcdqcpk 2416151 2416150 2026-05-19T18:52:51Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1135-1138].eqiad.wmnet 2416151 wikitext text/x-wiki jqq14gii7o65rnhkxzjbn5cxs4g4860 2416152 2416151 2026-05-19T18:55:40Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir-magru and A:ncredir 2416152 wikitext text/x-wiki 89oayy8jvgrftn9bbl83wp16nuo7z3b 2416153 2416152 2026-05-19T19:02:23Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1135-1138].eqiad.wmnet 2416153 wikitext text/x-wiki 0rfbvmaf0dey5ytt4ru81zvj9p9smx3 2416154 2416153 2026-05-19T19:02:25Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1135-1138].eqiad.wmnet 2416154 wikitext text/x-wiki pvy9pksyi8hp4wgkb1q8l3t9qql38lk 2416155 2416154 2026-05-19T19:02:34Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1139-1142].eqiad.wmnet 2416155 wikitext text/x-wiki k149kchuabf3lf4wvad6h5oe4dpppo1 2416156 2416155 2026-05-19T19:02:49Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir-magru and A:ncredir 2416156 wikitext text/x-wiki 3o38w507e5159pphspj747z2qt7zksx 2416157 2416156 2026-05-19T19:02:55Z Stashbot 7414 brett@cumin2002: END (ERROR) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=97) rolling reboot on A:ncredir-magru and A:ncredir 2416157 wikitext text/x-wiki nq3crr8x0ze3ka17qh7u6wx23gsuy0q 2416158 2416157 2026-05-19T19:03:38Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir and not A:ncredir-magru and A:ncredir 2416158 wikitext text/x-wiki pwc3crvwipegpbnogyagvpnp29k0prw 2416159 2416158 2026-05-19T19:04:46Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1139-1142].eqiad.wmnet 2416159 wikitext text/x-wiki fzsqz94jhnpc3qvzgz88hgj8csc41xz 2416160 2416159 2026-05-19T19:06:04Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp7009.magru.wmnet 2416160 wikitext text/x-wiki aiqhysw3tx1124ujq1a511gj4oz8siw 2416161 2416160 2026-05-19T19:06:06Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp7001.magru.wmnet,cp7009.magru.wmnet} and A:cp 2416161 wikitext text/x-wiki jmfvm50l2iayy99w8il0te4pmsibq34 2416162 2416161 2026-05-19T19:11:26Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1139-1142].eqiad.wmnet 2416162 wikitext text/x-wiki 0hsp09i81rkxa4vr7il0z99wr6ga37t 2416163 2416162 2026-05-19T19:11:28Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1139-1142].eqiad.wmnet 2416163 wikitext text/x-wiki p47xv6j6h6usg6blg4it3q3ga86x7kl 2416164 2416163 2026-05-19T19:11:38Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1143-1146].eqiad.wmnet 2416164 wikitext text/x-wiki cilwsow3ti25g9qhbuk6o0loln20x15 2416165 2416164 2026-05-19T19:14:31Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1143-1146].eqiad.wmnet 2416165 wikitext text/x-wiki 2xcx7uyrk5avqklmh6d3uux5soumns9 2416169 2416165 2026-05-19T19:21:07Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1143-1146].eqiad.wmnet 2416169 wikitext text/x-wiki leh4ovdajwvp0di6fekq1iwzkzb15xo 2416170 2416169 2026-05-19T19:21:09Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1143-1146].eqiad.wmnet 2416170 wikitext text/x-wiki bnikpho3j5o6nrmgqzy2xsueq2242oj 2416171 2416170 2026-05-19T19:21:10Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp7010.magru.wmnet} and A:cp 2416171 wikitext text/x-wiki fkn6sdpfd1a44xcs0lvwxkz6w2ozdkt 2416172 2416171 2026-05-19T19:21:16Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp7002.magru.wmnet} and A:cp 2416172 wikitext text/x-wiki 8bcfxbktkren60aumhtuwvc1555txik 2416173 2416172 2026-05-19T19:21:18Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1147-1150].eqiad.wmnet 2416173 wikitext text/x-wiki dblhjsco9qu6izww2qn9tldpit700xx 2416174 2416173 2026-05-19T19:23:33Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1147-1150].eqiad.wmnet 2416174 wikitext text/x-wiki safrq709qyihr7fioodqbkuid45f9cp 2416175 2416174 2026-05-19T19:25:00Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on gitlab1003.wikimedia.org with reason: T426563 2416175 wikitext text/x-wiki 34k3orkcro5585rrc2tq4ry06he92rx 2416176 2416175 2026-05-19T19:25:14Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org 2416176 wikitext text/x-wiki 6qmwy3bczh16aywdcj1i3o6ffggm7wv 2416177 2416176 2026-05-19T19:25:56Z Stashbot 7414 mutante: rebooting gitlab-replica-a.wikimedia.org 2416177 wikitext text/x-wiki iytsiku5lll2rrsnq644ofb8lnau06t 2416178 2416177 2026-05-19T19:26:38Z Stashbot 7414 tchin@deploy1003: Started deploy [analytics/refinery@eeef7f3] (hadoop-test): ReHotfix Hadoop-test analytics/refinery@eeef7f3d] 2416178 wikitext text/x-wiki bsh9mg4w4ke2zel7m5lhry76r9z97jo 2416179 2416178 2026-05-19T19:28:39Z Stashbot 7414 tchin@deploy1003: Finished deploy [analytics/refinery@eeef7f3] (hadoop-test): ReHotfix Hadoop-test analytics/refinery@eeef7f3d] (duration: 02m 00s) 2416179 wikitext text/x-wiki rgmctp9fiaglmx5cid4p71qke8f1cfk 2416180 2416179 2026-05-19T19:30:15Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1147-1150].eqiad.wmnet 2416180 wikitext text/x-wiki d0sgzxwkoknlnks7v77tjii2mtaap5y 2416181 2416180 2026-05-19T19:30:17Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1147-1150].eqiad.wmnet 2416181 wikitext text/x-wiki 45m3wgs573ag1bbgp7obh4f2e33a4o4 2416182 2416181 2026-05-19T19:30:27Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1151-1154].eqiad.wmnet 2416182 wikitext text/x-wiki n9pi2v1spy1lfmuocttzj1ymf8z4eju 2416183 2416182 2026-05-19T19:32:43Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1151-1154].eqiad.wmnet 2416183 wikitext text/x-wiki lmh32y9y9u5pqe926un5di3u9muxwik 2416184 2416183 2026-05-19T19:32:49Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp7002.magru.wmnet 2416184 wikitext text/x-wiki 4hrym9halbhz1xjqs695ku2418cjzqs 2416185 2416184 2026-05-19T19:32:50Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp7002.magru.wmnet} and A:cp 2416185 wikitext text/x-wiki hcu4ngl9gpz9or89b6v47ousn328130 2416186 2416185 2026-05-19T19:32:55Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp7010.magru.wmnet 2416186 wikitext text/x-wiki ikdi0ugaixzv5vh3r8bfsb35dybov8a 2416187 2416186 2026-05-19T19:32:57Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp7010.magru.wmnet} and A:cp 2416187 wikitext text/x-wiki 84kjmvitnfe3ms4utbbm6ejqq571bvu 2416188 2416187 2026-05-19T19:33:02Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org 2416188 wikitext text/x-wiki rtm56j0lflttpypcpg30rsfkam653ov 2416189 2416188 2026-05-19T19:34:33Z Stashbot 7414 tchin@deploy1003: Started deploy [analytics/refinery@eeef7f3]: Redeploy v0.3.14 [analytics/refinery@eeef7f3d] 2416189 wikitext text/x-wiki kzecutog7vszjj2v29y2id0u77xs7om 2416190 2416189 2026-05-19T19:39:08Z Stashbot 7414 tchin@deploy1003: Finished deploy [analytics/refinery@eeef7f3]: Redeploy v0.3.14 [analytics/refinery@eeef7f3d] (duration: 04m 35s) 2416190 wikitext text/x-wiki 17zozzpq8lxzgexoolroaxowya1rvs2 2416191 2416190 2026-05-19T19:39:49Z Stashbot 7414 tchin@deploy1003: Started deploy [analytics/refinery@eeef7f3] (thin): Redeploy v0.3.14 THIN [analytics/refinery@eeef7f3d] 2416191 wikitext text/x-wiki 7ya6i4osvanjdwja6u35y7yd846rpag 2416192 2416191 2026-05-19T19:40:46Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host pybal-test2003.codfw.wmnet 2416192 wikitext text/x-wiki hbz4wa7dpbwyqdi73t32xxf87ma7jyl 2416193 2416192 2026-05-19T19:41:23Z Stashbot 7414 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1289408|BugFix: Emit page_change at version 1.6.0 to pick up user wiki_id (T426198)]] 2416193 wikitext text/x-wiki 4jhypl43rjkbu8jzgnqsz4f593ycwdb 2416194 2416193 2026-05-19T19:41:36Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp701[1-2].magru.wmnet} and A:cp 2416194 wikitext text/x-wiki 2vbasmm64ss7byf5tzw1viqpntr20x2 2416195 2416194 2026-05-19T19:41:57Z Stashbot 7414 tchin@deploy1003: Finished deploy [analytics/refinery@eeef7f3] (thin): Redeploy v0.3.14 THIN [analytics/refinery@eeef7f3d] (duration: 02m 07s) 2416195 wikitext text/x-wiki fbiephfkzqs6riuxamtt9n766rlmvgj 2416196 2416195 2026-05-19T19:42:10Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1151-1154].eqiad.wmnet 2416196 wikitext text/x-wiki kvnl1q6rjnuse1qsiz2vvvgjy0978ga 2416197 2416196 2026-05-19T19:42:12Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1151-1154].eqiad.wmnet 2416197 wikitext text/x-wiki 56dd3phndzb4t3ta08h25kzbmc4qu0h 2416198 2416197 2026-05-19T19:42:18Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp700[3-4].magru.wmnet} and A:cp 2416198 wikitext text/x-wiki kbjbz0ueuegckswg7cj8ut8axv1gmz1 2416199 2416198 2026-05-19T19:42:22Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1155-1158].eqiad.wmnet 2416199 wikitext text/x-wiki 5w4ltyy2ot1jl5phupdcakwwocwd8i1 2416200 2416199 2026-05-19T19:44:00Z Stashbot 7414 otto@deploy1003: otto: Backport for [[gerrit:1289408|BugFix: Emit page_change at version 1.6.0 to pick up user wiki_id (T426198)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416200 wikitext text/x-wiki 2dlvuu5a24aevlly31a6vs8eeqic3rr 2416201 2416200 2026-05-19T19:44:34Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pybal-test2003.codfw.wmnet 2416201 wikitext text/x-wiki g61e77eggsc204be9xeneuhnlvji5na 2416202 2416201 2026-05-19T19:44:39Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1155-1158].eqiad.wmnet 2416202 wikitext text/x-wiki e95o1r2r8vgsa4zuflvshg8b8i8g99d 2416203 2416202 2026-05-19T19:47:06Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet 2416203 wikitext text/x-wiki tp588y7i0r9vqi1dw1g8do7oti75q23 2416205 2416203 2026-05-19T19:50:43Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet 2416205 wikitext text/x-wiki 2ns5b4npukhrmuaxay3giwbpq17g5w8 2416206 2416205 2026-05-19T19:51:17Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1155-1158].eqiad.wmnet 2416206 wikitext text/x-wiki h52t5u40zujmr7h9yj6fpvgwkqje62p 2416207 2416206 2026-05-19T19:51:19Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1155-1158].eqiad.wmnet 2416207 wikitext text/x-wiki ky5k0fwjgzmg3up7y7r4a6s4j67ghtd 2416208 2416207 2026-05-19T19:51:30Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1159-1162].eqiad.wmnet 2416208 wikitext text/x-wiki sit46mbnj2kmv1pbqljnogvde9i0xcd 2416209 2416208 2026-05-19T19:51:33Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet 2416209 wikitext text/x-wiki gtfom2k82s2e0v085lleuixqvljrssu 2416210 2416209 2026-05-19T19:53:05Z Stashbot 7414 otto@deploy1003: otto: Continuing with deployment 2416210 wikitext text/x-wiki qp6rel97fvdwitpkx3iy0inypr5f8hh 2416211 2416210 2026-05-19T19:53:29Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp7011.magru.wmnet 2416211 wikitext text/x-wiki 7fba018k91rmbfc3i7x19tv0sfdft2n 2416212 2416211 2026-05-19T19:53:51Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp7003.magru.wmnet 2416212 wikitext text/x-wiki d8lc86x5qnpot1cwcqt3xr1dw6240v3 2416213 2416212 2026-05-19T19:54:20Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1159-1162].eqiad.wmnet 2416213 wikitext text/x-wiki bw09niy6mar4u3wqozc6crs06ukr2tw 2416214 2416213 2026-05-19T19:55:24Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet 2416214 wikitext text/x-wiki auk2kqwyzjly4oix07xic2ay4r3reok 2416215 2416214 2026-05-19T19:57:38Z Stashbot 7414 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289408|BugFix: Emit page_change at version 1.6.0 to pick up user wiki_id (T426198)]] (duration: 16m 15s) 2416215 wikitext text/x-wiki qs1ibr3ymbepp4f0s0yl3ggd632lfbv 2416216 2416215 2026-05-19T20:01:21Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1159-1162].eqiad.wmnet 2416216 wikitext text/x-wiki 9duw0lpvnd1jp3h8y2k750gps0oa5fy 2416217 2416216 2026-05-19T20:01:23Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1159-1162].eqiad.wmnet 2416217 wikitext text/x-wiki pbcf8i4a7v3oc0ayo1i3tyw11ruzfvf 2416218 2416217 2026-05-19T20:01:33Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1163-1165,1240].eqiad.wmnet 2416218 wikitext text/x-wiki rzv46st5fr6pnekdugwcsnkguca87f8 2416219 2416218 2026-05-19T20:02:32Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host acmechief1002.eqiad.wmnet 2416219 wikitext text/x-wiki 9029ghz2gvvo712k1f8dmnjupnzv325 2416220 2416219 2026-05-19T20:03:51Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1163-1165,1240].eqiad.wmnet 2416220 wikitext text/x-wiki mkwgy18gidwbcz1sjds3q8sf8gktwwz 2416221 2416220 2026-05-19T20:06:24Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1002.eqiad.wmnet 2416221 wikitext text/x-wiki 1r6ozgm3ov5vo254y5pznkxtl7pm9d9 2416222 2416221 2026-05-19T20:07:07Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host acmechief2002.codfw.wmnet 2416222 wikitext text/x-wiki 2rbpiiy2lluz9woemu2y3q5csrv9nkm 2416223 2416222 2026-05-19T20:08:17Z Stashbot 7414 tgr@deploy1003: Started scap sync-world: Backport for [[gerrit:1288953|Add CommonsFinder to $wgUrlProtocols (T426614)]], [[gerrit:1289352|Remove unused ParsoidFragmentInput and ParsoidFragmentSupport]] 2416223 wikitext text/x-wiki ckc289qyhnjtps3am5fd82gkadv2558 2416224 2416223 2026-05-19T20:10:14Z Stashbot 7414 tgr@deploy1003: cscott, tgr: Backport for [[gerrit:1288953|Add CommonsFinder to $wgUrlProtocols (T426614)]], [[gerrit:1289352|Remove unused ParsoidFragmentInput and ParsoidFragmentSupport]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416224 wikitext text/x-wiki 6o5lqomloealchdhlmkmldgfrm5jquh 2416225 2416224 2026-05-19T20:10:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1163-1165,1240].eqiad.wmnet 2416225 wikitext text/x-wiki pxz1oisr8ctkah1a6zfx9ibiaz5a67u 2416226 2416225 2026-05-19T20:10:51Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1163-1165,1240].eqiad.wmnet 2416226 wikitext text/x-wiki manvvlbqk12u7waifh81ms6jhxwpwm7 2416227 2416226 2026-05-19T20:10:57Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2002.codfw.wmnet 2416227 wikitext text/x-wiki aekt4onhc83wtp620nmcl7b79jwnoct 2416228 2416227 2026-05-19T20:11:01Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1241-1244].eqiad.wmnet 2416228 wikitext text/x-wiki j0qsb1h1tv0sx2j5e4srub3yo8osjns 2416229 2416228 2026-05-19T20:12:26Z Stashbot 7414 tgr@deploy1003: cscott, tgr: Continuing with deployment 2416229 wikitext text/x-wiki kxefa7um2of33sm45hyymemz9vauxcl 2416230 2416229 2026-05-19T20:13:12Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1241-1244].eqiad.wmnet 2416230 wikitext text/x-wiki 0l2scieyg7l5zfdyrc27q15h4fko94y 2416231 2416230 2026-05-19T20:16:42Z Stashbot 7414 tgr@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288953|Add CommonsFinder to $wgUrlProtocols (T426614)]], [[gerrit:1289352|Remove unused ParsoidFragmentInput and ParsoidFragmentSupport]] (duration: 08m 25s) 2416231 wikitext text/x-wiki t7taqhc55og5tkvgvuz421s55c3zk57 2416232 2416231 2026-05-19T20:20:47Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1241-1244].eqiad.wmnet 2416232 wikitext text/x-wiki gkydjaieonshb4vzsvgnxp9rn5alq5e 2416233 2416232 2026-05-19T20:20:50Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1241-1244].eqiad.wmnet 2416233 wikitext text/x-wiki n5c1zczqsc423vjlkqignwgpeo48pum 2416234 2416233 2026-05-19T20:21:00Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1245-1248].eqiad.wmnet 2416234 wikitext text/x-wiki jgijhbv50158704g6mjlts0t0dlr7t5 2416235 2416234 2026-05-19T20:23:11Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1245-1248].eqiad.wmnet 2416235 wikitext text/x-wiki pt41ieo9i751ehvioi8iphvbzccmtyh 2416238 2416235 2026-05-19T20:29:27Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir and not A:ncredir-magru and A:ncredir 2416238 wikitext text/x-wiki 7w9n7xf3ghxw4eynyoa15tgqu2nhjy5 2416239 2416238 2026-05-19T20:32:21Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1288983|Revert^2 "Include xff in search logs"]] 2416239 wikitext text/x-wiki rhsvdkfjhqfzzk1bo27xerx24irmbi0 2416240 2416239 2026-05-19T20:34:15Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1288983|Revert^2 "Include xff in search logs"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416240 wikitext text/x-wiki cmgj38p70mq9hn8zqq0hj7a4jmharq5 2416241 2416240 2026-05-19T20:35:16Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp7004.magru.wmnet 2416241 wikitext text/x-wiki ixm7ottwev1c21273vlwakcfuncp7el 2416242 2416241 2026-05-19T20:35:18Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp700[3-4].magru.wmnet} and A:cp 2416242 wikitext text/x-wiki m9u6xz9fo24xa23pr0210tgk1xe0ncr 2416243 2416242 2026-05-19T20:35:21Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp7012.magru.wmnet 2416243 wikitext text/x-wiki k2z3ofvsy0k83xdtzxunnn0mj3b2c65 2416244 2416243 2026-05-19T20:35:23Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp701[1-2].magru.wmnet} and A:cp 2416244 wikitext text/x-wiki 1jf60b313erltkharxhusagw8bt6x17 2416245 2416244 2026-05-19T20:36:24Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Continuing with deployment 2416245 wikitext text/x-wiki jvac19hv43v22yoqy68fg1yu7b4he6q 2416246 2416245 2026-05-19T20:40:33Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288983|Revert^2 "Include xff in search logs"]] (duration: 08m 12s) 2416246 wikitext text/x-wiki glgqebdhqgudlgf4kir7a5ffm0ws92u 2416247 2416246 2026-05-19T20:47:16Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7003.magru.wmnet} and A:liberica 2416247 wikitext text/x-wiki ffzauu8nwcq3q53mqpbwp9e3ce4pk08 2416248 2416247 2026-05-19T20:49:11Z Stashbot 7414 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1288999|Explicitly set wgCSPUseReportURIDirective and not wmgCSPUseReportURIDirective to true (T424058)]] 2416248 wikitext text/x-wiki 5fa9sd86lagoq0yvcj7fhlh3n1y948n 2416249 2416248 2026-05-19T20:51:05Z Stashbot 7414 sbassett@deploy1003: sbassett: Backport for [[gerrit:1288999|Explicitly set wgCSPUseReportURIDirective and not wmgCSPUseReportURIDirective to true (T424058)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416249 wikitext text/x-wiki iqou93e1ic0s3l6pei84ty6yt3yrfig 2416250 2416249 2026-05-19T20:51:11Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7003.magru.wmnet} and A:liberica 2416250 wikitext text/x-wiki ggh6o3r0p9s1kixcpgcixz0cbv6ua62 2416251 2416250 2026-05-19T20:51:38Z Stashbot 7414 sbassett@deploy1003: sbassett: Continuing with deployment 2416251 wikitext text/x-wiki 3vlhcrqjfynngp23nglkjw02drn9m8k 2416252 2416251 2026-05-19T20:55:52Z Stashbot 7414 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288999|Explicitly set wgCSPUseReportURIDirective and not wmgCSPUseReportURIDirective to true (T424058)]] (duration: 06m 40s) 2416252 wikitext text/x-wiki 8hf76gfop52z99s03jpu47u5d8r21hi 2416253 2416252 2026-05-19T21:04:37Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2416253 wikitext text/x-wiki rrow1diswsc5o0gw3o6ifo31lghgzpy 2416254 2416253 2026-05-19T21:08:30Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs1036 to eqiad - jclark@cumin1003" 2416254 wikitext text/x-wiki 0fy68x8q982x2yrdoxgewh47nty5snj 2416255 2416254 2026-05-19T21:08:35Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs1036 to eqiad - jclark@cumin1003" 2416255 wikitext text/x-wiki avtv3x9xkky899zyaxzpojnpyrbk2el 2416256 2416255 2026-05-19T21:08:37Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2416256 wikitext text/x-wiki dnv3gdwe886wvsrxsj7qjzkpokraik7 2416257 2416256 2026-05-19T21:09:06Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416257 wikitext text/x-wiki hv9faai6tg6ljwe28gquydipvbwnh4d 2416258 2416257 2026-05-19T21:09:09Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416258 wikitext text/x-wiki jkxzdbcxzeq8thn1z2fukdohog1fe1d 2416259 2416258 2026-05-19T21:09:46Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416259 wikitext text/x-wiki s26w289u6ku76pfgkdsb9panza8ikps 2416260 2416259 2026-05-19T21:09:49Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416260 wikitext text/x-wiki 68t2px6z3e1o7ytmybff2kzg7ggod0g 2416261 2416260 2026-05-19T21:10:08Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416261 wikitext text/x-wiki gyq8wv8mo3j405z3vdzhphr27i8x930 2416262 2416261 2026-05-19T21:10:57Z Stashbot 7414 cdanis: 💔cdanis@apt1002.wikimedia.org ~ 🕔🍺 sudo -i reprepro -C main --ignore=wrongdistribution copy bookworm-wikimedia trixie-wikimedia cidergrinder 2416262 wikitext text/x-wiki 85mzhsmcavto262z5ne25hx22ynz484 2416263 2416262 2026-05-19T21:11:16Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416263 wikitext text/x-wiki nymdgvcfwxo6koyjwjmn8egfz73ckis 2416264 2416263 2026-05-19T21:12:59Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416264 wikitext text/x-wiki m8o7rh1ypzqqiuvssgffdm1hedvfoar 2416265 2416264 2026-05-19T21:13:02Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416265 wikitext text/x-wiki 5pxr5n4796x833gkdfsvx0nev6n5dzd 2416266 2416265 2026-05-19T21:15:21Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2014.codfw.wmnet 2416266 wikitext text/x-wiki ksqb6dv1wsd09mj6aor2hs10rj5gdxp 2416267 2416266 2026-05-19T21:16:08Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1036 2416267 wikitext text/x-wiki 1yr9cuz8afn0v9fwbvvrb5g5mwjjivp 2416268 2416267 2026-05-19T21:16:11Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs1036 2416268 wikitext text/x-wiki 0cd5ki9glmv58qknudooz1cdtz9pwgj 2416269 2416268 2026-05-19T21:18:34Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416269 wikitext text/x-wiki bx4atmw58y5yn63bgns1qkiai7p4gbc 2416270 2416269 2026-05-19T21:18:37Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1037 2416270 wikitext text/x-wiki o4iz96ia08slkkvdkq6641oibzgpo4z 2416271 2416270 2026-05-19T21:18:38Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416271 wikitext text/x-wiki rd16eyfu7icft6xyjm503kuwlecqryx 2416272 2416271 2026-05-19T21:19:33Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1037 2416272 wikitext text/x-wiki da1woimxcjznhknegsvic88f1fkt161 2416273 2416272 2026-05-19T21:20:41Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2014.codfw.wmnet 2416273 wikitext text/x-wiki 5j9smwj81dnvcqb3tr73ryn9xwho3ur 2416274 2416273 2026-05-19T21:22:08Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416274 wikitext text/x-wiki b7s8c05fl6kdwbulgbarllqjaaosnr8 2416276 2416274 2026-05-19T21:23:20Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416276 wikitext text/x-wiki mwjar4qz7t5enl43vetj2mdo7y342xy 2416277 2416276 2026-05-19T21:27:30Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416277 wikitext text/x-wiki b90g78k6j9naq6zoq1twmhei94drk26 2416280 2416277 2026-05-19T21:33:46Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416280 wikitext text/x-wiki d275zgsb2zdrciqcl1lm63dexv3kjrj 2416281 2416280 2026-05-19T21:34:03Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416281 wikitext text/x-wiki pvp1cfcuyx9o791ybpq6qoikrcf41qo 2416283 2416281 2026-05-19T21:41:14Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1037 2416283 wikitext text/x-wiki eub1jsggwojflfwad4mzkua1imrjros 2416284 2416283 2026-05-19T21:41:31Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416284 wikitext text/x-wiki rw3c2w5bwiugxlm0f61d8q7926p0sjw 2416285 2416284 2026-05-19T21:42:13Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1037 2416285 wikitext text/x-wiki 4yr30tnoo1td9h9qkc88tazw3qr5vk3 2416286 2416285 2026-05-19T21:46:20Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.reboot-nodes (exit_code=1) rolling reboot on P{wikikube-worker[1006-1007,1015-1016,1021,1034-1057,1064-1081,1084-1087,1093-1095,1113-1165,1240-1289,1291-1327,1375-1384].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2416286 wikitext text/x-wiki m18mqzush748kldvmcxjy5vr7ck0w7s 2416287 2416286 2026-05-19T21:48:46Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2416287 wikitext text/x-wiki dcm0qsouaoje7zt5w17cwfbo08hwy0x 2416295 2416287 2026-05-19T22:16:42Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs2012.codfw.wmnet 2416295 wikitext text/x-wiki ny94o9yf73s11fugx6q3ojdxnp8r6mo 2416296 2416295 2026-05-19T22:16:44Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2012.codfw.wmnet 2416296 wikitext text/x-wiki aje16upndky8uuz7rmf3mtrpq9k65go 2416297 2416296 2026-05-19T22:18:03Z Stashbot 7414 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on lvs2012.codfw.wmnet with reason: MD RAID failure 2416297 wikitext text/x-wiki 46go2vlgtgriw0h1ryxxmy5xl9l2hf1 2416298 2416297 2026-05-19T22:39:20Z Stashbot 7414 brett: disabling pybal/puppet on lvs2012 due to hardware misconfiguration/failure - T425890 2416298 wikitext text/x-wiki 2e8a6ai1c1trdvpjhq9esq4uxme50yj 2416305 2416298 2026-05-19T23:20:48Z Stashbot 7414 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf11u2+icu72u1 into component/php83-icu72 2416305 wikitext text/x-wiki nz2owct3e4710cz3904z9tg2lbn36ce 2416313 2416305 2026-05-20T00:05:15Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1015.eqiad.wmnet with OS trixie 2416313 wikitext text/x-wiki m6b51cv22v653k7yni1yxoanc24tjy4 2416319 2416313 2026-05-20T00:36:40Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1287441|Limit $wgThumbLimits to three options (T426328)]] 2416319 wikitext text/x-wiki 5iic27tgylqy680io5v6ynwbr8by7jn 2416320 2416319 2026-05-20T00:38:40Z Stashbot 7414 ladsgroup@deploy1003: jdlrobson, ladsgroup: Backport for [[gerrit:1287441|Limit $wgThumbLimits to three options (T426328)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416320 wikitext text/x-wiki toz178nxfre2ac6dsbbg19p1ntjeff5 2416321 2416320 2026-05-20T00:39:03Z Stashbot 7414 ladsgroup@deploy1003: jdlrobson, ladsgroup: Continuing with deployment 2416321 wikitext text/x-wiki b9oiyeze162vr98qm1ym7xe2wf9kpbi 2416322 2416321 2026-05-20T00:43:13Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287441|Limit $wgThumbLimits to three options (T426328)]] (duration: 06m 33s) 2416322 wikitext text/x-wiki fdpxrjzz4qbg4c3zwamyicp71tegtkl 2416324 2416322 2026-05-20T00:46:39Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1288274|404.php: Force a redirect to /wiki/ in very obvious cases (T129433)]] 2416324 wikitext text/x-wiki rl2rvu3jv9s51aow8h9qgnzsui8hd8a 2416326 2416324 2026-05-20T00:47:22Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage 2416326 wikitext text/x-wiki s2fsmvn4yz9hbrnfc12oo38jwrxqlwb 2416328 2416326 2026-05-20T00:48:30Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1288274|404.php: Force a redirect to /wiki/ in very obvious cases (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416328 wikitext text/x-wiki o4dhze07ab44y64kk69s6aoaoispurk 2416329 2416328 2026-05-20T00:50:34Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2416329 wikitext text/x-wiki c9jnsl23road2z0xotuafnih0xdjk5m 2416330 2416329 2026-05-20T00:54:15Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage 2416330 wikitext text/x-wiki rkuqxi1uamjvmpd1myabk8i2obhydhs 2416331 2416330 2026-05-20T00:54:47Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288274|404.php: Force a redirect to /wiki/ in very obvious cases (T129433)]] (duration: 08m 10s) 2416331 wikitext text/x-wiki 3gmgbosqd9daq8jdhluvfkaix87o7ti 2416334 2416331 2026-05-20T01:10:34Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1003" 2416334 wikitext text/x-wiki 4n7eae09we99bc9wssb8l4qdbvqunhm 2416336 2416334 2026-05-20T01:23:39Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7011.* 2416336 wikitext text/x-wiki nf5x7qlqs6p7wwv2xtbpr39yb3euoln 2416337 2416336 2026-05-20T01:26:09Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host cp7011.magru.wmnet 2416337 wikitext text/x-wiki 98yr6tk5dz3uzl2mrrvm6s30hhpwilq 2416339 2416337 2026-05-20T01:29:54Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=ncredir7003.* 2416339 wikitext text/x-wiki pzk4daeayvf9uol8qvwsjzfe7qbv2yu 2416340 2416339 2026-05-20T01:30:47Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1003" 2416340 wikitext text/x-wiki m5mjakpffl4w2parjlueg9rwbch2sv5 2416341 2416340 2026-05-20T01:30:49Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1015.eqiad.wmnet with OS trixie 2416341 wikitext text/x-wiki abo4a6n57nd198ac16xz77tzqpqjkvd 2416343 2416341 2026-05-20T01:36:12Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp7011.magru.wmnet 2416343 wikitext text/x-wiki 4zvpblaw5s964w1jo4dytqzf64zlr3t 2416344 2416343 2026-05-20T01:41:17Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.* 2416344 wikitext text/x-wiki jlfrmubovs1vmcrnvec5iua06p155t5 2416345 2416344 2026-05-20T02:17:34Z Stashbot 7414 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply 2416345 wikitext text/x-wiki ffrkpm7joa8dzac34xvecvo1vmp4euj 2416346 2416345 2026-05-20T02:17:47Z Stashbot 7414 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply 2416346 wikitext text/x-wiki g54ns27r5apktsn334pa0cw2nzoioyp 2416347 2416346 2026-05-20T02:17:49Z Stashbot 7414 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2416347 wikitext text/x-wiki rwe7cozn0vspy54owr9tqes5nkn59h6 2416348 2416347 2026-05-20T02:18:00Z Stashbot 7414 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2416348 wikitext text/x-wiki dsemgwmtpluxvp2xyfulpabuzx1co1v 2416349 2416348 2026-05-20T02:18:01Z Stashbot 7414 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2416349 wikitext text/x-wiki 7eluab4xp3fhhulo4c5eqxrbq2endba 2416350 2416349 2026-05-20T02:18:15Z Stashbot 7414 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2416350 wikitext text/x-wiki 1a7gmz9xkntwvx73kxhojtvf6xzaxi1 2416355 2416350 2026-05-20T05:27:04Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2022.codfw.wmnet,pc[1012,1022].eqiad.wmnet with reason: Maintenance on pc1 2416355 wikitext text/x-wiki 3npn21bm5tdpaop6eg1r5f5ia31ds6q 2416356 2416355 2026-05-20T05:27:19Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1012.eqiad.wmnet: Maintenance on pc2 2416356 wikitext text/x-wiki 9oagysgpppjffanlbxaagxl0mak1moy 2416357 2416356 2026-05-20T05:29:00Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2416357 wikitext text/x-wiki 2ckbro34svidil29q5p81km11t40tvl 2416358 2416357 2026-05-20T05:29:08Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2416358 wikitext text/x-wiki 3qi1zwn5qkeeol8nmpvd28vjqm84h47 2416359 2416358 2026-05-20T05:29:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1012.eqiad.wmnet: Maintenance on pc2 2416359 wikitext text/x-wiki m9y7nc4r9hwdk13aew82xya1kdij9hj 2416360 2416359 2026-05-20T05:41:50Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1011 from dbctl T426806', diff saved to https://phabricator.wikimedia.org/P92647 and previous config saved to /var/cache/conftool/dbconfig/20260520-054146-marostegui.json 2416360 wikitext text/x-wiki lv18iumxgxnkqq7bz818h0jx5aaiqit 2416361 2416360 2026-05-20T06:02:07Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on ms-backup[2003-2004].codfw.wmnet with reason: restart 2416361 wikitext text/x-wiki 0qfdg6jzvdt99r4gfgvhnf7zleri32p 2416362 2416361 2026-05-20T06:07:36Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 18 hosts with reason: restart 2416362 wikitext text/x-wiki anlygyweh3kwvjly6k9r2g8unjjn2pm 2416363 2416362 2026-05-20T06:21:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1011.eqiad.wmnet 2416363 wikitext text/x-wiki 9degi2vyutrrh45kec4prbvnkib7hq4 2416364 2416363 2026-05-20T06:27:09Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2416364 wikitext text/x-wiki o2j56lvopucqeydgyam3p5sxwpzcnka 2416365 2416364 2026-05-20T06:31:05Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 18 hosts with reason: restart 2416365 wikitext text/x-wiki 5nh8ubh9jvel6vxibpa8xavsz7x3vsz 2416366 2416365 2026-05-20T06:31:34Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet 2416366 wikitext text/x-wiki jj47qcg7w55c0udv5da644ponnpxh1v 2416367 2416366 2026-05-20T06:31:41Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2030.codfw.wmnet 2416367 wikitext text/x-wiki fgkg2lbhkhtsp1fzxysofgx46bi8pxc 2416368 2416367 2026-05-20T06:31:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet 2416368 wikitext text/x-wiki 5hw6xab5gcdk2e6g0f9reha80o3ozo0 2416369 2416368 2026-05-20T06:33:01Z Stashbot 7414 moritzm: failover Ganeti cluster in drmrs02 to ganeti6002 2416369 wikitext text/x-wiki tmoo7ns1acwcg2mi5r5exrlcae0t874 2416370 2416369 2026-05-20T06:34:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1011.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2416370 wikitext text/x-wiki lnurjwgj99z7nmuvlyocu9z2lk18a9j 2416371 2416370 2026-05-20T06:35:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1011.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2416371 wikitext text/x-wiki gyficjq0c7u2neptoxg9iq86avb930f 2416372 2416371 2026-05-20T06:35:39Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2416372 wikitext text/x-wiki b7mrlichcqvarrvm4p7b79to96fg3vd 2416373 2416372 2026-05-20T06:35:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1011.eqiad.wmnet 2416373 wikitext text/x-wiki 0s5inquf39lj8gxvcec63a2m1yx0yeg 2416374 2416373 2026-05-20T06:44:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet 2416374 wikitext text/x-wiki tk25w6821gttkjwuotgl53qay740s1m 2416375 2416374 2026-05-20T06:44:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet 2416375 wikitext text/x-wiki n797blwb1lcg83mn6zuvg225rwbp425 2416376 2416375 2026-05-20T06:45:11Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1249: Repooling after boot 2416376 wikitext text/x-wiki h6vdxrerzi82tip1effvzmttcreg7d6 2416377 2416376 2026-05-20T06:46:20Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1249.eqiad.wmnet 2416377 wikitext text/x-wiki lco324t01hqy7w4i51xz8lymd23d6mr 2416378 2416377 2026-05-20T06:46:23Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1249.eqiad.wmnet 2416378 wikitext text/x-wiki dqw1o6t4r120pri4ohuu4yyz7zrlaan 2416379 2416378 2026-05-20T06:50:49Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet 2416379 wikitext text/x-wiki bh8qsfu4b3cbrtan2c9h5jng0hw98ea 2416380 2416379 2026-05-20T06:50:57Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet 2416380 wikitext text/x-wiki 0i9sycxl806iwb2wumj15z2do0stdkk 2416383 2416380 2026-05-20T06:53:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet 2416383 wikitext text/x-wiki il45ajfj79d42dr0kutdmgb2deebies 2416384 2416383 2026-05-20T07:04:23Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1289743|Squashed diff to master]], [[gerrit:1289744|Squashed diff to master]] 2416384 wikitext text/x-wiki eibu3kxw7gftkgac5llbr1ye4p2vpoe 2416385 2416384 2026-05-20T07:04:37Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1055.eqiad.wmnet 2416385 wikitext text/x-wiki 4mlye8220jdmz6cpdq2gyx8hi5vabky 2416386 2416385 2026-05-20T07:06:28Z Stashbot 7414 mlitn@deploy1003: mlitn: Backport for [[gerrit:1289743|Squashed diff to master]], [[gerrit:1289744|Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416386 wikitext text/x-wiki qcq0synhrxydjpimyj3tl668ojzdg95 2416387 2416386 2026-05-20T07:09:10Z Stashbot 7414 moritzm: remove haveged 2416387 wikitext text/x-wiki aqnkc0jkudd34ig9hunow03z0gtwhd4 2416388 2416387 2026-05-20T07:10:21Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1055.eqiad.wmnet 2416388 wikitext text/x-wiki 4o1hj7ebqha4eubnyckype8ouyje2oo 2416389 2416388 2026-05-20T07:11:06Z Stashbot 7414 mlitn@deploy1003: mlitn: Continuing with deployment 2416389 wikitext text/x-wiki 6b1c72qitxa8xu5g8t0wquosn9agnjd 2416390 2416389 2026-05-20T07:15:14Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289743|Squashed diff to master]], [[gerrit:1289744|Squashed diff to master]] (duration: 10m 51s) 2416390 wikitext text/x-wiki 6id1ykg7e6rowd0gatr55gr52sm0qsq 2416391 2416390 2026-05-20T07:17:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1056.eqiad.wmnet 2416391 wikitext text/x-wiki r7awt61youudx868mv4pwez4kyxdmyu 2416392 2416391 2026-05-20T07:17:38Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1289748|Fix wordmark dimensions]], [[gerrit:1289749|Fix wordmark dimensions]] 2416392 wikitext text/x-wiki 9rlsjdmkwr5uhw7vccgyj1npx65ljvw 2416393 2416392 2026-05-20T07:17:57Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host build2001.codfw.wmnet 2416393 wikitext text/x-wiki sohqaodph2spfxz4bqz9rsoyuuexkok 2416394 2416393 2026-05-20T07:18:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet 2416394 wikitext text/x-wiki ptdvzuhnf0c1kujy82cun2fz2jwbbd9 2416395 2416394 2026-05-20T07:19:27Z Stashbot 7414 mlitn@deploy1003: mlitn: Backport for [[gerrit:1289748|Fix wordmark dimensions]], [[gerrit:1289749|Fix wordmark dimensions]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416395 wikitext text/x-wiki lkgtva8yked34sztke5tpb5w0pbruac 2416396 2416395 2026-05-20T07:19:59Z Stashbot 7414 mlitn@deploy1003: mlitn: Continuing with deployment 2416396 wikitext text/x-wiki kqrhxpl2bi7kx4enibtnm2zsex3756u 2416397 2416396 2026-05-20T07:23:20Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1056.eqiad.wmnet 2416397 wikitext text/x-wiki avw7c7e8w8bzx6erts9uceq1584hhof 2416398 2416397 2026-05-20T07:24:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2001.codfw.wmnet 2416398 wikitext text/x-wiki 6myoailxsduphoui0y8bfauniynd10b 2416399 2416398 2026-05-20T07:24:06Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289748|Fix wordmark dimensions]], [[gerrit:1289749|Fix wordmark dimensions]] (duration: 06m 28s) 2416399 wikitext text/x-wiki q27s39ik2nuc5o1qjac1vfc3pz3x24a 2416400 2416399 2026-05-20T07:25:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet 2416400 wikitext text/x-wiki o3f8rqckprnujvjgklzam2fc0p13tar 2416401 2416400 2026-05-20T07:25:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet 2416401 wikitext text/x-wiki 15lbr6rtu236mo92rr6frjvdv6sn1an 2416402 2416401 2026-05-20T07:25:27Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1092.eqiad.wmnet 2416402 wikitext text/x-wiki tdzj5920up2gh4voi70gz4897go45g3 2416404 2416402 2026-05-20T07:25:51Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2092.codfw.wmnet 2416404 wikitext text/x-wiki edu0zl2qzru95bel13vlkqmec8uidzf 2416405 2416404 2026-05-20T07:27:09Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host serpens.wikimedia.org 2416405 wikitext text/x-wiki m84p7r0q0ohlcxwwh38kfbgiy44tc7j 2416406 2416405 2026-05-20T07:27:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2416406 wikitext text/x-wiki sqs181o1fa0n8l1l3vu5xmy1nwsn9i2 2416407 2416406 2026-05-20T07:27:18Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet 2416407 wikitext text/x-wiki n7orhvqtx271k8fvasbums4sqjutzid 2416408 2416407 2026-05-20T07:27:33Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2187: Upgrading db2187.codfw.wmnet 2416408 wikitext text/x-wiki iexk2a5h4grt06f0crjunyj7tzjmo84 2416409 2416408 2026-05-20T07:28:00Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2187: Upgrading db2187.codfw.wmnet 2416409 wikitext text/x-wiki 5lu1rh0ptwvcuo8or9wo61lr1k7tj7l 2416410 2416409 2026-05-20T07:30:40Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1249: Repooling after boot 2416410 wikitext text/x-wiki i3spgkkftkeuqttkyoj6bqb2ds9m89e 2416411 2416410 2026-05-20T07:30:53Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host serpens.wikimedia.org 2416411 wikitext text/x-wiki n7fid337fm3qho1ca0x2yga5patqvqf 2416412 2416411 2026-05-20T07:32:10Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1092.eqiad.wmnet 2416412 wikitext text/x-wiki nrjkex0v6ao6lt7oc8c7x4vqfis0tn6 2416413 2416412 2026-05-20T07:32:15Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1093.eqiad.wmnet 2416413 wikitext text/x-wiki biur817p4toupf6flcl6mwumv0hma7h 2416414 2416413 2026-05-20T07:32:37Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet 2416414 wikitext text/x-wiki 0ys5kvn1o6hfab0tndzxlheb14opp08 2416415 2416414 2026-05-20T07:32:45Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2092.codfw.wmnet 2416415 wikitext text/x-wiki hzcc7l889s3z8zk8wisey85o4gi2kdv 2416416 2416415 2026-05-20T07:32:50Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2093.codfw.wmnet 2416416 wikitext text/x-wiki cmrq1b2kn6q9q28zuvimz3sr2qa72j9 2416417 2416416 2026-05-20T07:32:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host seaborgium.wikimedia.org 2416417 wikitext text/x-wiki 736asp5w0ib3fneii8e4ao75a4kkvjp 2416418 2416417 2026-05-20T07:34:44Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet 2416418 wikitext text/x-wiki lsxlftza4h8oe8zz33xc7eybtt7k1zl 2416419 2416418 2026-05-20T07:36:35Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host seaborgium.wikimedia.org 2416419 wikitext text/x-wiki brv9pqzs8zrqqi9d2ebi6wjqm6ijeeu 2416420 2416419 2026-05-20T07:38:58Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1093.eqiad.wmnet 2416420 wikitext text/x-wiki e25gpilxhrlxpyafhkbxdh51rg39fl8 2416421 2416420 2026-05-20T07:39:02Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1094.eqiad.wmnet 2416421 wikitext text/x-wiki 8i1ohqga5txfo6gx9exhr0c3urz5m5w 2416422 2416421 2026-05-20T07:39:37Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2093.codfw.wmnet 2416422 wikitext text/x-wiki rk2br90pjghk90dvtfulzvqumk14rhw 2416423 2416422 2026-05-20T07:39:42Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2094.codfw.wmnet 2416423 wikitext text/x-wiki sjmk06v0v5uxv5j2h4pwjepuan8v9ia 2416424 2416423 2026-05-20T07:40:26Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet 2416424 wikitext text/x-wiki hbbi65skub3gb3ci4turckho4jjdcfo 2416425 2416424 2026-05-20T07:43:26Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2187.codfw.wmnet with OS trixie 2416425 wikitext text/x-wiki drp3575eimq476zuv9h0ks7nfk8kedt 2416426 2416425 2026-05-20T07:45:33Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1094.eqiad.wmnet 2416426 wikitext text/x-wiki dk4e2obvb3x5trmrp6ffr6ec2bcvgd9 2416427 2416426 2026-05-20T07:45:38Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1095.eqiad.wmnet 2416427 wikitext text/x-wiki if58eibtrot2q6ezhi1udveqil13lr1 2416428 2416427 2026-05-20T07:46:09Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2094.codfw.wmnet 2416428 wikitext text/x-wiki cyd5jou6ug8kzwfc4vumxzsds80w6as 2416429 2416428 2026-05-20T07:46:14Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2095.codfw.wmnet 2416429 wikitext text/x-wiki mipuliore9vpbciojr26ltmpf05wim2 2416430 2416429 2026-05-20T07:51:47Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1024.eqiad.wmnet with reason: Reboot 2416430 wikitext text/x-wiki ml9klbhz3zyv0dt1kq8j4m1mqqqet3i 2416431 2416430 2026-05-20T07:52:43Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2095.codfw.wmnet 2416431 wikitext text/x-wiki 0zmhteqdeq1831gyuphwfeez52zcfne 2416432 2416431 2026-05-20T07:52:47Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2096.codfw.wmnet 2416432 wikitext text/x-wiki hnxofqateuv0h93uk3du8mgmyjoqdcv 2416433 2416432 2026-05-20T07:52:53Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1095.eqiad.wmnet 2416433 wikitext text/x-wiki h10pn70ucd8bh3xxy4ugzvne6vhtihx 2416434 2416433 2026-05-20T07:52:57Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1096.eqiad.wmnet 2416434 wikitext text/x-wiki fjuseh3gfbjp2xt9qfaup1ca2i5uq9u 2416435 2416434 2026-05-20T07:55:13Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1022.eqiad.wmnet with reason: Reboot 2416435 wikitext text/x-wiki rza033snedzk4nxrdu2wkvvbw6catmk 2416436 2416435 2026-05-20T07:59:07Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2096.codfw.wmnet 2416436 wikitext text/x-wiki nddsi2xxsu59ea9o2doemgyv4cmqcks 2416437 2416436 2026-05-20T07:59:40Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1096.eqiad.wmnet 2416437 wikitext text/x-wiki 5akcqmqxc8t89t4pdwii9z075k6sxf5 2416438 2416437 2026-05-20T07:59:45Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be1097.eqiad.wmnet 2416438 wikitext text/x-wiki n7fqgwfyq6t75dyd4cd3gaahiqnz208 2416439 2416438 2026-05-20T08:00:26Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1023.eqiad.wmnet with reason: Reboot 2416439 wikitext text/x-wiki 1fhdzmo84zbg6zhxnoirokxlhwjyreq 2416440 2416439 2026-05-20T08:01:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org 2416440 wikitext text/x-wiki f5hplo0osfwsn3smcm3egfek6otc4tn 2416441 2416440 2026-05-20T08:02:39Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2187.codfw.wmnet with reason: host reimage 2416441 wikitext text/x-wiki 6cx7oyxow0myv9msznhqjkmmi6mv8mr 2416442 2416441 2026-05-20T08:07:40Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2187.codfw.wmnet with reason: host reimage 2416442 wikitext text/x-wiki s88862rizpc5ji8kq5q3jm17tmcmikd 2416443 2416442 2026-05-20T08:08:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org 2416443 wikitext text/x-wiki 3d4piznn1d1w66u5b9aggw08nri03c8 2416444 2416443 2026-05-20T08:09:13Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1097.eqiad.wmnet 2416444 wikitext text/x-wiki 4uh4q2o9ylv3bpicr2jj8an9mnseitk 2416445 2416444 2026-05-20T08:11:39Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1026.eqiad.wmnet with reason: Reboot 2416445 wikitext text/x-wiki mtcn1bhkepbaog7n1lrwpgruvl2kmef 2416446 2416445 2026-05-20T08:17:09Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1029.eqiad.wmnet with reason: Reboot 2416446 wikitext text/x-wiki 0ncvwnd3xb2aby4sz4aaww2w03az3hr 2416447 2416446 2026-05-20T08:18:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org 2416447 wikitext text/x-wiki 4cqnnb5s7048h5am69fs264o15qm4ja 2416449 2416447 2026-05-20T08:20:57Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy[2005-2008].codfw.wmnet with reason: Reboot 2416449 wikitext text/x-wiki o1gku3ys2unjcsg79v1bptvcu1j5sqx 2416450 2416449 2026-05-20T08:21:44Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy[2006-2008].codfw.wmnet with reason: Reboot 2416450 wikitext text/x-wiki 4daqfnrzqnjye0ywsj6re76asjg724r 2416451 2416450 2026-05-20T08:24:38Z Stashbot 7414 moritzm: imported openjdk-8u492-ga-1~deb11u1 to component/jdk8 for bookworm (forward port of latest Java 8 security release) 2416451 wikitext text/x-wiki tu11a2cuvm58962l1u0u9on3n5eg5d4 2416452 2416451 2026-05-20T08:25:01Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org 2416452 wikitext text/x-wiki athq3l9fjas0a77y3vavofrx9qwpqpf 2416453 2416452 2026-05-20T08:25:41Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2187.codfw.wmnet with OS trixie 2416453 wikitext text/x-wiki 68xh6phflhspbijevs34e9j3rylk8nb 2416454 2416453 2026-05-20T08:26:06Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.3 refs T423912 2416454 wikitext text/x-wiki l8jknpr0pho7utyc2i9gvfa4uzw4kl5 2416458 2416454 2026-05-20T08:26:43Z Stashbot 7414 moritzm: installing Java 11 security updates 2416458 wikitext text/x-wiki krjw5pfprtegu3vi1wq8nok8x9luohj 2416460 2416458 2026-05-20T08:27:56Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2187: Migration of db2187.codfw.wmnet completed 2416460 wikitext text/x-wiki rvkj00g89zv27rrwuehqjznu38c4tf9 2416461 2416460 2026-05-20T08:35:28Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp700[5-6].magru.wmnet} and A:cp 2416461 wikitext text/x-wiki 2mwroy43hfggnn2oe6n1v8wkshyand6 2416462 2416461 2026-05-20T08:41:40Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply 2416462 wikitext text/x-wiki kna8koiiw9wwcx9rqz8f4ytf42z5uyu 2416463 2416462 2026-05-20T08:42:51Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply 2416463 wikitext text/x-wiki bummyaonwg8u246s9qdvu563lgawf4j 2416464 2416463 2026-05-20T08:46:50Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7005.magru.wmnet 2416464 wikitext text/x-wiki sezp1tz1ijhx8ip6iqz7li20zw2rztx 2416465 2416464 2026-05-20T08:50:46Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org 2416465 wikitext text/x-wiki l0uek0v66vlb5zw51tp6r1siybw1lce 2416466 2416465 2026-05-20T08:53:52Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-codfw 2416466 wikitext text/x-wiki dz2fplhvf94s91p6rysip0k8wzzi9s4 2416467 2416466 2026-05-20T08:53:57Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2001.codfw.wmnet 2416467 wikitext text/x-wiki nx5wmlwda9a1lrs7hbaveoqvc4rs9n7 2416468 2416467 2026-05-20T08:54:33Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2416468 wikitext text/x-wiki duovd4pbs43jh1zt8k8axaw1or5w602 2416469 2416468 2026-05-20T08:54:56Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2416469 wikitext text/x-wiki 3lqzqcoao5ee75d84nwdncndin93l8v 2416470 2416469 2026-05-20T08:57:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org 2416470 wikitext text/x-wiki oe1a3aw4rrdx1v47axsd2my1k2ees1w 2416471 2416470 2026-05-20T08:59:02Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2001.codfw.wmnet 2416471 wikitext text/x-wiki diwiqbawx0m8q3of7g05amwmfp55xdq 2416472 2416471 2026-05-20T09:01:35Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org 2416472 wikitext text/x-wiki i2qprrhf318jqsi3p6hkfcpso5oukxd 2416473 2416472 2026-05-20T09:02:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2416473 wikitext text/x-wiki h2dvq67o1yht25n0ccocc35yni6iv5u 2416474 2416473 2026-05-20T09:02:24Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2242: Upgrading db2242.codfw.wmnet 2416474 wikitext text/x-wiki 66ktq25hkav96hn99zpn45l5q5ykkwd 2416475 2416474 2026-05-20T09:03:04Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2242: Upgrading db2242.codfw.wmnet 2416475 wikitext text/x-wiki ioyxe9x4vm5vbmbsoplxm9p5zu0a0un 2416476 2416475 2026-05-20T09:04:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2242.codfw.wmnet with OS trixie 2416476 wikitext text/x-wiki 27qtldqa4jq7i8w18f9qprbhyo7q28c 2416477 2416476 2026-05-20T09:04:38Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 207947 2416477 wikitext text/x-wiki kzlx0yp3ushb48gezr9oy147xb2nbew 2416478 2416477 2026-05-20T09:04:55Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2001.codfw.wmnet 2416478 wikitext text/x-wiki mijglwkl1cka9gjme204yaog6d8go52 2416479 2416478 2026-05-20T09:04:57Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2001.codfw.wmnet 2416479 wikitext text/x-wiki kwmqmdd21vvufrsdjbpzi7pq3bz7t75 2416480 2416479 2026-05-20T09:05:00Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2416480 wikitext text/x-wiki qvw1o7ex7enpylfjyai5udfhoxyq9a9 2416481 2416480 2026-05-20T09:05:04Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2002.codfw.wmnet 2416481 wikitext text/x-wiki elfo0geu7ssrnhzvlb83uae5xu3yajx 2416482 2416481 2026-05-20T09:05:09Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 207947 2416482 wikitext text/x-wiki 5i8bynfbx9lflunfwmhy19yakw7o7on 2416483 2416482 2026-05-20T09:06:29Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 12389 2416483 wikitext text/x-wiki hc09ghmnw58aip8ehrk45v4c1ztj433 2416484 2416483 2026-05-20T09:07:08Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 12389 2416484 wikitext text/x-wiki k69zxieri0cfabfnh9qlbwi7re61p92 2416485 2416484 2026-05-20T09:08:26Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org 2416485 wikitext text/x-wiki l5lfohol9lz5g9gh2uy7eptx1mav08n 2416486 2416485 2026-05-20T09:10:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2416486 wikitext text/x-wiki 2dxbvrirluhg3kx8x5tnznwzmgi5g6i 2416487 2416486 2026-05-20T09:13:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org 2416487 wikitext text/x-wiki aud1j3m20cfn27ilgdrpg98zk4wttqz 2416489 2416487 2026-05-20T09:13:25Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2187: Migration of db2187.codfw.wmnet completed 2416489 wikitext text/x-wiki o6mv4nzjg75ueiis948d9qs1tsxavwe 2416490 2416489 2026-05-20T09:13:27Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2416490 wikitext text/x-wiki 0axeltwzsjr4y9xtc1wsvo9bnzz1dfm 2416491 2416490 2026-05-20T09:15:09Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2002.codfw.wmnet 2416491 wikitext text/x-wiki 4j8wo2zg8wxluxncpwogkklo4duc2qk 2416492 2416491 2026-05-20T09:15:12Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet 2416492 wikitext text/x-wiki h0v9n9v4cqlodvg14174xjihnr0xu2z 2416493 2416492 2026-05-20T09:18:57Z Stashbot 7414 moritzm: temporarily drop ganeti2030 from the codfw cluster T426199 2416493 wikitext text/x-wiki 3ynuszzzcsgo44buj6qonn2tantwqij 2416494 2416493 2026-05-20T09:19:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet 2416494 wikitext text/x-wiki 2b88n4kml9nylc2xkss2pg0194za7lh 2416495 2416494 2026-05-20T09:19:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org 2416495 wikitext text/x-wiki anb96yxds4pevyyvuztx0n89wxv9htz 2416496 2416495 2026-05-20T09:20:00Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2242.codfw.wmnet with reason: host reimage 2416496 wikitext text/x-wiki g7pjuhwqgneq1rein66ph5jb6go2xeq 2416497 2416496 2026-05-20T09:21:32Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org 2416497 wikitext text/x-wiki lu30lgfmzyg2ohifyuapm990az7hdry 2416498 2416497 2026-05-20T09:23:02Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet 2416498 wikitext text/x-wiki 2z94n5cgoovyqeuu0sylsnjb8ktmvec 2416499 2416498 2026-05-20T09:24:37Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2242.codfw.wmnet with reason: host reimage 2416499 wikitext text/x-wiki e0w1rpbepkp0ven63v5zjm773c0p457 2416500 2416499 2026-05-20T09:24:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet 2416500 wikitext text/x-wiki c0nb93wnkftiis5rsipnm0hc7pf9x0s 2416501 2416500 2026-05-20T09:26:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad 2416501 wikitext text/x-wiki ojvz2tig6oxsxx0cqodnyh8qwz85nui 2416502 2416501 2026-05-20T09:26:49Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet 2416502 wikitext text/x-wiki ooub6bwnkmdf2dbo49ag670fysbiszf 2416503 2416502 2026-05-20T09:27:15Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2002.codfw.wmnet 2416503 wikitext text/x-wiki kll579jx7dpe01l1tulv5ncvhzi7jpq 2416504 2416503 2026-05-20T09:27:17Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2002.codfw.wmnet 2416504 wikitext text/x-wiki 0dqjk3btdgxtqn0ntkabgxfbbtljd8j 2416505 2416504 2026-05-20T09:27:23Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2003.codfw.wmnet 2416505 wikitext text/x-wiki eugja5kikzaz93sghi8mr76der765wd 2416506 2416505 2026-05-20T09:28:11Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org 2416506 wikitext text/x-wiki nhd65hoq7t3zulz3lwxk47b3e8ov9la 2416507 2416506 2026-05-20T09:28:27Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7006.magru.wmnet 2416507 wikitext text/x-wiki si0d8qe3syw9f6k1srq6gp8kmfx9ihi 2416508 2416507 2026-05-20T09:28:30Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp700[5-6].magru.wmnet} and A:cp 2416508 wikitext text/x-wiki dx1redkyw068rjinr4fetjcj0fspqhr 2416509 2416508 2026-05-20T09:28:32Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.presto.reboot-workers for Presto an-presto cluster: Reboot Presto nodes 2416509 wikitext text/x-wiki pghexdj8j28ipj8gw7xo9gl494p03uk 2416510 2416509 2026-05-20T09:30:04Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2001.codfw.wmnet 2416510 wikitext text/x-wiki bo8ghkqv6nzp42uiwivsnw7dn77ijop 2416511 2416510 2026-05-20T09:30:20Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2002.codfw.wmnet 2416511 wikitext text/x-wiki lm2uo82ytr3njutlui0b4h00ihnrx3g 2416512 2416511 2026-05-20T09:30:48Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet 2416512 wikitext text/x-wiki 869hy05himokg47l8ja0xs10gf2hstq 2416513 2416512 2026-05-20T09:31:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet 2416513 wikitext text/x-wiki dmmv90r1mp00cuth3xehw25qvkvva80 2416514 2416513 2026-05-20T09:32:29Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2003.codfw.wmnet 2416514 wikitext text/x-wiki awc6kxev1oy3p79kiyhf93oim50yx3a 2416515 2416514 2026-05-20T09:33:36Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2002.codfw.wmnet 2416515 wikitext text/x-wiki 88w10qercjmjkw9p11don2x4lkr9jea 2416516 2416515 2026-05-20T09:33:57Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1002.eqiad.wmnet 2416516 wikitext text/x-wiki t523roeaatppqzci99ghtr58zskwj8g 2416517 2416516 2026-05-20T09:36:25Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2003.codfw.wmnet 2416517 wikitext text/x-wiki ls4nfp4yobbmbtmb33pm576yqclot1m 2416518 2416517 2026-05-20T09:36:28Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2003.codfw.wmnet 2416518 wikitext text/x-wiki 3i8kf5n8z4i5z3lzc81kgjl4asyfq9o 2416519 2416518 2026-05-20T09:36:33Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2004.codfw.wmnet 2416519 wikitext text/x-wiki ciptqxr9fgm7czpm5b0zqjbftfu867y 2416520 2416519 2026-05-20T09:37:25Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1002.eqiad.wmnet 2416520 wikitext text/x-wiki tqp0yxhnbklau86zg166cxegzt9wrqi 2416521 2416520 2026-05-20T09:37:36Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1001.eqiad.wmnet 2416521 wikitext text/x-wiki mnzmrn5q9fbn2461l3ioegiv69dhk8m 2416522 2416521 2026-05-20T09:40:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2032.codfw.wmnet 2416522 wikitext text/x-wiki nvs4t5n7arv7zc15h4ygd0pe03hicf2 2416523 2416522 2026-05-20T09:41:05Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1001.eqiad.wmnet 2416523 wikitext text/x-wiki ijd7wsar3j5rbc8wvwhfwrfyfw8ymtq 2416524 2416523 2026-05-20T09:41:38Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2004.codfw.wmnet 2416524 wikitext text/x-wiki tuxydrgj8bwytapi682k2gp90ks1zns 2416525 2416524 2026-05-20T09:42:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet 2416525 wikitext text/x-wiki ajdvb9omhg2zn93ejsytp1rpwefrb1h 2416526 2416525 2026-05-20T09:43:48Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2242.codfw.wmnet with OS trixie 2416526 wikitext text/x-wiki hpx7w2ft3x1unnllarsrtww5bxryav3 2416527 2416526 2026-05-20T09:43:58Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2416527 wikitext text/x-wiki h1evygusmcu4m3612vu24bbj64q29dd 2416528 2416527 2026-05-20T09:44:28Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2416528 wikitext text/x-wiki qc81gexm4xx55bfv2pgo505qm830fdq 2416529 2416528 2026-05-20T09:46:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2242: Migration of db2242.codfw.wmnet completed 2416529 wikitext text/x-wiki a2tv9lunacalsz4u4lsefto1aefk18t 2416530 2416529 2026-05-20T09:47:23Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2004.codfw.wmnet 2416530 wikitext text/x-wiki ajfegxyw1vwwl7k65lrbyne0xhd8yxt 2416531 2416530 2026-05-20T09:47:25Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2004.codfw.wmnet 2416531 wikitext text/x-wiki rt5l6qlq2yjovk218bvmz7reunp8rqw 2416532 2416531 2026-05-20T09:47:30Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet 2416532 wikitext text/x-wiki jstck1j0rezioku00cfb4wt52ux6f6b 2416533 2416532 2026-05-20T09:48:37Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet 2416533 wikitext text/x-wiki 3lu7020xl63l57hkmifhp2ncowr1dvw 2416534 2416533 2026-05-20T09:48:46Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2032.codfw.wmnet 2416534 wikitext text/x-wiki hl37bj3f4at3aaei2z86621nxhnvqqr 2416535 2416534 2026-05-20T09:52:36Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet 2416535 wikitext text/x-wiki ltgi3jflytvlbouezwfkpr1mgdgi3y7 2416536 2416535 2026-05-20T09:52:58Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2416536 wikitext text/x-wiki stvhyagm5lgx2x1gxcxdf4750ushx8y 2416538 2416536 2026-05-20T09:53:32Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2416538 wikitext text/x-wiki 84i4zgeu4kmjltjfounwlv2bxik7zlg 2416539 2416538 2026-05-20T09:56:55Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet 2416539 wikitext text/x-wiki qkp575o3brxu04v2ze3bc4nv7siak7h 2416540 2416539 2026-05-20T09:57:00Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet 2416540 wikitext text/x-wiki fnyysomil6k0x6ca3ouyn48js621xn4 2416541 2416540 2026-05-20T09:57:26Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet 2416541 wikitext text/x-wiki mog02y72uokn6j6xze974h81vc1ynt8 2416542 2416541 2026-05-20T09:57:28Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet 2416542 wikitext text/x-wiki 4hz2qp3d75v909tspzuq8eyv75rkibd 2416543 2416542 2026-05-20T09:57:34Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2006.codfw.wmnet 2416543 wikitext text/x-wiki h0wpuuet111fzfkv458au3nvcqwswnt 2416544 2416543 2026-05-20T09:57:45Z Stashbot 7414 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1015.eqiad.wmnet with reason: Rebooting clouddb1015 T415165 2416544 wikitext text/x-wiki n45xs0rzv2zhyvlso4sjtsue4h02qoo 2416545 2416544 2026-05-20T10:02:39Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2006.codfw.wmnet 2416545 wikitext text/x-wiki 5zi8loebady9y12jucxso5k3ox0ntr2 2416546 2416545 2026-05-20T10:02:56Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.hosts.reimage for host clouddb1015.eqiad.wmnet with OS trixie 2416546 wikitext text/x-wiki 52ui8fmaw0smmhvzj55yon7swb5iq5s 2416547 2416546 2026-05-20T10:03:44Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet 2416547 wikitext text/x-wiki ralfi5y3f7tixeyeqb3uwtz7n5xspxu 2416548 2416547 2026-05-20T10:03:46Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet 2416548 wikitext text/x-wiki 5ozlb6k1yc4fsj80urhj6qpjumq5loq 2416549 2416548 2026-05-20T10:03:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet 2416549 wikitext text/x-wiki erw2lzc7fnpd89dk27af5dfxdjqdd89 2416550 2416549 2026-05-20T10:04:08Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet 2416550 wikitext text/x-wiki nhppidm0sc4btnr7tdoc9u7nuv3081a 2416551 2416550 2026-05-20T10:04:24Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dborch1002.wikimedia.org with reason: Reboot 2416551 wikitext text/x-wiki o6q8554wejs8t1alolx49gykmmvzb35 2416552 2416551 2026-05-20T10:05:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2048.codfw.wmnet 2416552 wikitext text/x-wiki dh3zo5xd5894qvmnoy2ka6otrdlpxmp 2416553 2416552 2026-05-20T10:07:10Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2048.codfw.wmnet 2416553 wikitext text/x-wiki 7lb9stz93mkps08g9tgvrs938wd1fmb 2416554 2416553 2026-05-20T10:09:08Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2006.codfw.wmnet 2416554 wikitext text/x-wiki debjv7a3njll1k6b10lxl5p13lofx4d 2416555 2416554 2026-05-20T10:09:10Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2006.codfw.wmnet 2416555 wikitext text/x-wiki dw3yuwc46p2lku7cx8c2szq7cimb2j9 2416556 2416555 2026-05-20T10:09:16Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2007.codfw.wmnet 2416556 wikitext text/x-wiki 4jt7lh96gogc1of0g87hm3t1pu8ssl5 2416557 2416556 2026-05-20T10:11:04Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet 2416557 wikitext text/x-wiki 79itc8s4nyvayrifilpejx609bxx0df 2416558 2416557 2026-05-20T10:11:06Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet 2416558 wikitext text/x-wiki r2ohqpdk97a5dtp5hibvik7qmpuq6xh 2416559 2416558 2026-05-20T10:11:11Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet 2416559 wikitext text/x-wiki df5kwfaxmik1y23jzsbi2u2e9nxhwmt 2416560 2416559 2026-05-20T10:11:43Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet 2416560 wikitext text/x-wiki ax4c2ei40yj2mng7ijptjr43vbn3ayz 2416561 2416560 2026-05-20T10:12:26Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2048.codfw.wmnet 2416561 wikitext text/x-wiki 86zlqev7tfd9ki2kjd8nlecyk31rxsj 2416562 2416561 2026-05-20T10:12:31Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp701[3-4].magru.wmnet} and A:cp 2416562 wikitext text/x-wiki tipbfpuojqxj9euo4g954tpxlih0id0 2416563 2416562 2026-05-20T10:12:32Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2048.codfw.wmnet 2416563 wikitext text/x-wiki k259ow9qkjc8sn4xafmu8t72o7l1ryk 2416564 2416563 2026-05-20T10:13:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs-test1001.eqiad.wmnet with reason: host reimage 2416564 wikitext text/x-wiki bt0tl6mdinna0s9awsbr0du6509q6i0 2416565 2416564 2026-05-20T10:13:40Z Stashbot 7414 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on es2039.codfw.wmnet with reason: Maintenance 2416565 wikitext text/x-wiki 2g6hj729j3fuitcwjkch8z6bywwmc4v 2416566 2416565 2026-05-20T10:14:22Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2007.codfw.wmnet 2416566 wikitext text/x-wiki sxy1gv9r03slustcbu26yo71zxuppop 2416567 2416566 2026-05-20T10:15:30Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2049.codfw.wmnet 2416567 wikitext text/x-wiki n57r0ofkq28naer13m6fisyuka1uxdm 2416568 2416567 2026-05-20T10:17:48Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs-test1001.eqiad.wmnet with reason: host reimage 2416568 wikitext text/x-wiki gzpzpav9ldvmrinsp3ys1w4tpj4pp5x 2416569 2416568 2026-05-20T10:20:51Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2007.codfw.wmnet 2416569 wikitext text/x-wiki p1s2xy928udrt2c3vh93hdpjx7wz2q1 2416570 2416569 2026-05-20T10:20:53Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2007.codfw.wmnet 2416570 wikitext text/x-wiki 8nwk003crlar3zwwm6r9orwvxp0cf9s 2416571 2416570 2026-05-20T10:20:58Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2008.codfw.wmnet 2416571 wikitext text/x-wiki aaid2jrypdm55v3dob9um9w7oxycjio 2416572 2416571 2026-05-20T10:21:23Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2416572 wikitext text/x-wiki jbai348pxmq04as7edgybw8dgljqryw 2416574 2416572 2026-05-20T10:22:30Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2049.codfw.wmnet 2416574 wikitext text/x-wiki qroroh5uivkmswd34yewkkwnoh33735 2416575 2416574 2026-05-20T10:22:38Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw 2416575 wikitext text/x-wiki bxyi9844dtop3pfi8uxskg9q8i8t4cd 2416576 2416575 2026-05-20T10:23:02Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet 2416576 wikitext text/x-wiki 7uwms0z1nhu87vtwwf2y9lnsq2qz9ha 2416577 2416576 2026-05-20T10:23:04Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet 2416577 wikitext text/x-wiki 6b2d6svut69ndmfmsyypihstvywiiw3 2416578 2416577 2026-05-20T10:23:08Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet 2416578 wikitext text/x-wiki f0urwmooj31ben0tfdqwttd5gofmsz0 2416579 2416578 2026-05-20T10:23:20Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2416579 wikitext text/x-wiki d91j462x9djhicxu00jgnx674qiocny 2416580 2416579 2026-05-20T10:23:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet 2416580 wikitext text/x-wiki j5ki11x71rtj321y5h9k9wchwjvili5 2416581 2416580 2026-05-20T10:24:12Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7013.magru.wmnet 2416581 wikitext text/x-wiki h6k3kqvsj39ojp32akj7v05vkztn1t9 2416582 2416581 2026-05-20T10:27:22Z Stashbot 7414 fnegri@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddb1015.eqiad.wmnet with OS trixie 2416582 wikitext text/x-wiki 7jbxibrn08ykzvrhn4got42g0z8hy1a 2416583 2416582 2026-05-20T10:27:50Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2049.codfw.wmnet 2416583 wikitext text/x-wiki be8cxm2t5ggmsb1miy1z6e7fel5o5wl 2416584 2416583 2026-05-20T10:27:57Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2049.codfw.wmnet 2416584 wikitext text/x-wiki jzs3m0jjf89sv2ls2pdaseo0a6lbr08 2416585 2416584 2026-05-20T10:28:29Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.hosts.reimage for host clouddb1015.eqiad.wmnet with OS trixie 2416585 wikitext text/x-wiki rcnfi18kfu3s1tyl2yh2p7xj3p0akn2 2416586 2416585 2026-05-20T10:30:46Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet 2416586 wikitext text/x-wiki mkuxisf5e6lfsxyp53ctb2m0ybtevao 2416587 2416586 2026-05-20T10:30:48Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet 2416587 wikitext text/x-wiki hofcf06vtbt4q3z1ocfhbm12rv3lqly 2416588 2416587 2026-05-20T10:30:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2050.codfw.wmnet 2416588 wikitext text/x-wiki amp2uv3izyr9gi2nmnt8d9v9r1k73is 2416589 2416588 2026-05-20T10:30:52Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet 2416589 wikitext text/x-wiki 8tmyv6pdc2nhzne3oiioo6el6juccl3 2416590 2416589 2026-05-20T10:31:04Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2008.codfw.wmnet 2416590 wikitext text/x-wiki 6ms2f110kmhx6s7xmcvg2bjk5aiyr2e 2416591 2416590 2026-05-20T10:31:29Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2242: Migration of db2242.codfw.wmnet completed 2416591 wikitext text/x-wiki q9owth3pz9nk3zzuzkzqjsj7cgzlztn 2416592 2416591 2026-05-20T10:31:31Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2416592 wikitext text/x-wiki 0z8dpkhnbe9ubyknibcsy1ghxc9ktic 2416593 2416592 2026-05-20T10:32:54Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1005.eqiad.wmnet 2416593 wikitext text/x-wiki 3d8vdzbkb3hr52xhfojlvy9ee4cptbr 2416594 2416593 2026-05-20T10:36:10Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet 2416594 wikitext text/x-wiki f5ru0rgip917v03410k2625oeowwx5a 2416595 2416594 2026-05-20T10:36:12Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2008.codfw.wmnet 2416595 wikitext text/x-wiki 6bklho5sxq4mumg6kbkyq9t2oao5cge 2416596 2416595 2026-05-20T10:36:14Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2008.codfw.wmnet 2416596 wikitext text/x-wiki bitdz16goewwio5d9llh55tfudhe4eo 2416597 2416596 2026-05-20T10:36:19Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2009.codfw.wmnet 2416597 wikitext text/x-wiki t474imltsnsrjbh2as7nnl9kgmlkabl 2416598 2416597 2026-05-20T10:37:00Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2416598 wikitext text/x-wiki kf5jlqftaiu2srm7cb5eznwszs1zied 2416599 2416598 2026-05-20T10:39:22Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2416599 wikitext text/x-wiki o5rp63izg06okzz07fo9jq7yp0g7oq3 2416600 2416599 2026-05-20T10:39:44Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2162: Upgrading db2162.codfw.wmnet 2416600 wikitext text/x-wiki thigpabquig4zziqkommgvorte61el9 2416601 2416600 2026-05-20T10:40:14Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2162: Upgrading db2162.codfw.wmnet 2416601 wikitext text/x-wiki gkvl1wrsn10guhfzdmp8a6tcnrxe1uu 2416602 2416601 2026-05-20T10:40:58Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb1015.eqiad.wmnet with reason: host reimage 2416602 wikitext text/x-wiki a5lxh5c5g5xx5q4x4v96avbc54j6rhw 2416603 2416602 2026-05-20T10:41:24Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2009.codfw.wmnet 2416603 wikitext text/x-wiki 4qll77quwopdyegeerlkarhpioaaeqq 2416604 2416603 2026-05-20T10:41:31Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet 2416604 wikitext text/x-wiki ho8x9lxuzzsy9kkk6e6as7711wpweb4 2416605 2416604 2026-05-20T10:41:38Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2050.codfw.wmnet 2416605 wikitext text/x-wiki 4gnxls8yu7ih4c75fw43tm8ui6fxy52 2416606 2416605 2026-05-20T10:42:21Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2162.codfw.wmnet with OS trixie 2416606 wikitext text/x-wiki ajxpxcm5qnncqu1il4um1vab8zvxxec 2416607 2416606 2026-05-20T10:44:44Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1015.eqiad.wmnet with reason: host reimage 2416607 wikitext text/x-wiki beejmdhx03nbpot7glcvz61w4dyga5q 2416608 2416607 2026-05-20T10:46:07Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2009.codfw.wmnet 2416608 wikitext text/x-wiki reiffx55eoolbp81jw23rn4njfbd11c 2416609 2416608 2026-05-20T10:46:09Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2009.codfw.wmnet 2416609 wikitext text/x-wiki ghdwn6ng6ypw02ksiofsbe1cx1uyxbz 2416612 2416609 2026-05-20T10:46:21Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2010.codfw.wmnet 2416612 wikitext text/x-wiki e8mbzlv3joiikx548xanyy4umzy1o1n 2416614 2416612 2026-05-20T10:51:24Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2010.codfw.wmnet 2416614 wikitext text/x-wiki 5e9zyhb23e3uv3itaroijx9c0ggq586 2416615 2416614 2026-05-20T10:51:36Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2416615 wikitext text/x-wiki 8c84yuhuybtq54xl4hyxzcq26lup74f 2416616 2416615 2026-05-20T10:51:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet 2416616 wikitext text/x-wiki nzfsp98pqcdrrae7n9hrzmv57wtyqak 2416617 2416616 2026-05-20T10:51:40Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet 2416617 wikitext text/x-wiki c2x9ezi0b5lxv4mlj8nx36h2f27vg9j 2416618 2416617 2026-05-20T10:51:46Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet 2416618 wikitext text/x-wiki czdzexqehghg0fx8qpx8e14b63yrof2 2416620 2416618 2026-05-20T10:53:47Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1006.eqiad.wmnet 2416620 wikitext text/x-wiki 4t382hlsdsgnhyh84w0dtxjqyj12d0o 2416621 2416620 2026-05-20T10:55:59Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2010.codfw.wmnet 2416621 wikitext text/x-wiki e0jxt8xmzlha3b6kmm38qcs8xe0iepz 2416622 2416621 2026-05-20T10:56:01Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2010.codfw.wmnet 2416622 wikitext text/x-wiki lx91w2z6l4etzzlwk7xtx4e7t78f7la 2416623 2416622 2026-05-20T10:56:08Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2011.codfw.wmnet 2416623 wikitext text/x-wiki m4ccqhx7b3ruherq5i7889xzk6aiz0h 2416625 2416623 2026-05-20T10:57:31Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.presto.reboot-workers (exit_code=0) for Presto an-presto cluster: Reboot Presto nodes 2416625 wikitext text/x-wiki dueikiitbslb0nfhfz2bcgzo39opvno 2416626 2416625 2026-05-20T10:59:46Z Stashbot 7414 moritzm: failover Ganeti cluster in codfw to ganeti2048 2416626 wikitext text/x-wiki gtcb1jzhr3d7cdy2i5cz53jz0i2pbmb 2416627 2416626 2026-05-20T11:00:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2162.codfw.wmnet with reason: host reimage 2416627 wikitext text/x-wiki t1nsk7rcl37jcmxnxytaob8yhq0eunh 2416628 2416627 2026-05-20T11:01:17Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2011.codfw.wmnet 2416628 wikitext text/x-wiki q9aslc4ggnih5wmeb0dr6za03f6maa6 2416629 2416628 2026-05-20T11:01:32Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1010.eqiad.wmnet with OS trixie 2416629 wikitext text/x-wiki aq5wzii7qaysz4xu9r7isziy72yoxwi 2416630 2416629 2026-05-20T11:01:34Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw 2416630 wikitext text/x-wiki ozr4uccs9k7xdubjvv4d1ds1fgka3as 2416631 2416630 2026-05-20T11:04:28Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes 2416631 wikitext text/x-wiki 273d6x9pff5dgqq3qs1h9vy7xi4t23g 2416632 2416631 2026-05-20T11:04:59Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2162.codfw.wmnet with reason: host reimage 2416632 wikitext text/x-wiki iotvvgeto63a1odymacpcuynsop5vax 2416633 2416632 2026-05-20T11:05:01Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad 2416633 wikitext text/x-wiki skxlbyqpxue4q2cg1rem0dk0agsl28m 2416634 2416633 2026-05-20T11:05:34Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1016.eqiad.wmnet with OS trixie 2416634 wikitext text/x-wiki qzwrb3cumg1c8h2r51r394g31sqd74u 2416635 2416634 2026-05-20T11:05:42Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2011.codfw.wmnet 2416635 wikitext text/x-wiki 2v0fb83mkweamwrcbi3b9ml3ei3caat 2416636 2416635 2026-05-20T11:05:43Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2011.codfw.wmnet 2416636 wikitext text/x-wiki ffvwryiymgcltg8gtt55r2wrg4lfp6l 2416637 2416636 2026-05-20T11:05:46Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-codfw 2416637 wikitext text/x-wiki mo28vhkps2wxtq49cgaygz6v6ein5mq 2416638 2416637 2026-05-20T11:05:54Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{wikikube-worker[1249-1289,1291-1327,1375-1384].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2416638 wikitext text/x-wiki aj272ywfh395v6a83zhy59rx52nrm01 2416639 2416638 2026-05-20T11:05:58Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7014.magru.wmnet 2416639 wikitext text/x-wiki km15j6dpot4nry2yx6e60op93i9dfa3 2416640 2416639 2026-05-20T11:06:00Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp701[3-4].magru.wmnet} and A:cp 2416640 wikitext text/x-wiki 1eivvw8oshyyplig8hy27lpq4fzkfmv 2416641 2416640 2026-05-20T11:06:03Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1249-1252].eqiad.wmnet 2416641 wikitext text/x-wiki dnz5aiwqwp0vuov5rie1zgzk2t127zn 2416642 2416641 2026-05-20T11:06:20Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp700[7-8].magru.wmnet} and A:cp 2416642 wikitext text/x-wiki 7hv3ra1gbfrlkvgxm96jtqjbrbaykhg 2416643 2416642 2026-05-20T11:08:18Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1249-1252].eqiad.wmnet 2416643 wikitext text/x-wiki gohod9vnzobdcza0lxh8wll5zwtsi5u 2416644 2416643 2026-05-20T11:11:20Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1055.eqiad.wmnet 2416644 wikitext text/x-wiki 9f8b9k5vplh18uh152y8zcpby9f4qnh 2416645 2416644 2026-05-20T11:12:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet 2416645 wikitext text/x-wiki qk0lml1vpxz6hfd1flf2gpvs0mqhmns 2416646 2416645 2026-05-20T11:12:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet 2416646 wikitext text/x-wiki as6jisgqjbsrtmp1scg0e7owo5eqz4m 2416647 2416646 2026-05-20T11:12:46Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet 2416647 wikitext text/x-wiki i4rrvcb95md3ldjk10sua4g4aa5240u 2416648 2416647 2026-05-20T11:13:35Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2047.codfw.wmnet 2416648 wikitext text/x-wiki ry99fimvenkg3x7bthhnjfhv7irtoko 2416650 2416648 2026-05-20T11:16:13Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1010.eqiad.wmnet with reason: host reimage 2416650 wikitext text/x-wiki a4glletwz0stgacksc32pnlr8x7e3gv 2416651 2416650 2026-05-20T11:16:20Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1055.eqiad.wmnet 2416651 wikitext text/x-wiki n7rsliaufu1hd4ls4ur0wz3z27r68cq 2416652 2416651 2026-05-20T11:16:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2047.codfw.wmnet 2416652 wikitext text/x-wiki mrvw0owoxd2pog6rjxymh6c99bvn1z8 2416655 2416652 2026-05-20T11:17:22Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage 2416655 wikitext text/x-wiki lrmqe8fnnae266bqe2g7a2cxnsjj2gi 2416656 2416655 2026-05-20T11:17:57Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7007.magru.wmnet 2416656 wikitext text/x-wiki lleepzbiny49kt99l3wuot89ee3k81x 2416657 2416656 2026-05-20T11:18:15Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1055 2416657 wikitext text/x-wiki 9rpznxmigw3672o182ce74ptfctakwm 2416658 2416657 2026-05-20T11:18:48Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1249-1252].eqiad.wmnet 2416658 wikitext text/x-wiki e525pk4yb0178vi8otc6v7h5bbm5pnn 2416659 2416658 2026-05-20T11:18:51Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1249-1252].eqiad.wmnet 2416659 wikitext text/x-wiki k7dmhfotg9ardslp07qixhu6z5vzcqh 2416660 2416659 2026-05-20T11:19:00Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1253-1256].eqiad.wmnet 2416660 wikitext text/x-wiki 6s4yz9lsb8v1golf4p89yya2s1wmati 2416661 2416660 2026-05-20T11:19:30Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1055 2416661 wikitext text/x-wiki 5dmjw369jgtr0o0y5n0y1wiyzihjg8g 2416662 2416661 2026-05-20T11:19:58Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1010.eqiad.wmnet with reason: host reimage 2416662 wikitext text/x-wiki r0ust3hi3q5e8c5hm8u22t79lcu6s93 2416663 2416662 2026-05-20T11:20:11Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1055.eqiad.wmnet to cluster codfw and group A 2416663 wikitext text/x-wiki 7t7dy4azr90v3aa8swtmwinfm20nb91 2416664 2416663 2026-05-20T11:20:21Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1055.eqiad.wmnet to cluster codfw and group A 2416664 wikitext text/x-wiki 1q31mg9ae7mmgqjtcn2uuwpof2i551g 2416665 2416664 2026-05-20T11:21:12Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1253-1256].eqiad.wmnet 2416665 wikitext text/x-wiki 8tnjmjqji36gw8r4lpe94g4dfhm6pr9 2416667 2416665 2026-05-20T11:21:54Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2047.codfw.wmnet 2416667 wikitext text/x-wiki bjvj2wdztc9zlr1wy3ipayc0y1yepqx 2416669 2416667 2026-05-20T11:22:01Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2047.codfw.wmnet 2416669 wikitext text/x-wiki ns8d5cedcj9ydz7nq6r2birhh7f8a0m 2416670 2416669 2026-05-20T11:22:20Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2162.codfw.wmnet with OS trixie 2416670 wikitext text/x-wiki h7anctathhxm7hmi648evlshx2l8nyz 2416671 2416670 2026-05-20T11:23:28Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage 2416671 wikitext text/x-wiki lntfbpwjd34rv9vanaob1t9go4oodbx 2416674 2416671 2026-05-20T11:24:29Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2162: Migration of db2162.codfw.wmnet completed 2416674 wikitext text/x-wiki kaa7rbhoyzz8kdutzez08gpr26c73n1 2416676 2416674 2026-05-20T11:25:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1055.eqiad.wmnet to cluster codfw and group A 2416676 wikitext text/x-wiki 25zeyqjqxj8htu9azv05gyrzdst8r6x 2416677 2416676 2026-05-20T11:25:59Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1055.eqiad.wmnet to cluster codfw and group A 2416677 wikitext text/x-wiki ddu8ghufjvj073v7034nmqw9bz7gr0w 2416681 2416677 2026-05-20T11:28:53Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1253-1256].eqiad.wmnet 2416681 wikitext text/x-wiki 5tt5b1mmz3i5fvhs76mb86sb4sch28x 2416682 2416681 2026-05-20T11:28:56Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1253-1256].eqiad.wmnet 2416682 wikitext text/x-wiki l3rf71ycpcf0qewhl2l9mj0hwe2bqlc 2416683 2416682 2026-05-20T11:29:05Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1257-1260].eqiad.wmnet 2416683 wikitext text/x-wiki 37jxank0u6ty0215eh8qb2qma2azhqm 2416685 2416683 2026-05-20T11:31:17Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1257-1260].eqiad.wmnet 2416685 wikitext text/x-wiki 2b6gbkekt91t2c90phwumrrmgjbgtva 2416686 2416685 2026-05-20T11:32:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2035.codfw.wmnet 2416686 wikitext text/x-wiki kdnyqvrs84fgzbcu3dwcucxn3vrhg0k 2416687 2416686 2026-05-20T11:35:15Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2035.codfw.wmnet 2416687 wikitext text/x-wiki fe1av60xlna5njxne30osq8zs1w3q9p 2416688 2416687 2026-05-20T11:38:40Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1257-1260].eqiad.wmnet 2416688 wikitext text/x-wiki 7x99k6fbklclcw7pltxqat8gm4354ux 2416689 2416688 2026-05-20T11:38:43Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1257-1260].eqiad.wmnet 2416689 wikitext text/x-wiki afrupo76i7ido7x3uh041lmo0o8ee76 2416690 2416689 2026-05-20T11:38:52Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1261-1264].eqiad.wmnet 2416690 wikitext text/x-wiki t8u8kpfw685krkmeevnfvrk8fvhe8eq 2416691 2416690 2026-05-20T11:39:36Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1010.eqiad.wmnet with OS trixie 2416691 wikitext text/x-wiki hxyxngjsvwp6ww33iyoqc1xmkbu6rc7 2416692 2416691 2026-05-20T11:39:38Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2416692 wikitext text/x-wiki 6m0zsgcju2uiq28ysq5fpgb9uhey7gj 2416694 2416692 2026-05-20T11:40:07Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2416694 wikitext text/x-wiki 6mofinq2zy3pf4oy4wkk4lpd9kbl692 2416695 2416694 2026-05-20T11:40:10Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1016.eqiad.wmnet with OS trixie 2416695 wikitext text/x-wiki 1iwmeko50ayymrh84ez6io0hrk62e2q 2416696 2416695 2026-05-20T11:40:37Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2035.codfw.wmnet 2416696 wikitext text/x-wiki imycw4xp1pmtrxp74l6up281ss7ijiv 2416697 2416696 2026-05-20T11:40:44Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2035.codfw.wmnet 2416697 wikitext text/x-wiki 14k2g3ifoi91karbid1jcsk08frz7p0 2416698 2416697 2026-05-20T11:40:56Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2036.codfw.wmnet 2416698 wikitext text/x-wiki 888k56aidr0oskfwsckh4mlihiqsggl 2416699 2416698 2026-05-20T11:41:09Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1261-1264].eqiad.wmnet 2416699 wikitext text/x-wiki l2jeafww3773fux8urxl51o8rvbrg5y 2416700 2416699 2026-05-20T11:42:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2036.codfw.wmnet 2416700 wikitext text/x-wiki a0upagyrdm0ikd8rhm6028i9y8zbc34 2416701 2416700 2026-05-20T11:42:51Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet 2416701 wikitext text/x-wiki m6rjq8ii76tv7w028rog7pjdgdbmfjf 2416702 2416701 2026-05-20T11:43:06Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2416702 wikitext text/x-wiki mkegii06ac4wsxuuhqsvgb7r4x6fsop 2416703 2416702 2026-05-20T11:43:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1258: Upgrading db1258.eqiad.wmnet 2416703 wikitext text/x-wiki 3fzirio1iyipm7g2oj52lg9lx05xr8i 2416704 2416703 2026-05-20T11:43:56Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1258: Upgrading db1258.eqiad.wmnet 2416704 wikitext text/x-wiki rejekr0kt5e92q6lqixvn65neyrcren 2416705 2416704 2026-05-20T11:47:54Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2036.codfw.wmnet 2416705 wikitext text/x-wiki l03euhcfspte7o876hf0nyhg9y6cz3f 2416706 2416705 2026-05-20T11:48:01Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2036.codfw.wmnet 2416706 wikitext text/x-wiki le5w2j6li07ss9nnvhr1ntckuud29c9 2416707 2416706 2026-05-20T11:48:11Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2037.codfw.wmnet 2416707 wikitext text/x-wiki 2cm8a9qmg0r26cohe6bugf8n1wwpdqc 2416708 2416707 2026-05-20T11:49:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet 2416708 wikitext text/x-wiki p9qr1mal67m99pvm84phf2dxibcexm9 2416709 2416708 2026-05-20T11:49:52Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet 2416709 wikitext text/x-wiki 814drnryanfc19wvgy9mf0u878iylyr 2416711 2416709 2026-05-20T11:50:01Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet 2416711 wikitext text/x-wiki ipj2lffzzg0sy3pjw7kameqd31u180n 2416712 2416711 2026-05-20T11:51:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1261-1264].eqiad.wmnet 2416712 wikitext text/x-wiki 85flcwyezdwg311grbjrti2s1eiekd3 2416713 2416712 2026-05-20T11:51:23Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1261-1264].eqiad.wmnet 2416713 wikitext text/x-wiki f7kpcxe88v3i045tuwqahnnblpcorzb 2416714 2416713 2026-05-20T11:51:33Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1265-1268].eqiad.wmnet 2416714 wikitext text/x-wiki m03cm5ondo3wx034qoaposspd1wkvc2 2416715 2416714 2026-05-20T11:51:36Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1289928|Fix UserGroupManager::getUserAutopromoteGroups with interwiki users (T426832)]] 2416715 wikitext text/x-wiki 9z0nozzu4vb7e3gjbrabixmrzx50zn1 2416716 2416715 2026-05-20T11:52:00Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1008.eqiad.wmnet 2416716 wikitext text/x-wiki gyj5y682p1v4yuted9z1ssn1owv0ht0 2416717 2416716 2026-05-20T11:52:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2037.codfw.wmnet 2416717 wikitext text/x-wiki r40774ldcb5f5smn4zwmmz2tfuls6mq 2416718 2416717 2026-05-20T11:53:34Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1289928|Fix UserGroupManager::getUserAutopromoteGroups with interwiki users (T426832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416718 wikitext text/x-wiki eehmyfx9nn0tv0g96w7c0xwbef073jx 2416720 2416718 2026-05-20T11:54:22Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1265-1268].eqiad.wmnet 2416720 wikitext text/x-wiki ew2klsy4zolf3x4dyelikbsz9d2rz10 2416722 2416720 2026-05-20T11:54:27Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2416722 wikitext text/x-wiki 2olsw6orqs3ckea84lj45vs2tk1ne4i 2416723 2416722 2026-05-20T11:55:48Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2416723 wikitext text/x-wiki jlmaid65nwwy6cw0ddrzau33n9r87lz 2416724 2416723 2026-05-20T11:55:54Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2416724 wikitext text/x-wiki r6iy7owmegmvz1hu8pq3pqok24j7e8v 2416725 2416724 2026-05-20T11:55:56Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2416725 wikitext text/x-wiki 5lap0nxxdn51958v6myjzhvvpuc4w4p 2416727 2416725 2026-05-20T11:58:10Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2037.codfw.wmnet 2416727 wikitext text/x-wiki 7xwfr8ddwqa0wittm7vxmr8kaegat6a 2416728 2416727 2026-05-20T11:58:17Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2037.codfw.wmnet 2416728 wikitext text/x-wiki 16psfdzq8lr9c4vxwn2uflp99ebxrx7 2416729 2416728 2026-05-20T11:58:40Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289928|Fix UserGroupManager::getUserAutopromoteGroups with interwiki users (T426832)]] (duration: 07m 04s) 2416729 wikitext text/x-wiki qn0l89h4w52873o00m1y8vahj5yghjk 116 15998 2416278 2132326 2026-05-19T21:27:49Z BryanDavis 1604 update git repo 2416278 wikitext text/x-wiki 7vujfpt80un7nyql1mnecjczd7iw2bl 2416279 2416278 2026-05-19T21:30:09Z BryanDavis 1604 update phab 2416279 wikitext text/x-wiki cvcfph2ja1b3tbuor4b8fbzkx7gdhy2 0 17290 2416128 2415133 2026-05-19T18:07:52Z Stashbot 7414 James_F: Zuul: [mediawiki/libs/ZestJQ] Add basic PHP and Node CI 2416128 wikitext text/x-wiki dxnjwqujjb1roqb6279cfqpsfrnlsh3 498 30942 2416118 2415619 2026-05-19T17:58:03Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416118 wikitext text/x-wiki l1ayb5avsxlqea8bk35jiloquncwud0 2416119 2416118 2026-05-19T17:58:08Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416119 wikitext text/x-wiki bse4gj98315fekcebfu48sdcui4xhnm 2416120 2416119 2026-05-19T17:58:11Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416120 wikitext text/x-wiki s5sbqzgg1ihh81twj1xquiad3hwzbyc 2416143 2416120 2026-05-19T18:42:44Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416143 wikitext text/x-wiki i2edguqg2drylwgiqg219fv7tlomplk 2416146 2416143 2026-05-19T18:48:12Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416146 wikitext text/x-wiki s4zorihovce20usi5svle0iog3rzkku 2416288 2416146 2026-05-19T21:51:30Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'P{O:wmcs::openstack::codfw1dev::control}' 2416288 wikitext text/x-wiki a7zyokvz601cu57wf4fnr2wtpt9mxgl 2416292 2416288 2026-05-19T22:03:20Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::codfw1dev::control}' 2416292 wikitext text/x-wiki 5193zak4bzdsiysryna2lm2hts89nkq 2416294 2416292 2026-05-19T22:09:52Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service} AND NOT P{F:kernelversion = 6.12.88}' 2416294 wikitext text/x-wiki 83v9cjxq16rs6i8nuco0uava6vxsjtx 2416299 2416294 2026-05-19T22:56:51Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service} AND NOT P{F:kernelversion = 6.12.88}' 2416299 wikitext text/x-wiki dr6xn2o681koldfm8yfr1tnmjkvlh4a 2416338 2416299 2026-05-20T01:26:22Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416338 wikitext text/x-wiki akqlzj5i5u7o5hkjw45n64lgw44h8zm 2416342 2416338 2026-05-20T01:33:54Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416342 wikitext text/x-wiki j04isg9uuw4t51ib0d4ehqietaxztbt 2416610 2416342 2026-05-20T10:46:14Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'A:cloudlb AND A:CODFW' 2416610 wikitext text/x-wiki qjsh6ab1d7pqx0w7dto6ez9qmu5vnme 2416611 2416610 2026-05-20T10:46:18Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=99) on hosts matched by 'A:cloudlb AND A:CODFW' 2416611 wikitext text/x-wiki cvd6xgnw5yxq71q1drnbfdrzg4ycxih 2416613 2416611 2026-05-20T10:46:56Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'A:cloudlb AND A:codfw' 2416613 wikitext text/x-wiki 1k7phwacdwuc0keqmzl1s5jo8097vvx 2416619 2416613 2026-05-20T10:53:43Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=99) on hosts matched by 'A:cloudlb AND A:codfw' 2416619 wikitext text/x-wiki 4wl4byfzd3v5k7woqkltjwchjm7mno9 2416624 2416619 2026-05-20T10:56:16Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'A:cloudlb AND A:codfw' 2416624 wikitext text/x-wiki foikjne6zw9h6evcurmdxuyyfrbgrx7 2416649 2416624 2026-05-20T11:15:44Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=0) on hosts matched by 'A:cloudlb AND A:codfw' 2416649 wikitext text/x-wiki 3mk2pxti6s4mt05nnbw2eryu41ktdo1 2416653 2416649 2026-05-20T11:16:29Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'A:cloudlb AND A:eqiad' 2416653 wikitext text/x-wiki 70gvg4q67czoe7qsqwj49wotn3ln10u 2416680 2416653 2026-05-20T11:28:10Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=0) on hosts matched by 'A:cloudlb AND A:eqiad' 2416680 wikitext text/x-wiki 7awt4mnscbm3kd6k0ccdl9tnwcht47i 2416710 2416680 2026-05-20T11:49:57Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'D{cloudlb2002-dev.codfw.wmnet}' 2416710 wikitext text/x-wiki kkvirzitzoupqh627j17vdffgg1r862 2416726 2416710 2026-05-20T11:56:29Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=0) on hosts matched by 'D{cloudlb2002-dev.codfw.wmnet}' 2416726 wikitext text/x-wiki plrd4271avjwbtutorqvuh4hc66exyi 116 67609 2416289 2163318 2026-05-19T21:55:12Z BryanDavis 1604 update git repo; update lede 2416289 wikitext text/x-wiki 6s1hif55yikyb5tn3faripixdbeyswb 498 73236 2416448 2409156 2026-05-20T08:20:56Z Stashbot 7414 wmbot~m7@tools-bastion-15: stewardbots/StewardBot/manage.sh restart # Disconnected 2416448 wikitext text/x-wiki ho6r7ax8k4h2g2o90ek8eh8pdy5gtqq 116 81869 2416275 2371924 2026-05-19T21:22:52Z BryanDavis 1604 update git repo 2416275 wikitext text/x-wiki cybgso40iyr6n9z3itmngaok87qy6zj 0 440900 2415760 2361333 2026-05-19T13:29:11Z Taavi-WMF 41365 /* cloudvirtXXXX */ exclude-kernel 2415760 wikitext text/x-wiki hvo7zni6wsfz23jktj2nwo1slcscxa4 2415763 2415760 2026-05-19T13:29:37Z Taavi-WMF 41365 /* cloudvirtXXXX */ oops, should have read the next section 2415763 wikitext text/x-wiki gt3ylh6govs9eopw5puori9etjh09pq 2416672 2415763 2026-05-20T11:23:45Z Taavi-WMF 41365 /* cloudvirt-wdqsXXXX */ cloudvirt-wdqs no longer exists 2416672 wikitext text/x-wiki 7alsly5dk3hb05itl0feni5ulj9mvsz 2416675 2416672 2026-05-20T11:25:46Z Taavi-WMF 41365 cloudlb, assuming [[gerrit:1289914]] gets merged 2416675 wikitext text/x-wiki 7ey5rjk27bghh4j1ttu4zr2rgcscxeh 0 442642 2415655 2289639 2026-05-19T12:18:00Z Krinkle 44 Refresh! 2415655 wikitext text/x-wiki 4fs4cm2aldxua2vpj2q38f10n1deols 12 443694 2416354 2381016 2026-05-20T03:00:08Z Star walker 40724 Tidied up help page 2416354 wikitext text/x-wiki q78pj8rz0yxvsyjqzkuspbs8260ecab 498 443946 2416113 2405916 2026-05-19T17:52:58Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 80583945a8 (migrate to OAuth 2) 2416113 wikitext text/x-wiki envo4tjoq521q7e4a8vdgtlbl8sajhf 2416121 2416113 2026-05-19T17:59:36Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 43b18b8299 (prevent OAuthLib InsecureTransportError more strongly) 2416121 wikitext text/x-wiki fhckjqa4xqtm99xo2nslwgtxd14zjg6 498 447660 2416403 2193282 2026-05-20T07:25:49Z Samwilson 1287 Update repo URLs. T420317. 2416403 wikitext text/x-wiki qzgez6btsgtu5msl1rlfi52rwxz9nbx 0 447862 2416488 2415371 2026-05-20T09:13:09Z FCeratto-WMF 39104 Updated as per https://gitlab.wikimedia.org/repos/data_persistence/dbtools/scripts/-/merge_requests/15 2416488 wikitext text/x-wiki lh3t7j79u1kkcza9y47jjer857vn9yy 0 449160 2415878 2415245 2026-05-19T14:36:55Z Dexbot 30554 Bot: Updating the report 2415878 wikitext text/x-wiki n4lllhohddilhw3sqy7ofi9uew2ntgz 2416312 2415878 2026-05-20T00:02:00Z Dexbot 30554 Bot: Updating the report 2416312 wikitext text/x-wiki 8ikgag9kt6rwvj61eu9jdxu0bx37k8k 0 451180 2416236 2401059 2026-05-19T20:24:56Z BCornwall-WMF 31521 /* Headers */ Promote headings to appropriate levels 2416236 wikitext text/x-wiki tkk758n7u8u28k1k2bns49eg324mrft 2416237 2416236 2026-05-19T20:26:19Z BCornwall-WMF 31521 /* Headers */ Remove orphaned definition indentations 2416237 wikitext text/x-wiki kt92zxqlk4k6wf6dfatq2vn9uj4c7ww 2416282 2416237 2026-05-19T21:35:28Z BCornwall-WMF 31521 /* Headers */ Fix broken example newlines 2416282 wikitext text/x-wiki alxpgnm1xo3vlp7sozk7cl0qd1wyiz0 2416290 2416282 2026-05-19T21:55:44Z BCornwall-WMF 31521 /* Headers */ Clean up headings, move them to tables, and unify examples layouts. 2416290 wikitext text/x-wiki si46sex5ujnjzdejok3qv96w5tym2ru 2416291 2416290 2026-05-19T21:55:57Z BCornwall-WMF 31521 rm extra newlines 2416291 wikitext text/x-wiki b7z4y46gd52vxxqhz77m6cvse90zhkr 116 453906 2416335 2415167 2026-05-20T01:21:14Z Gitlabaccountapprovalbot 37332 beedellrokejulianlockhart was rejected. 2416335 wikitext text/x-wiki 0mrdp03ja5kvgxoj4a1a5nrprdrmdp3 6 455428 2416351 2211330 2026-05-20T02:19:46Z Star walker 40724 now commons 2416351 wikitext text/x-wiki 0pe3kadplz92gbb019nxcuy3pfcsczk 4 458614 2416353 2394416 2026-05-20T02:27:37Z Star walker 40724 internal link 2416353 wikitext text/x-wiki ig9tle8ax4gjbxcr5ge6mqo4odnywke 4 458615 2416352 2412762 2026-05-20T02:26:07Z Star walker 40724 slash trick 2416352 wikitext text/x-wiki io459jsp37i6vcjo8is90y64wjvftpy 0 459560 2416719 2404923 2026-05-20T11:53:37Z DAdedoyin (WMF) 39150 Test 2416719 wikitext text/x-wiki if0qe76ifjlmxg4tqcvtckkfzyggnox 2416721 2416719 2026-05-20T11:54:23Z DAdedoyin (WMF) 39150 /* 2026 */ Change Sprint H end date 2416721 wikitext text/x-wiki 1675rv13jvypwnqmkqqi50vvbcos1hq 0 459699 2416301 2415196 2026-05-19T23:08:12Z Stashbot 7414 eileen: civicrm upgraded from 6cff86c1 to fcbbf763 2416301 wikitext text/x-wiki qukoyx0x6n5gs5lu0gnhrua1cj5z9tj 0 459922 2416166 2389711 2026-05-19T19:17:36Z JKilani-WMF 37651 Sophroid bootstrap documentation 2416166 wikitext text/x-wiki 3g4iu9djc6ehhgtv1x98p9fubmu3sul 2416167 2416166 2026-05-19T19:18:43Z JKilani-WMF 37651 2416167 wikitext text/x-wiki 0nxa4guv4ovnx7rausokcbng04ywoka 2416168 2416167 2026-05-19T19:20:16Z JKilani-WMF 37651 2416168 wikitext text/x-wiki 5menqmlewfe9fr0cmj5fdvymuctny4w 2416302 2416168 2026-05-19T23:15:11Z JKilani-WMF 37651 Language for clarity 2416302 wikitext text/x-wiki nzxbe8gbcxpicvc5hvhtosh5s0yizaz 2416303 2416302 2026-05-19T23:16:10Z JKilani-WMF 37651 Language for clarity 2416303 wikitext text/x-wiki j40otll74cjje4hqf2dplvmp58fze0z 2416304 2416303 2026-05-19T23:18:42Z JKilani-WMF 37651 Language for clarity 2416304 wikitext text/x-wiki 8jv6xk8f7e37qs8ojc6oaxxj8jee8q0 2416306 2416304 2026-05-19T23:21:45Z JKilani-WMF 37651 Language for clarity, hyperlinks 2416306 wikitext text/x-wiki mtqwqh1421b9xc9enxjsyvxlloh6q84 2416307 2416306 2026-05-19T23:25:12Z JKilani-WMF 37651 Language for clarity 2416307 wikitext text/x-wiki 35tiwkiqoal2oleur0m2nkc2kei6u2t 2416308 2416307 2026-05-19T23:26:37Z JKilani-WMF 37651 Language for clarity 2416308 wikitext text/x-wiki qa2coq8fm17lbrtvfjnfdyovhmle7d5 0 460089 2415827 2407726 2026-05-19T14:08:51Z EEvans (WMF) 5035 /* Logging */ link to dash 2415827 wikitext text/x-wiki fr0vcowwex8qdsul561vi66g0qr9k1p 2415834 2415827 2026-05-19T14:12:01Z EEvans (WMF) 5035 link to metrics dash 2415834 wikitext text/x-wiki rqdfqc3mmc2h4pf0piq2snccjdqly03 2 460097 2416654 2414248 2026-05-20T11:16:36Z Effie Mouzeli (WMF) 12880 2416654 wikitext text/x-wiki qj1kszrmxacrm6m2zxhjmtgx4prabcd 2 460100 2415660 2415535 2026-05-19T12:19:04Z Effie Mouzeli (WMF) 12880 2415660 wikitext text/x-wiki ncslu9i1e87n2fi8ci2jrftihc6729i 2415678 2415660 2026-05-19T12:31:52Z Effie Mouzeli (WMF) 12880 2415678 wikitext text/x-wiki 33zesk1ye05eetavmhvje4vckzcgkvr 2415699 2415678 2026-05-19T12:45:04Z Effie Mouzeli (WMF) 12880 2415699 wikitext text/x-wiki gksle4pahcvp5b38ivgkbjmetvcr7rz 2415700 2415699 2026-05-19T12:45:26Z Effie Mouzeli (WMF) 12880 2415700 wikitext text/x-wiki qy07mfcb2bqo7f7b9hhgz6jf76ks730 2 460176 2415714 2414394 2026-05-19T12:59:33Z Effie Mouzeli (WMF) 12880 2415714 wikitext text/x-wiki 9huckmgq5a74tmr13l5yuan264fmx3h 2415715 2415714 2026-05-19T13:02:03Z Effie Mouzeli (WMF) 12880 2415715 wikitext text/x-wiki emnahn8urkm6sf9xe8u0sdz4dqn02n5 0 460178 2415640 2414276 2026-05-19T12:10:44Z Effie Mouzeli (WMF) 12880 2415640 wikitext text/x-wiki c5ytrfqqummbypy0tk4vmkgo5uwcllo 2415654 2415640 2026-05-19T12:17:30Z Effie Mouzeli (WMF) 12880 /* 1. Service Summary */ 2415654 wikitext text/x-wiki isj0ut22jfu2n86ixg7m48aodvrrat0 2415665 2415654 2026-05-19T12:22:50Z Effie Mouzeli (WMF) 12880 2415665 wikitext text/x-wiki sg9f1ho5vagmjc0e6ut7m8jimju7noj 2415681 2415665 2026-05-19T12:32:20Z Effie Mouzeli (WMF) 12880 2415681 wikitext text/x-wiki 4hdr87esv14uk0e14qyu79bgmdudevd 2415702 2415681 2026-05-19T12:46:26Z Effie Mouzeli (WMF) 12880 2415702 wikitext text/x-wiki t799a1zfcgskd24crlwagmtrnj5g6qt 1 460190 2416293 2026-05-19T22:04:09Z BCornwall-WMF 31521 /* Proposal to merge into CDN/Headers */ new section 2416293 wikitext text/x-wiki jhc8m8zav9y9c6k08tpbbr21w6rd1js 2416300 2416293 2026-05-19T23:04:50Z Neil Shah-Quinn (WMF) 36601 /* Proposal to merge into CDN/Headers */ Reply 2416300 wikitext text/x-wiki ik432vwnp3jzhhlka76739w4chvqwwa 0 460191 2416666 2026-05-20T11:21:22Z Effie Mouzeli (WMF) 12880 Created page with "[[File:Books_and_Scroll_Ornament_with_Open_Book.png|right|frameless|150px|alt=A scroll]] Welcome to '''SCROLL''', the production readiness checklist for Wikimedia services! SCROLL stands for '''S'''ervice '''C'''hecklist for '''R'''eadiness, '''O'''perations, '''L'''aunch and '''L'''ifecycle. It is a shared checklist that walks Owners and SRE through what a component needs before it goes live, from documentation and monitoring through to traffic and reliability. Think..." 2416666 wikitext text/x-wiki iqxltuko3oporfd3xyjnjwz3zs3294o 2416678 2416666 2026-05-20T11:26:52Z Effie Mouzeli (WMF) 12880 2416678 wikitext text/x-wiki 0of9k8slxi7ixzwhgzwur1jmbcepysz 2416684 2416678 2026-05-20T11:31:15Z Effie Mouzeli (WMF) 12880 2416684 wikitext text/x-wiki 244yic8lp1oo8ch9wkruwvk6vdxe4fi 2416693 2416684 2026-05-20T11:39:46Z Effie Mouzeli (WMF) 12880 2416693 wikitext text/x-wiki luarrsmwwsenumgby7h6v7nlabnco1a 0 460192 2416668 2026-05-20T11:22:01Z Effie Mouzeli (WMF) 12880 Created page with "{{Draft}} {{DISPLAYTITLE:SCROLL: {{SUBPAGENAME}}}} <templatestyles src="User:Effie_Mouzeli_(WMF)/SCROLL/styles.css"/> ''Service Checklist for Readiness, Operations, Launch and Lifecycle'' {| style="width: 100%; border: none; background: transparent;" | style="vertical-align: top; width: 38%; padding-right: 2%;" | {| class="wikitable" style="font-size: 90%; width: 100%; border: 1px solid #a2a9b1;" |- ! colspan="2" class="scroll-card-header" | 📜 Service/Component iden..." 2416668 wikitext text/x-wiki qy07mfcb2bqo7f7b9hhgz6jf76ks730 0 460193 2416673 2026-05-20T11:23:52Z Effie Mouzeli (WMF) 12880 Redirected page to [[SCROLL]] 2416673 wikitext text/x-wiki fsq82bhs5gum5319tsigfosiq3l1taf 0 460194 2416679 2026-05-20T11:27:43Z Effie Mouzeli (WMF) 12880 Created page with "{{Draft}} <templatestyles src="User:Effie_Mouzeli_(WMF)/SCROLL/styles.css"/> = SCROLL guide = ''Scrolling to production: a starter guide for humans.'' This page explains how a service moves from "we want to ship this" to "running in production with users", using the SCROLL (''Service Checklist for Readiness, Operations, Launch and Lifecycle'') process. If you are about to start your first SCROLL, read this page end-to-end. If you have done one before, skip to #The c..." 2416679 wikitext text/x-wiki emnahn8urkm6sf9xe8u0sdz4dqn02n5