Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.46.0-wmf.24 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 1516 2406178 2398034 2026-04-27T12:22:51Z Stashbot 7414 Krinkle: /cs flags #cvn-meta NathanVeritas voiced 2406178 wikitext text/x-wiki 72sc9nka5nxc9uj21hbtnpobbrns0k3 498 3086 2406771 2405339 2026-04-28T10:44:26Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2406771 wikitext text/x-wiki skynk82cqapiuuto4xi4nc52z7x9n30 2406777 2406771 2026-04-28T10:57:16Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2406777 wikitext text/x-wiki 33u2u9q6f8rfc0tb4gfe0afcyijjvqe 0 4108 2406392 2406158 2026-04-27T15:51:23Z JKilani-WMF 37651 Redis host reboots deployment 2406392 wikitext text/x-wiki 1dtuexmvbee8dl7argp78896pjadl08 2406452 2406392 2026-04-27T17:25:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1271215]] to Wednesday, April 29 UTC afternoon backport window 2406452 wikitext text/x-wiki gc6v08hwwx3l2sa2dvzdkljp5mbrhu7 2406457 2406452 2026-04-27T17:44:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1277656]] to Monday, April 27 UTC late backport window 2406457 wikitext text/x-wiki 5rz1b5w2xkxdiejyvqdlw24z3zbnj56 2406467 2406457 2026-04-27T18:02:21Z Duesentrieb 52342 /* Tuesday, April 28 */ Daniel deploying API rate limits, [[phab:T417779]] 2406467 wikitext text/x-wiki su8sdu0yofwjbx12f5apsgjjdes3zad 2406772 2406467 2026-04-28T10:44:59Z ScheduleDeploymentBot 37566 Add [[gerrit:1278381]] to Tuesday, April 28 UTC afternoon backport window 2406772 wikitext text/x-wiki 7zps1or33l59tatc0iu0qj7iqirbsne 498 6726 2406761 2405114 2026-04-28T10:32:37Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2406761 wikitext text/x-wiki 8calk215yybyk5uh6f9xnrnxqffsx2s 2406770 2406761 2026-04-28T10:44:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2406770 wikitext text/x-wiki qap3lah3y1ki2r45riw6usqnjrb77m0 0 7919 2406165 2406164 2026-04-27T12:03:55Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406165 wikitext text/x-wiki m97xui6q3dne2r66xfvq44owxd18g3q 2406166 2406165 2026-04-27T12:04:15Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406166 wikitext text/x-wiki nm0imzlxecylachzld6k9x19jw1khi6 2406167 2406166 2026-04-27T12:04:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P91586 and previous config saved to /var/cache/conftool/dbconfig/20260427-120453-fceratto.json 2406167 wikitext text/x-wiki h04dp0ayyx7o70kjpf96p5xxha7kyg9 2406169 2406167 2026-04-27T12:10:30Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2406169 wikitext text/x-wiki cky4xlf1ikxx9yp9iyn678n8yd5o2ic 2406170 2406169 2026-04-27T12:12:31Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1210: after reimage to trixie 2406170 wikitext text/x-wiki aj4ffg93atb96n80tw84x06a5op87ho 2406171 2406170 2026-04-27T12:15:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T419635)', diff saved to https://phabricator.wikimedia.org/P91588 and previous config saved to /var/cache/conftool/dbconfig/20260427-121501-fceratto.json 2406171 wikitext text/x-wiki qpzvk7w62u4a9dgl63p3q1s4vpaeydl 2406172 2406171 2026-04-27T12:15:22Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1200.eqiad.wmnet with reason: Maintenance 2406172 wikitext text/x-wiki 9ky7r3yc5z8kg6ftfcm7dnqkywgymdk 2406173 2406172 2026-04-27T12:15:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 (T419635)', diff saved to https://phabricator.wikimedia.org/P91589 and previous config saved to /var/cache/conftool/dbconfig/20260427-121529-fceratto.json 2406173 wikitext text/x-wiki 7w82pzbq8uq2tlhdty1iqes6s8d85x6 2406174 2406173 2026-04-27T12:15:55Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2157: after reimage to trixie 2406174 wikitext text/x-wiki g6i4c5a8lni606acxiq21xq9se9nazk 2406175 2406174 2026-04-27T12:17:41Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts testvm2002.codfw.wmnet 2406175 wikitext text/x-wiki bikccxffsd9xgwknczo8schkd3hn741 2406176 2406175 2026-04-27T12:22:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T419635)', diff saved to https://phabricator.wikimedia.org/P91591 and previous config saved to /var/cache/conftool/dbconfig/20260427-122213-fceratto.json 2406176 wikitext text/x-wiki tvg4o9hifsad47yrsxmdnnj92n9l5uv 2406177 2406176 2026-04-27T12:22:27Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2406177 wikitext text/x-wiki rudivfdcg1q4j4u1bdd27e26cy4zarj 2406179 2406177 2026-04-27T12:26:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406179 wikitext text/x-wiki 065vsy2rm6empspcizcij1p6osd8evu 2406180 2406179 2026-04-27T12:27:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406180 wikitext text/x-wiki jvelkrttk7nhjmxsyrviz33k410gttk 2406181 2406180 2026-04-27T12:27:07Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406181 wikitext text/x-wiki j9u3munwtiigyktdxdukxnaoxfjrb78 2406182 2406181 2026-04-27T12:27:09Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2002.codfw.wmnet 2406182 wikitext text/x-wiki pq7mm91g2mfaiuqodxlab8q4o0c1iig 2406183 2406182 2026-04-27T12:30:48Z Stashbot 7414 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406183 wikitext text/x-wiki 3j0obq7ho5s160k4ern5taz2koszskw 2406184 2406183 2026-04-27T12:32:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P91592 and previous config saved to /var/cache/conftool/dbconfig/20260427-123221-fceratto.json 2406184 wikitext text/x-wiki piz2tenieqhkyqedcjkuqxdq3572xo2 2406185 2406184 2026-04-27T12:32:30Z Stashbot 7414 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406185 wikitext text/x-wiki lfy3lsqsm0jcifxck5fnz6t7uyi8m57 2406186 2406185 2026-04-27T12:35:32Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2211.codfw.wmnet with reason: Reimage to Trixie 2406186 wikitext text/x-wiki ny72pmsnn6rl8cozf7o9kd0zxaiqmqw 2406187 2406186 2026-04-27T12:35:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2211: Reimage to Trixie 2406187 wikitext text/x-wiki a7phaazaz7ku7u8o0lobqq1mwt8sv5f 2406188 2406187 2026-04-27T12:35:40Z Stashbot 7414 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406188 wikitext text/x-wiki or74pzpyteylgi2lj0s7a7pdipeuqax 2406189 2406188 2026-04-27T12:35:53Z Stashbot 7414 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406189 wikitext text/x-wiki 8w1n4ezz4qf8c0cassibx80hef70n97 2406190 2406189 2026-04-27T12:35:56Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2211: Reimage to Trixie 2406190 wikitext text/x-wiki ayezvwx4o37bruj1cet8mcqx1gnmefg 2406191 2406190 2026-04-27T12:37:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2211.codfw.wmnet with OS trixie 2406191 wikitext text/x-wiki fpuh3ww5vr44ktzvnjbnhtfddzcy9at 2406192 2406191 2026-04-27T12:37:30Z Stashbot 7414 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Upgrading mw-mcrouter - effie (duration: 164m 14s) 2406192 wikitext text/x-wiki h3ug7m3wjjzz81bxv7x8lrfv4ih5wxu 2406194 2406192 2026-04-27T12:42:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P91594 and previous config saved to /var/cache/conftool/dbconfig/20260427-124228-fceratto.json 2406194 wikitext text/x-wiki jnezhbfyir0nqguj3sfscedlu5dqsm3 2406196 2406194 2026-04-27T12:52:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T419635)', diff saved to https://phabricator.wikimedia.org/P91595 and previous config saved to /var/cache/conftool/dbconfig/20260427-125236-fceratto.json 2406196 wikitext text/x-wiki m6hi7jbtb47f36tvqkbft4tnrvtw5lk 2406197 2406196 2026-04-27T12:52:56Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance 2406197 wikitext text/x-wiki ifbgk9hh81ueyh50kmom7xbjqq77psv 2406198 2406197 2026-04-27T12:53:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 (T419635)', diff saved to https://phabricator.wikimedia.org/P91596 and previous config saved to /var/cache/conftool/dbconfig/20260427-125301-fceratto.json 2406198 wikitext text/x-wiki 25j994i900fg9xjs2mib8j1qy0grhrd 2406199 2406198 2026-04-27T12:55:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2211.codfw.wmnet with reason: host reimage 2406199 wikitext text/x-wiki lcnv9ryx1zcw85abiw9me29oyo7s29c 2406200 2406199 2026-04-27T12:58:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T419635)', diff saved to https://phabricator.wikimedia.org/P91597 and previous config saved to /var/cache/conftool/dbconfig/20260427-125834-fceratto.json 2406200 wikitext text/x-wiki 6dveig0nmqslakeyyy4srh99r64ini6 2406201 2406200 2026-04-27T13:01:01Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2211.codfw.wmnet with reason: host reimage 2406201 wikitext text/x-wiki onezah32ph2scibvcgaml5scdt47nh9 2406202 2406201 2026-04-27T13:03:17Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1277499|missing.php: Fix Wikiversity logo and improve dark mode (T424511 T424512)]] 2406202 wikitext text/x-wiki 6x9n38eaq5ly5nm0zxab2hmkv7p61ik 2406203 2406202 2026-04-27T13:07:00Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1277499|missing.php: Fix Wikiversity logo and improve dark mode (T424511 T424512)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406203 wikitext text/x-wiki p39wt5ayx6yv75cxsf18htm7chx0bsk 2406204 2406203 2026-04-27T13:07:50Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Continuing with deployment 2406204 wikitext text/x-wiki 6n5vb0mfhn5t687r196w9jgmhwtcs9g 2406205 2406204 2026-04-27T13:08:26Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance 2406205 wikitext text/x-wiki 1cp314gi33ady0tlpb1wnr3uj65qcbp 2406206 2406205 2026-04-27T13:08:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1247 (T419961)', diff saved to https://phabricator.wikimedia.org/P91598 and previous config saved to /var/cache/conftool/dbconfig/20260427-130832-fceratto.json 2406206 wikitext text/x-wiki 7hhmjnpl9rlz2z1ij7am7v362to4ayp 2406207 2406206 2026-04-27T13:08:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P91599 and previous config saved to /var/cache/conftool/dbconfig/20260427-130842-fceratto.json 2406207 wikitext text/x-wiki o247ydcjrbmpl0cl4el55l3l8dq3q0o 2406209 2406207 2026-04-27T13:10:02Z Stashbot 7414 moritzm: updating debdeploy on bullseye to 0.0.99.15 2406209 wikitext text/x-wiki tsqs8gg1fpe1ed6qpu998mtyoj3gm95 2406210 2406209 2026-04-27T13:12:17Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply 2406210 wikitext text/x-wiki b5n1s2lv848rdflugwyujlyvp42zv99 2406211 2406210 2026-04-27T13:12:49Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply 2406211 wikitext text/x-wiki k1gjg1qpss9u8tnxxs8s970lpw5sluk 2406212 2406211 2026-04-27T13:13:57Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277499|missing.php: Fix Wikiversity logo and improve dark mode (T424511 T424512)]] (duration: 10m 39s) 2406212 wikitext text/x-wiki kj7ar6ng4r87qxngvbep1spkvjxeeny 2406213 2406212 2026-04-27T13:17:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T419961)', diff saved to https://phabricator.wikimedia.org/P91600 and previous config saved to /var/cache/conftool/dbconfig/20260427-131702-fceratto.json 2406213 wikitext text/x-wiki pzlfgjk7br7knobwbgha2pcp2yzogrx 2406214 2406213 2026-04-27T13:18:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P91601 and previous config saved to /var/cache/conftool/dbconfig/20260427-131849-fceratto.json 2406214 wikitext text/x-wiki 7qbsnp9ojrxwrvbdo8mekvqm2ra80jb 2406215 2406214 2026-04-27T13:19:13Z Stashbot 7414 moritzm: installing Bind security updates (client-side tools/libs) 2406215 wikitext text/x-wiki to8kaxhvxlyv11t5qr71eq7te8pbpkx 2406217 2406215 2026-04-27T13:23:20Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1277445|stats(CreateAccount): record baseline data for opening rates (T419916)]], [[gerrit:1277446|stats(CreateAccount): ignore overridden experiments (T419916)]], [[gerrit:1277456|bundlesize: unset mediaiwiki.base max size (T424324)]], [[gerrit:1277500|tests: Clear SiteNoticeAfter hook on SkinMinervaTest]] 2406217 wikitext text/x-wiki donpoosmh2vjr0dih5z7psjfg92i2qc 2406218 2406217 2026-04-27T13:23:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2211.codfw.wmnet with OS trixie 2406218 wikitext text/x-wiki la7aw6uljnrv5mq9gpwcea8m79f4k6l 2406219 2406218 2026-04-27T13:25:40Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1277445|stats(CreateAccount): record baseline data for opening rates (T419916)]], [[gerrit:1277446|stats(CreateAccount): ignore overridden experiments (T419916)]], [[gerrit:1277456|bundlesize: unset mediaiwiki.base max size (T424324)]], [[gerrit:1277500|tests: Clear SiteNoticeAfter hook on SkinMinervaTest]] synced to the testservers ( 2406219 wikitext text/x-wiki eyagkwpx5j04aci1gwkcr46dj85r48x 2406220 2406219 2026-04-27T13:25:42Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2211: after reimage to trixie 2406220 wikitext text/x-wiki 9hn18qckpsvmqtpotcf4zptjd6r86tu 2406221 2406220 2026-04-27T13:27:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P91603 and previous config saved to /var/cache/conftool/dbconfig/20260427-132710-fceratto.json 2406221 wikitext text/x-wiki 55lbuf7eenak4hyxpfuqk7xorer28kt 2406222 2406221 2026-04-27T13:28:45Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment 2406222 wikitext text/x-wiki lrh42q95zjlkqwe8mmhqfghfk5zsfhi 2406223 2406222 2026-04-27T13:29:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T419635)', diff saved to https://phabricator.wikimedia.org/P91604 and previous config saved to /var/cache/conftool/dbconfig/20260427-132857-fceratto.json 2406223 wikitext text/x-wiki 118yaouz969mb1mefq2ft42s9ja40ni 2406224 2406223 2026-04-27T13:29:15Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1210.eqiad.wmnet with reason: Maintenance 2406224 wikitext text/x-wiki 7by37ddb0xj2f2mzwcl4z3gu8g6yu8w 2406225 2406224 2026-04-27T13:29:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 (T419635)', diff saved to https://phabricator.wikimedia.org/P91605 and previous config saved to /var/cache/conftool/dbconfig/20260427-132921-fceratto.json 2406225 wikitext text/x-wiki c5rxes1w5y2dbxofblbbn1kwp06lbut 2406226 2406225 2026-04-27T13:33:00Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277445|stats(CreateAccount): record baseline data for opening rates (T419916)]], [[gerrit:1277446|stats(CreateAccount): ignore overridden experiments (T419916)]], [[gerrit:1277456|bundlesize: unset mediaiwiki.base max size (T424324)]], [[gerrit:1277500|tests: Clear SiteNoticeAfter hook on SkinMinervaTest]] (duration: 09m 40s) 2406226 wikitext text/x-wiki f6wytwbluwi4mxlf3109cat740srgap 2406227 2406226 2026-04-27T13:37:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260427-133719-fceratto.json 2406227 wikitext text/x-wiki dvzcj8t37b1azwbd0v1mh2oa322mcaq 2406228 2406227 2026-04-27T13:39:42Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reimage for host phab2003.codfw.wmnet with OS bullseye 2406228 wikitext text/x-wiki cqm3smoh7xkd8dp1uffzbc1j0kr7avb 2406230 2406228 2026-04-27T13:46:44Z Stashbot 7414 esanders@deploy1003: Started scap sync-world: Backport for [[gerrit:1277496|Add experiment for suggestion mode beta feature (T422740)]] 2406230 wikitext text/x-wiki q0aigyumx9l99wpbvwcbqr5f6scbbwo 2406231 2406230 2026-04-27T13:47:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T419961)', diff saved to https://phabricator.wikimedia.org/P91607 and previous config saved to /var/cache/conftool/dbconfig/20260427-134732-fceratto.json 2406231 wikitext text/x-wiki njfp0o5qv2oi61yv2gt701qgevdumkd 2406232 2406231 2026-04-27T13:47:53Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance 2406232 wikitext text/x-wiki l1t7ni58i72e0x3gqszks9e0r3f4xat 2406233 2406232 2026-04-27T13:48:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1248 (T419961)', diff saved to https://phabricator.wikimedia.org/P91608 and previous config saved to /var/cache/conftool/dbconfig/20260427-134800-fceratto.json 2406233 wikitext text/x-wiki fnt5as6atb9jxecjss35jrxi8z5kobk 2406234 2406233 2026-04-27T13:48:18Z Stashbot 7414 esanders@deploy1003: esanders: Backport for [[gerrit:1277496|Add experiment for suggestion mode beta feature (T422740)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406234 wikitext text/x-wiki l0b0mxqq8g729pqogfun43m0pk6f0mt 2406235 2406234 2026-04-27T13:50:24Z Stashbot 7414 esanders@deploy1003: esanders: Continuing with deployment 2406235 wikitext text/x-wiki ctza7rp3mtu55wzaqkem1yxwnp4hf66 2406236 2406235 2026-04-27T13:54:11Z Stashbot 7414 esanders@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277496|Add experiment for suggestion mode beta feature (T422740)]] (duration: 07m 26s) 2406236 wikitext text/x-wiki eadkflowjvgpngp2p8sbeofxbcd6j5b 2406237 2406236 2026-04-27T13:55:29Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2406237 wikitext text/x-wiki 5wmy1fcvk7yr9rzf8fril5enxn40b64 2406238 2406237 2026-04-27T13:56:01Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2406238 wikitext text/x-wiki t2v4p4wbohf92pqs1n2yzh96v26j8of 2406239 2406238 2026-04-27T13:56:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T419961)', diff saved to https://phabricator.wikimedia.org/P91610 and previous config saved to /var/cache/conftool/dbconfig/20260427-135622-fceratto.json 2406239 wikitext text/x-wiki lao2u6jnl11s3jg2lumhrz2dqeuaeon 2406240 2406239 2026-04-27T13:56:42Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply 2406240 wikitext text/x-wiki iysuhnz0rh0jsjv0gh4xryuwaw6duo0 2406241 2406240 2026-04-27T13:57:24Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply 2406241 wikitext text/x-wiki 0kbahw9sc7b1hw47nb88ci7gbonpubt 2406242 2406241 2026-04-27T14:03:03Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2406242 wikitext text/x-wiki izb1esp8l8bicqqpyw7dznlh3zohs31 2406243 2406242 2026-04-27T14:03:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91611 and previous config saved to /var/cache/conftool/dbconfig/20260427-140309-fceratto.json 2406243 wikitext text/x-wiki 75zp3bzzb3l39tlpq6ve3xb0pca5ni6 2406244 2406243 2026-04-27T14:04:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1159: Repooling 2406244 wikitext text/x-wiki dercrahpo16yb38un1xlsofqveshn9v 2406245 2406244 2026-04-27T14:06:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P91613 and previous config saved to /var/cache/conftool/dbconfig/20260427-140630-fceratto.json 2406245 wikitext text/x-wiki 8oue2vnt8niv5lm0lwjta4gqgv59hwm 2406246 2406245 2026-04-27T14:06:39Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1277457|hCaptcha: Emit load_duration once per load and add load_attempts (T421204)]] 2406246 wikitext text/x-wiki rhux4zfrn5b2ya4ixdwin7jwz8t8dc4 2406247 2406246 2026-04-27T14:08:15Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1277457|hCaptcha: Emit load_duration once per load and add load_attempts (T421204)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406247 wikitext text/x-wiki 8zydw8rg69d9g48vjrlvifs1oiljrpu 2406248 2406247 2026-04-27T14:08:36Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply 2406248 wikitext text/x-wiki cwc8ojm90rvyq0w5o1qngvf746221oi 2406249 2406248 2026-04-27T14:09:10Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply 2406249 wikitext text/x-wiki 339k3ou1lh7y0jdujky8njkbulrqa4w 2406250 2406249 2026-04-27T14:09:32Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2406250 wikitext text/x-wiki iskk99vhx1wmmhq7d6v4g32npx8gbps 2406251 2406250 2026-04-27T14:09:47Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2406251 wikitext text/x-wiki bbwxex52ah9ostf18bn8l4gxt7u2vaz 2406252 2406251 2026-04-27T14:10:21Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2406252 wikitext text/x-wiki kp36fw63fkd6mevq7u0dj5nztg3fmww 2406253 2406252 2026-04-27T14:11:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2211: after reimage to trixie 2406253 wikitext text/x-wiki 0ktn43cmc0ngqvq1dfagge02282z6kv 2406254 2406253 2026-04-27T14:12:51Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: dse-k8s-master-codfw@codfw 2406254 wikitext text/x-wiki gbwy2bfjfa1c9k5dtad7yvka4udbg55 2406255 2406254 2026-04-27T14:13:25Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277457|hCaptcha: Emit load_duration once per load and add load_attempts (T421204)]] (duration: 06m 46s) 2406255 wikitext text/x-wiki ea00u445hi21ltt6k832lrsh4edjhqd 2406256 2406255 2026-04-27T14:14:14Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1091.eqiad.wmnet with OS bullseye 2406256 wikitext text/x-wiki 26m78hsufqoij3kago600m1hy76c7h3 2406257 2406256 2026-04-27T14:14:42Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1091 2406257 wikitext text/x-wiki 7guj2duevuvb8j8wgfyage1ofn2vzoe 2406258 2406257 2026-04-27T14:14:53Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2406258 wikitext text/x-wiki kulb65bvo78f1xo2l8b1txlrhavb9ua 2406259 2406258 2026-04-27T14:16:37Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2406259 wikitext text/x-wiki gn7m99i7pte0g3p0ctmf02qs4uitu7x 2406260 2406259 2026-04-27T14:16:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P91615 and previous config saved to /var/cache/conftool/dbconfig/20260427-141638-fceratto.json 2406260 wikitext text/x-wiki i8tabs6bltjxj82i6bu27rva9a3fqh9 2406261 2406260 2026-04-27T14:17:08Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2406261 wikitext text/x-wiki lh6e9kx3mbbp8ck1i4scvpnq6ts71bo 2406262 2406261 2026-04-27T14:17:31Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: dse-k8s-master-codfw@codfw 2406262 wikitext text/x-wiki d7lr0km9b17del6e4p2aucgwzrpz3li 2406263 2406262 2026-04-27T14:17:38Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1277426|wmgPrivilegedGroups/wmgPrivilegedGlobalGroups: Update to include temporary account IP viewers]], [[gerrit:1277455|hCaptcha: enable for mobile apps account creation on testwiki (T412132)]] 2406263 wikitext text/x-wiki c0xk79r90lbk1y1efjjbr1ggmzwhyps 2406264 2406263 2026-04-27T14:17:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: dse-k8s-master-codfw@codfw 2406264 wikitext text/x-wiki o2mav9zlzz4fcpkz0kx0xhvnr2a02h2 2406265 2406264 2026-04-27T14:18:10Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2406265 wikitext text/x-wiki 9c28s6ft9c0ybzesibh4642fs7nei79 2406266 2406265 2026-04-27T14:18:43Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2406266 wikitext text/x-wiki dqeb7jrp7g71ddil9dv7spynleusae3 2406267 2406266 2026-04-27T14:18:46Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406267 wikitext text/x-wiki llkpx88z0pdr1xmmztr78qutsnvkvet 2406268 2406267 2026-04-27T14:19:16Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1277426|wmgPrivilegedGroups/wmgPrivilegedGlobalGroups: Update to include temporary account IP viewers]], [[gerrit:1277455|hCaptcha: enable for mobile apps account creation on testwiki (T412132)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406268 wikitext text/x-wiki grp5ko4xtst01k1kclq5x29kx2yduu8 2406269 2406268 2026-04-27T14:19:29Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1159: Repooling 2406269 wikitext text/x-wiki nlwsiov1pwp9y48c0bbcktv3wrmcq9w 2406270 2406269 2026-04-27T14:20:08Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2406270 wikitext text/x-wiki ii4cn34g3pjuz4zhk4z930360bn6smf 2406271 2406270 2026-04-27T14:20:37Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406271 wikitext text/x-wiki a0xbyi143z9o30y2ik5y1wq3o1ft7g3 2406272 2406271 2026-04-27T14:22:52Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply 2406272 wikitext text/x-wiki bohqnkhr52oi7ewetk8g873cffyf1no 2406273 2406272 2026-04-27T14:23:23Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1091 - mvernon@cumin2002" 2406273 wikitext text/x-wiki 31c7b74x0ks0q5yzqphvnkmsrf2r7yg 2406274 2406273 2026-04-27T14:23:26Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply 2406274 wikitext text/x-wiki j0yeprig9ebyan89f40nq6hds8w96al 2406275 2406274 2026-04-27T14:23:35Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2406275 wikitext text/x-wiki hxusvvla83426ya0559wwteg3foitda 2406276 2406275 2026-04-27T14:24:00Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277426|wmgPrivilegedGroups/wmgPrivilegedGlobalGroups: Update to include temporary account IP viewers]], [[gerrit:1277455|hCaptcha: enable for mobile apps account creation on testwiki (T412132)]] (duration: 06m 23s) 2406276 wikitext text/x-wiki tqc3opy9581quusooa0yjaq6iats2to 2406277 2406276 2026-04-27T14:24:03Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1091 - mvernon@cumin2002" 2406277 wikitext text/x-wiki jgza64rt63x37x1xmhodlyt2xem82yz 2406278 2406277 2026-04-27T14:24:06Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406278 wikitext text/x-wiki pzytuh4m9lkf023nknoxeao3dxvmqao 2406279 2406278 2026-04-27T14:24:08Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1091.eqiad.wmnet 21.48.64.10.in-addr.arpa 1.2.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2406279 wikitext text/x-wiki dun1f3yzvc4hh2vja5u979v47s6zlng 2406280 2406279 2026-04-27T14:24:12Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1091.eqiad.wmnet 21.48.64.10.in-addr.arpa 1.2.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2406280 wikitext text/x-wiki os7xvirhp7c2l1ee3xg9yx7bei65zl0 2406281 2406280 2026-04-27T14:24:15Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1091 2406281 wikitext text/x-wiki sq4fm0w0n7kp4xkuwefjcub2mvqg2xv 2406282 2406281 2026-04-27T14:24:19Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2406282 wikitext text/x-wiki 2kmi389k6z1dsi4avpmityxcy0o8vp5 2406283 2406282 2026-04-27T14:24:21Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2406283 wikitext text/x-wiki bfzjdc058ezeye9dvdox8ue98qpcara 2406284 2406283 2026-04-27T14:24:33Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1091 2406284 wikitext text/x-wiki 1auhgpjr6p9aj3ls8aeehg65q4zkk38 2406285 2406284 2026-04-27T14:24:36Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1091 2406285 wikitext text/x-wiki s81lve36dmc1el68hcaxi3petwvk8aq 2406286 2406285 2026-04-27T14:25:23Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2406286 wikitext text/x-wiki 9ji4klvbume6um1l6y1haxiwh4o5iyz 2406287 2406286 2026-04-27T14:25:25Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: dse-k8s-master-codfw@codfw 2406287 wikitext text/x-wiki l6pwcc2j2f9t7pemhkrnulau6cr7cnu 2406288 2406287 2026-04-27T14:26:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T419961)', diff saved to https://phabricator.wikimedia.org/P91617 and previous config saved to /var/cache/conftool/dbconfig/20260427-142646-fceratto.json 2406288 wikitext text/x-wiki 0s7osptpd32n5tl7hkscafc4vmayvv9 2406289 2406288 2026-04-27T14:27:08Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1249.eqiad.wmnet with reason: Maintenance 2406289 wikitext text/x-wiki pt96y5kmp4edq1tl9l75hr72kfp1mjg 2406290 2406289 2026-04-27T14:27:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1249 (T419961)', diff saved to https://phabricator.wikimedia.org/P91618 and previous config saved to /var/cache/conftool/dbconfig/20260427-142714-fceratto.json 2406290 wikitext text/x-wiki ed52ql302du81zx04yuose3o28ul55x 2406291 2406290 2026-04-27T14:28:35Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply 2406291 wikitext text/x-wiki j8ui3hy9xgy51w0zmsg9v2xj57jzvzb 2406292 2406291 2026-04-27T14:29:07Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply 2406292 wikitext text/x-wiki 1efd18h92kiunm04wlgbflk0id8ed8t 2406293 2406292 2026-04-27T14:29:32Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2406293 wikitext text/x-wiki g9t4276s4dypym3y2fqtpobbsiqxe9w 2406295 2406293 2026-04-27T14:35:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T419961)', diff saved to https://phabricator.wikimedia.org/P91619 and previous config saved to /var/cache/conftool/dbconfig/20260427-143523-fceratto.json 2406295 wikitext text/x-wiki abl0jn8xlhap3zyvm4ln6v0pvbuqd10 2406296 2406295 2026-04-27T14:37:14Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1091.eqiad.wmnet with reason: host reimage 2406296 wikitext text/x-wiki 317v762z5qqcjgnjr7ecculuiige47o 2406298 2406296 2026-04-27T14:42:44Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1091.eqiad.wmnet with reason: host reimage 2406298 wikitext text/x-wiki lw9oj1auwlwfwgezhzq17uzzm5cr5ns 2406299 2406298 2026-04-27T14:45:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P91620 and previous config saved to /var/cache/conftool/dbconfig/20260427-144531-fceratto.json 2406299 wikitext text/x-wiki jmyyg6td40mdk0is90qp955rqmkc09p 2406300 2406299 2026-04-27T14:47:07Z Stashbot 7414 XioNoX: add gnmic 0.45 to bookworm-wikimedia - T416360 2406300 wikitext text/x-wiki aa63l29u1j75mko2r42xfiec73h6wyz 2406301 2406300 2026-04-27T14:47:11Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance 2406301 wikitext text/x-wiki ker6t7jxuf48uv4mvo5k9rqmyvhp7jc 2406302 2406301 2026-04-27T14:47:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 (T419635)', diff saved to https://phabricator.wikimedia.org/P91621 and previous config saved to /var/cache/conftool/dbconfig/20260427-144718-fceratto.json 2406302 wikitext text/x-wiki 9w3n8blqcym8ncwoea0t9766stet532 2406304 2406302 2026-04-27T14:49:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T419635)', diff saved to https://phabricator.wikimedia.org/P91622 and previous config saved to /var/cache/conftool/dbconfig/20260427-144949-fceratto.json 2406304 wikitext text/x-wiki e1tlm1f61nf4ubp1xj6fpwc1mt9g2dk 2406305 2406304 2026-04-27T14:50:48Z Stashbot 7414 XioNoX: add gnmic 0.45 to trixie-wikimedia - T416360 2406305 wikitext text/x-wiki i4ok75uoo4tkvktxwh1ffmd9q763ixk 2406306 2406305 2026-04-27T14:50:52Z Stashbot 7414 herron@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw 2406306 wikitext text/x-wiki 0zoeyhqegvke1gylei4jjf4b34egm2h 2406309 2406306 2026-04-27T14:51:50Z Stashbot 7414 aokoth@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host phab2003.codfw.wmnet with OS bullseye 2406309 wikitext text/x-wiki kc6y0w2fy64b3l2xu8gy58rcqp3gepa 2406310 2406309 2026-04-27T14:55:03Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reimage for host phab2003.codfw.wmnet with OS bullseye 2406310 wikitext text/x-wiki ah21zt502zkfd0hkkwn2jd0ymwlyhza 2406311 2406310 2026-04-27T14:55:11Z Stashbot 7414 XioNoX: upgrade gnmic on netflow7002 - T416360 2406311 wikitext text/x-wiki fs3p2qok59eiqdwml6yk8zf7mr92urp 2406312 2406311 2026-04-27T14:55:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P91623 and previous config saved to /var/cache/conftool/dbconfig/20260427-145539-fceratto.json 2406312 wikitext text/x-wiki h7b0f5fe7ah8u1c5lx8z66qv9xp5mns 2406313 2406312 2026-04-27T14:58:09Z Stashbot 7414 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: Upgrading mw-mcrouter - effie 2406313 wikitext text/x-wiki 62jk7rlx344m9cgbdvli2q8136d5tne 2406314 2406313 2026-04-27T14:59:45Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1091.eqiad.wmnet with OS bullseye 2406314 wikitext text/x-wiki ddn563kbrx3qxle0qz6rk5s463mi0tc 2406315 2406314 2026-04-27T14:59:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P91624 and previous config saved to /var/cache/conftool/dbconfig/20260427-145957-fceratto.json 2406315 wikitext text/x-wiki dq24voot5rtu48c6iwy2z3s5bbezo22 2406316 2406315 2026-04-27T15:00:24Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1092.eqiad.wmnet with OS bullseye 2406316 wikitext text/x-wiki 2kocv0washep2yh0ff9mfssc1kasi58 2406317 2406316 2026-04-27T15:00:56Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1092 2406317 wikitext text/x-wiki 6ij7wgqe5vogkn31n73pq55004q6an3 2406318 2406317 2026-04-27T15:02:45Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2406318 wikitext text/x-wiki swwrf56it6ssnm3ovy62ymmpvuu5ibu 2406319 2406318 2026-04-27T15:05:22Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406319 wikitext text/x-wiki jstxrs7qqz9ntpejutghgedsyelh0zg 2406320 2406319 2026-04-27T15:05:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T419961)', diff saved to https://phabricator.wikimedia.org/P91625 and previous config saved to /var/cache/conftool/dbconfig/20260427-150547-fceratto.json 2406320 wikitext text/x-wiki 7g5drog4l3053a0anzs9ibpvsrt834m 2406321 2406320 2026-04-27T15:06:09Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance 2406321 wikitext text/x-wiki bz99lgyxuryeotnkcna21zu11zv0fe4 2406322 2406321 2026-04-27T15:06:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1252 (T419961)', diff saved to https://phabricator.wikimedia.org/P91626 and previous config saved to /var/cache/conftool/dbconfig/20260427-150616-fceratto.json 2406322 wikitext text/x-wiki g03ovg1smqj81xpo12433pdgj4ehe41 2406323 2406322 2026-04-27T15:06:33Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1070.eqiad.wmnet with reason: vacuum overlarge container dbs 2406323 wikitext text/x-wiki 353gojhu1ji284khiy8ttort23rzhkr 2406325 2406323 2026-04-27T15:07:12Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406325 wikitext text/x-wiki sf8cizg0yw8f39qdy607ln7tgw34ryb 2406326 2406325 2026-04-27T15:07:18Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1092 - mvernon@cumin2002" 2406326 wikitext text/x-wiki 394gohqasn1lbv4mbf925dndawkl9k0 2406327 2406326 2026-04-27T15:07:23Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1092 - mvernon@cumin2002" 2406327 wikitext text/x-wiki dgue51pfg4af453rg813d3n8h0nic7t 2406328 2406327 2026-04-27T15:07:26Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406328 wikitext text/x-wiki ht8axhn91mdvt0qd0g0saqwyrhdl7zg 2406329 2406328 2026-04-27T15:07:28Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1092.eqiad.wmnet 32.32.64.10.in-addr.arpa 2.3.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2406329 wikitext text/x-wiki owa3y550bev6zhqebyqp0h05c17gc6r 2406330 2406329 2026-04-27T15:07:31Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1092.eqiad.wmnet 32.32.64.10.in-addr.arpa 2.3.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2406330 wikitext text/x-wiki 8xd5nf7i4t3yzlhoe8lm5pin3epypnb 2406331 2406330 2026-04-27T15:07:34Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1092 2406331 wikitext text/x-wiki qkslxcxg2kqi3ptkiqjfoqk6xqrhrhu 2406332 2406331 2026-04-27T15:07:36Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2406332 wikitext text/x-wiki 4yhgew8qzxyjs9vxignxbm4yedeap4d 2406333 2406332 2026-04-27T15:07:39Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406333 wikitext text/x-wiki jjx335444dh51lq14s2rvo182lg8wfp 2406334 2406333 2026-04-27T15:07:41Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1010.eqiad.wmnet with OS trixie 2406334 wikitext text/x-wiki 2rv2oddltwwckju8cgs7r1mjv0zhyjq 2406335 2406334 2026-04-27T15:07:55Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1092 2406335 wikitext text/x-wiki 1ij99piu61m2dmi0x676uabd5z0hj5k 2406336 2406335 2026-04-27T15:07:58Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1092 2406336 wikitext text/x-wiki ifwuar0xmswtgkepafv69fbqq01cdk7 2406337 2406336 2026-04-27T15:08:13Z Stashbot 7414 herron@cumin1003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw 2406337 wikitext text/x-wiki 6iecs7ftowhbhovhzvox2yldf6096zy 2406338 2406337 2026-04-27T15:08:48Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2406338 wikitext text/x-wiki 6kh3kwv1h2tkq2ag1bwbip6ygeoqcht 2406339 2406338 2026-04-27T15:09:24Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406339 wikitext text/x-wiki awyw8ibroeowpz0v30f4po5u6wzii2k 2406340 2406339 2026-04-27T15:10:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P91627 and previous config saved to /var/cache/conftool/dbconfig/20260427-151005-fceratto.json 2406340 wikitext text/x-wiki g4f26zcgx2wkddtljqy1w77mvfzrsyf 2406341 2406340 2026-04-27T15:10:52Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2406341 wikitext text/x-wiki j4gk0odvk5z64q5nybv74gr6x8rj773 2406342 2406341 2026-04-27T15:11:02Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2406342 wikitext text/x-wiki 8ikkj62u52iarnxqwp86epu2dr5mqlx 2406343 2406342 2026-04-27T15:12:50Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2406343 wikitext text/x-wiki 8qtub28d7izq2w2qw05i38fbk1ajba0 2406344 2406343 2026-04-27T15:15:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T419961)', diff saved to https://phabricator.wikimedia.org/P91629 and previous config saved to /var/cache/conftool/dbconfig/20260427-151528-fceratto.json 2406344 wikitext text/x-wiki kov2bfk47xtjwutep2i9wkq4o1p5s2u 2406345 2406344 2026-04-27T15:20:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T419635)', diff saved to https://phabricator.wikimedia.org/P91630 and previous config saved to /var/cache/conftool/dbconfig/20260427-152013-fceratto.json 2406345 wikitext text/x-wiki ou7l0y1j1ohvk9b0n8fav1sk39t9t1g 2406346 2406345 2026-04-27T15:20:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance 2406346 wikitext text/x-wiki 1k87iwdae2sftaj3ei583okc3hj4yc4 2406347 2406346 2026-04-27T15:20:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 (T419635)', diff saved to https://phabricator.wikimedia.org/P91631 and previous config saved to /var/cache/conftool/dbconfig/20260427-152038-fceratto.json 2406347 wikitext text/x-wiki ssykctgwji7pwj2eiu9678dnkc75bhy 2406349 2406347 2026-04-27T15:23:48Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406349 wikitext text/x-wiki hoywffdg77f0de4r37xhv2n9ml6sz9l 2406350 2406349 2026-04-27T15:25:37Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406350 wikitext text/x-wiki al468fznxces2nzuh24n1dyb082s5id 2406351 2406350 2026-04-27T15:25:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P91632 and previous config saved to /var/cache/conftool/dbconfig/20260427-152536-fceratto.json 2406351 wikitext text/x-wiki 9menjxq0b94jz0r6w2ymxc3t0w5jt1e 2406352 2406351 2026-04-27T15:25:45Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406352 wikitext text/x-wiki 5c4s96av4s4uftv96546afdc6s4w72v 2406353 2406352 2026-04-27T15:26:02Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1010.eqiad.wmnet with reason: host reimage 2406353 wikitext text/x-wiki 9an2dc34tqyvledpfw1psz8t063d3zv 2406354 2406353 2026-04-27T15:26:31Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply 2406354 wikitext text/x-wiki 1yfm642ewtkazby6ll1hffn4341vcbf 2406355 2406354 2026-04-27T15:26:35Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply 2406355 wikitext text/x-wiki o72av7i7cuaab9npxvmy3gt5kdeei31 2406356 2406355 2026-04-27T15:26:55Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply 2406356 wikitext text/x-wiki q8a4i9efk60jz31ssnxlembwlupk286 2406357 2406356 2026-04-27T15:27:02Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply 2406357 wikitext text/x-wiki gevejux8b0dfey78f7ahwzet3x2i2ua 2406358 2406357 2026-04-27T15:27:38Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406358 wikitext text/x-wiki a8w69ceew8s94chjiw7z42i4hm4ibmq 2406359 2406358 2026-04-27T15:29:47Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage 2406359 wikitext text/x-wiki klan8d3h8usd3lrhx5by3enjop5uyix 2406360 2406359 2026-04-27T15:30:08Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406360 wikitext text/x-wiki s4m34q1d4m3jm2fnhejx0o2psdqhgua 2406361 2406360 2026-04-27T15:30:15Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406361 wikitext text/x-wiki 1egedx66qy27r32akpbvi09c5pp0bof 2406362 2406361 2026-04-27T15:30:36Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406362 wikitext text/x-wiki 9sm5mljfzoglnmc4szb8cgwoe20m6z5 2406363 2406362 2026-04-27T15:30:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406363 wikitext text/x-wiki 9gj4r3bl1au0s1wchj4000f0oigc7td 2406364 2406363 2026-04-27T15:31:46Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1010.eqiad.wmnet with reason: host reimage 2406364 wikitext text/x-wiki 67mqc6h1jvjha5t5e7j4aichguv75dh 2406366 2406364 2026-04-27T15:35:21Z Stashbot 7414 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Upgrading mw-mcrouter - effie (duration: 37m 12s) 2406366 wikitext text/x-wiki hetj091epfnjl7bwse466bwr6vw48j9 2406367 2406366 2026-04-27T15:35:33Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406367 wikitext text/x-wiki mu8a46tti98ojr34s391a2yd8ri6j95 2406368 2406367 2026-04-27T15:35:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P91633 and previous config saved to /var/cache/conftool/dbconfig/20260427-153544-fceratto.json 2406368 wikitext text/x-wiki 6hcg7irfmguhnldpj94esyl4ykf0vp6 2406369 2406368 2026-04-27T15:36:20Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1092.eqiad.wmnet with reason: host reimage 2406369 wikitext text/x-wiki g31o5qcltezuzmgmsr68mq2qe7bnufz 2406370 2406369 2026-04-27T15:36:30Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.252.0" for 2 host(s) 2406370 wikitext text/x-wiki e4hqf1va7bz44pq4wg83uq4vg7fvwy3 2406371 2406370 2026-04-27T15:37:16Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406371 wikitext text/x-wiki jonxyro36b7civ6av801cntj9timm0x 2406372 2406371 2026-04-27T15:37:33Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406372 wikitext text/x-wiki 710is4ghpcjzhvkr67yf1mxevwqsoa8 2406373 2406372 2026-04-27T15:37:38Z Stashbot 7414 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 echo-subscriptions-email-page-review (T406724) 2406373 wikitext text/x-wiki a9vhkotwl5dfiv2xgpf7xrkos0wptce 2406374 2406373 2026-04-27T15:38:04Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406374 wikitext text/x-wiki hfbz3r765km3l2oe24v48tr9uhnued6 2406375 2406374 2026-04-27T15:38:22Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.252.0" completed for 2 hosts 2406375 wikitext text/x-wiki f8w98k5er2ppfzjh9baei5c3zalntg7 2406376 2406375 2026-04-27T15:38:39Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406376 wikitext text/x-wiki kn02qzwlx9zl7mtjq4febx4p9xruli3 2406377 2406376 2026-04-27T15:40:29Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406377 wikitext text/x-wiki oyxldhw3meqq7wpvgm51sz2uqc9m2u8 2406378 2406377 2026-04-27T15:42:36Z Stashbot 7414 aokoth@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host phab2003.codfw.wmnet with OS bullseye 2406378 wikitext text/x-wiki 5hfeplxldxhamkos07pcm0kaf4nipr4 2406379 2406378 2026-04-27T15:43:25Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reimage for host phab2003.codfw.wmnet with OS trixie 2406379 wikitext text/x-wiki o09ijpypledrirdwgleik7s5hcja12b 2406380 2406379 2026-04-27T15:44:18Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406380 wikitext text/x-wiki pxr172ylo9aec49fr0ae30nup3nhs77 2406381 2406380 2026-04-27T15:44:48Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1277572|Backwards compatibility for mw.testKitchen.compat (ironically) (T422740)]] 2406381 wikitext text/x-wiki rnid39vyea2e1iyw5v2tgjdycvaa1qe 2406382 2406381 2026-04-27T15:45:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T419961)', diff saved to https://phabricator.wikimedia.org/P91634 and previous config saved to /var/cache/conftool/dbconfig/20260427-154552-fceratto.json 2406382 wikitext text/x-wiki tc6a4rs9slbzzf3zb2p7hdrwc877w6r 2406383 2406382 2026-04-27T15:46:10Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406383 wikitext text/x-wiki i1sz81xote50vo689ex5swk6s0gsb5e 2406384 2406383 2026-04-27T15:46:14Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance 2406384 wikitext text/x-wiki ozc9f87neeooo3uqn9nz615fif3r3x6 2406385 2406384 2026-04-27T15:46:19Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.hosts.remove-downtime for ms-be1070.eqiad.wmnet 2406385 wikitext text/x-wiki l2wa58lrz46vaq74gy43yehyo5ygdqj 2406386 2406385 2026-04-27T15:46:21Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1070.eqiad.wmnet 2406386 wikitext text/x-wiki qp61meptjveg5k44vpvdbobka7zjov4 2406387 2406386 2026-04-27T15:46:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1260 (T419961)', diff saved to https://phabricator.wikimedia.org/P91635 and previous config saved to /var/cache/conftool/dbconfig/20260427-154620-fceratto.json 2406387 wikitext text/x-wiki c38s9epynmy12t6ichyrl1awt24askc 2406388 2406387 2026-04-27T15:46:27Z Stashbot 7414 kemayo@deploy1003: kemayo: Backport for [[gerrit:1277572|Backwards compatibility for mw.testKitchen.compat (ironically) (T422740)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406388 wikitext text/x-wiki eon07sq04dfierjbn24as6ixgnu8efk 2406389 2406388 2026-04-27T15:47:34Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2406389 wikitext text/x-wiki bbjtewk5s66667ojntckgwbeuqn9chh 2406390 2406389 2026-04-27T15:49:27Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2406390 wikitext text/x-wiki g7k2kvfv49qfzx9qariwteph8gua3he 2406391 2406390 2026-04-27T15:50:58Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2171: Repooling 2406391 wikitext text/x-wiki 6gx01qu2fd4jaf6of6qbxa69tsxnvau 2406393 2406391 2026-04-27T15:53:05Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1210: Repooling 2406393 wikitext text/x-wiki azgaro4wvbse7w800og9mcvoev9unn6 2406395 2406393 2026-04-27T15:56:00Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1092.eqiad.wmnet with OS bullseye 2406395 wikitext text/x-wiki nwu9v1pctq2ex1kfnvlgwnb1ejq4e68 2406396 2406395 2026-04-27T15:56:52Z Stashbot 7414 kemayo@deploy1003: kemayo: Continuing with deployment 2406396 wikitext text/x-wiki keg8eqx1bs2f0w9sb3mjqt5w9vca7b4 2406397 2406396 2026-04-27T15:57:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 (T419961)', diff saved to https://phabricator.wikimedia.org/P91638 and previous config saved to /var/cache/conftool/dbconfig/20260427-155655-fceratto.json 2406397 wikitext text/x-wiki 63u5ghxoimw3nn2ts3outbdd8qdpga5 2406398 2406397 2026-04-27T15:57:06Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS bullseye 2406398 wikitext text/x-wiki bl1u0czesqkikr5v6599d4wj8hmtlxv 2406399 2406398 2026-04-27T15:57:38Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1093 2406399 wikitext text/x-wiki 80qerkmg48l7lclshrtxclfcyig4p0n 2406400 2406399 2026-04-27T15:57:42Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on phab2003.codfw.wmnet with reason: host reimage 2406400 wikitext text/x-wiki csjn3il44mx97mo1hjvnwg8xjc2xqor 2406401 2406400 2026-04-27T15:57:48Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2406401 wikitext text/x-wiki lzu4lep23xjw96eh4231bp1ze4ulaoq 2406402 2406401 2026-04-27T16:00:46Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277572|Backwards compatibility for mw.testKitchen.compat (ironically) (T422740)]] (duration: 15m 57s) 2406402 wikitext text/x-wiki ior5xw8f86znumqwormt6nc7hxjikw3 2406403 2406402 2026-04-27T16:01:56Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1093 - mvernon@cumin2002" 2406403 wikitext text/x-wiki nnfezom689g27g6d0zswyqn2zcvd63b 2406404 2406403 2026-04-27T16:02:02Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1093 - mvernon@cumin2002" 2406404 wikitext text/x-wiki 1aa1vxbq77ozzbnz6n6n1hlt3o3v7qg 2406405 2406404 2026-04-27T16:02:05Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406405 wikitext text/x-wiki 153n4dvtnwtccga0b1z3uhkzmjbymiy 2406406 2406405 2026-04-27T16:02:07Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1093.eqiad.wmnet 133.48.64.10.in-addr.arpa 3.3.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2406406 wikitext text/x-wiki jy7e0n4rgyhgp2bg8xrun9umysvz0sd 2406407 2406406 2026-04-27T16:02:10Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1093.eqiad.wmnet 133.48.64.10.in-addr.arpa 3.3.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2406407 wikitext text/x-wiki kcr2zcucvv49e9q1f8ljjm3ou18qsx7 2406408 2406407 2026-04-27T16:02:13Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1093 2406408 wikitext text/x-wiki sveqpm5u7rhbqtssdqt10efvgvadvtf 2406409 2406408 2026-04-27T16:05:19Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1093 2406409 wikitext text/x-wiki 1pwyhlf7zqncxyb1i8bfncgsnksmoix 2406410 2406409 2026-04-27T16:05:21Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1093 2406410 wikitext text/x-wiki 64c3tiqjnz8nqrox8hkj5zxeiaa2hpb 2406411 2406410 2026-04-27T16:06:12Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2171: Repooling 2406411 wikitext text/x-wiki 47qtiwfa8ffcrqk4fa41d3epfnnmcut 2406412 2406411 2026-04-27T16:06:25Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab2003.codfw.wmnet with reason: host reimage 2406412 wikitext text/x-wiki i1kl6ll10161o8rpzy8r8y78ftvn3y2 2406414 2406412 2026-04-27T16:07:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P91640 and previous config saved to /var/cache/conftool/dbconfig/20260427-160704-fceratto.json 2406414 wikitext text/x-wiki eygbtzmiaacmbhjn9zbuj0dk22vii8n 2406416 2406414 2026-04-27T16:08:24Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1210: Repooling 2406416 wikitext text/x-wiki tlajlraoctln43x30hsd4pxv2vi4pbp 2406417 2406416 2026-04-27T16:10:17Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2406417 wikitext text/x-wiki ajuowfkotli2i3cc6783vdn63fkq5rx 2406418 2406417 2026-04-27T16:10:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91642 and previous config saved to /var/cache/conftool/dbconfig/20260427-161023-fceratto.json 2406418 wikitext text/x-wiki iasubpv1sk14pimcg5mdmozdiuc7u1r 2406419 2406418 2026-04-27T16:12:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91643 and previous config saved to /var/cache/conftool/dbconfig/20260427-161233-fceratto.json 2406419 wikitext text/x-wiki ml99tt1qix51x3mjxtcl1pwnf3cr4aa 2406420 2406419 2026-04-27T16:17:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P91644 and previous config saved to /var/cache/conftool/dbconfig/20260427-161710-fceratto.json 2406420 wikitext text/x-wiki t1pxnv5lth633fhbrix8gj4drgxhy5m 2406421 2406420 2026-04-27T16:22:44Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host phab2003.codfw.wmnet with OS trixie 2406421 wikitext text/x-wiki ao5p9r5eahp8dm39m4h06gk7sngzk8e 2406422 2406421 2026-04-27T16:22:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P91646 and previous config saved to /var/cache/conftool/dbconfig/20260427-162243-fceratto.json 2406422 wikitext text/x-wiki 486v4z0qm9zssp8zwtkxiem5n4prya1 2406423 2406422 2026-04-27T16:27:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 (T419961)', diff saved to https://phabricator.wikimedia.org/P91647 and previous config saved to /var/cache/conftool/dbconfig/20260427-162718-fceratto.json 2406423 wikitext text/x-wiki cgwr9zxpyqkch5n5nlyd6s97vzrthwm 2406424 2406423 2026-04-27T16:27:24Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage 2406424 wikitext text/x-wiki bdixjdmmlx1nn74x2vcpkruz5lzl4fl 2406425 2406424 2026-04-27T16:27:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance 2406425 wikitext text/x-wiki 6ehi513uce638sfgpsthkcq0u1q6k7y 2406426 2406425 2026-04-27T16:27:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1261 (T419961)', diff saved to https://phabricator.wikimedia.org/P91648 and previous config saved to /var/cache/conftool/dbconfig/20260427-162746-fceratto.json 2406426 wikitext text/x-wiki dbx6b09vo9kydsx1jwq7xfxextua5gx 2406427 2406426 2026-04-27T16:32:09Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage 2406427 wikitext text/x-wiki bfyth5axr2ic07c313n5zxi8su9oj3o 2406428 2406427 2026-04-27T16:32:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P91649 and previous config saved to /var/cache/conftool/dbconfig/20260427-163251-fceratto.json 2406428 wikitext text/x-wiki ltnb0r4oi76q2ogkc9kqhftwywe3mdn 2406429 2406428 2026-04-27T16:37:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 (T419961)', diff saved to https://phabricator.wikimedia.org/P91650 and previous config saved to /var/cache/conftool/dbconfig/20260427-163725-fceratto.json 2406429 wikitext text/x-wiki a5pulzt18aemfxyl1vs4wulsc01bbof 2406430 2406429 2026-04-27T16:43:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91651 and previous config saved to /var/cache/conftool/dbconfig/20260427-164259-fceratto.json 2406430 wikitext text/x-wiki 2pmuo2ekpkzt4bvusie0kh9kpz70tcx 2406431 2406430 2026-04-27T16:43:08Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance 2406431 wikitext text/x-wiki 6xtrmdnle5wqdteqn5gqqxxfz1rsgcr 2406432 2406431 2026-04-27T16:43:17Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2406432 wikitext text/x-wiki k93gmj4587ak7ns93finwstx8dmdisc 2406433 2406432 2026-04-27T16:43:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 (T419635)', diff saved to https://phabricator.wikimedia.org/P91652 and previous config saved to /var/cache/conftool/dbconfig/20260427-164323-fceratto.json 2406433 wikitext text/x-wiki 6xmeu2ezzfu6au6ulr6tkrogm1ig46y 2406434 2406433 2026-04-27T16:45:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T419635)', diff saved to https://phabricator.wikimedia.org/P91653 and previous config saved to /var/cache/conftool/dbconfig/20260427-164533-fceratto.json 2406434 wikitext text/x-wiki 8zz5yd677g1hztycxjh3d65lmo57fam 2406435 2406434 2026-04-27T16:47:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P91654 and previous config saved to /var/cache/conftool/dbconfig/20260427-164734-fceratto.json 2406435 wikitext text/x-wiki 3byo6unso5q4jxjl2ems2411ei9wrpo 2406436 2406435 2026-04-27T16:51:56Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS bullseye 2406436 wikitext text/x-wiki fpei5m67n8w0ykrlgookxde2gdeg7f2 2406437 2406436 2026-04-27T16:55:38Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudelastic1010.eqiad.wmnet with OS trixie 2406437 wikitext text/x-wiki 8u0hw3kq9emtx3l6bjd56k57eo2r64y 2406438 2406437 2026-04-27T16:55:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P91656 and previous config saved to /var/cache/conftool/dbconfig/20260427-165541-fceratto.json 2406438 wikitext text/x-wiki 7dg3ta5z991ymcxceiw6nz162vu89hz 2406439 2406438 2026-04-27T16:57:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P91657 and previous config saved to /var/cache/conftool/dbconfig/20260427-165742-fceratto.json 2406439 wikitext text/x-wiki klixaw9tmhqsjpnan8m0us7g9qlhd49 2406440 2406439 2026-04-27T17:00:15Z Stashbot 7414 bking@cumin2002: conftool action : set/weight=10; selector: name=cloudelastic1012. 2406440 wikitext text/x-wiki d3gr4gh2ups26mobrfvkhn804pqpfxa 2406441 2406440 2026-04-27T17:00:24Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=yes; selector: name=cloudelastic1012. 2406441 wikitext text/x-wiki 0ddahgcwjwvna9hkcuuk3061lc6p599 2406442 2406441 2026-04-27T17:05:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P91658 and previous config saved to /var/cache/conftool/dbconfig/20260427-170550-fceratto.json 2406442 wikitext text/x-wiki exo7mkfogwl7vpfkmkmk7l3jzjkbi3h 2406443 2406442 2026-04-27T17:07:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 (T419961)', diff saved to https://phabricator.wikimedia.org/P91659 and previous config saved to /var/cache/conftool/dbconfig/20260427-170750-fceratto.json 2406443 wikitext text/x-wiki e0mlgwdl6vwyhom7uc3e251pvstijlx 2406444 2406443 2026-04-27T17:08:12Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance 2406444 wikitext text/x-wiki 9utep4yh9l12wq96prlqof46hn1p30n 2406445 2406444 2026-04-27T17:08:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1262 (T419961)', diff saved to https://phabricator.wikimedia.org/P91660 and previous config saved to /var/cache/conftool/dbconfig/20260427-170819-fceratto.json 2406445 wikitext text/x-wiki ld8pjk216uknik0utj9lmso6v3d027q 2406446 2406445 2026-04-27T17:16:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T419635)', diff saved to https://phabricator.wikimedia.org/P91661 and previous config saved to /var/cache/conftool/dbconfig/20260427-171600-fceratto.json 2406446 wikitext text/x-wiki gpryl4gbk1jwpaly6kj0516up90obct 2406448 2406446 2026-04-27T17:16:09Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1185.eqiad.wmnet with reason: Maintenance 2406448 wikitext text/x-wiki cz3bfr24fdddkwqh4nbpzay2zteh9x7 2406449 2406448 2026-04-27T17:16:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 (T419635)', diff saved to https://phabricator.wikimedia.org/P91662 and previous config saved to /var/cache/conftool/dbconfig/20260427-171615-fceratto.json 2406449 wikitext text/x-wiki j9mquq5pmbg6l9r1zywx60ojnsd3uej 2406450 2406449 2026-04-27T17:17:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 (T419961)', diff saved to https://phabricator.wikimedia.org/P91663 and previous config saved to /var/cache/conftool/dbconfig/20260427-171659-fceratto.json 2406450 wikitext text/x-wiki chjzlsst79el37z7gyzo0in81ghynks 2406451 2406450 2026-04-27T17:18:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T419635)', diff saved to https://phabricator.wikimedia.org/P91664 and previous config saved to /var/cache/conftool/dbconfig/20260427-171826-fceratto.json 2406451 wikitext text/x-wiki 6dkutm9802mdqfxch0tdp3861xl7sry 2406453 2406451 2026-04-27T17:27:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P91665 and previous config saved to /var/cache/conftool/dbconfig/20260427-172707-fceratto.json 2406453 wikitext text/x-wiki jb04yzlgqykfvlgyayeb0lxo8zbenko 2406454 2406453 2026-04-27T17:28:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P91666 and previous config saved to /var/cache/conftool/dbconfig/20260427-172835-fceratto.json 2406454 wikitext text/x-wiki r4s4nnykflluy6vt0qzrpvac51egjpo 2406455 2406454 2026-04-27T17:37:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P91667 and previous config saved to /var/cache/conftool/dbconfig/20260427-173715-fceratto.json 2406455 wikitext text/x-wiki ieh7qtpxl5p0mytt3g3esw9pnspjd1b 2406456 2406455 2026-04-27T17:38:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P91668 and previous config saved to /var/cache/conftool/dbconfig/20260427-173843-fceratto.json 2406456 wikitext text/x-wiki gey06aw7li8aj6lpgvizrl9ls2tc3ai 2406458 2406456 2026-04-27T17:47:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 (T419961)', diff saved to https://phabricator.wikimedia.org/P91669 and previous config saved to /var/cache/conftool/dbconfig/20260427-174723-fceratto.json 2406458 wikitext text/x-wiki sn5kt5ks136543u91e2zgq71awa1lcn 2406459 2406458 2026-04-27T17:47:46Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance 2406459 wikitext text/x-wiki r1qul6htp52wmk2kl6z767q6t1yxbq0 2406460 2406459 2026-04-27T17:47:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1263 (T419961)', diff saved to https://phabricator.wikimedia.org/P91670 and previous config saved to /var/cache/conftool/dbconfig/20260427-174751-fceratto.json 2406460 wikitext text/x-wiki ocog8b84c26vug7iuxxg7udax9vjyc5 2406461 2406460 2026-04-27T17:48:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T419635)', diff saved to https://phabricator.wikimedia.org/P91671 and previous config saved to /var/cache/conftool/dbconfig/20260427-174852-fceratto.json 2406461 wikitext text/x-wiki bnzb9fsnbj83v9hozc9vlf0u4m7at0e 2406462 2406461 2026-04-27T17:48:59Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1200.eqiad.wmnet with reason: Maintenance 2406462 wikitext text/x-wiki a5vg8squwzlz0aizxl4hxsgwgkh4afi 2406463 2406462 2026-04-27T17:49:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 (T419635)', diff saved to https://phabricator.wikimedia.org/P91672 and previous config saved to /var/cache/conftool/dbconfig/20260427-174906-fceratto.json 2406463 wikitext text/x-wiki a2xprf5n26qxevngp9jqdeomzz2rt8x 2406464 2406463 2026-04-27T17:51:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T419635)', diff saved to https://phabricator.wikimedia.org/P91673 and previous config saved to /var/cache/conftool/dbconfig/20260427-175116-fceratto.json 2406464 wikitext text/x-wiki gwlk6o0i1xg59uh93f9ariisawd4z9w 2406465 2406464 2026-04-27T17:56:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 (T419961)', diff saved to https://phabricator.wikimedia.org/P91674 and previous config saved to /var/cache/conftool/dbconfig/20260427-175626-fceratto.json 2406465 wikitext text/x-wiki de9hmq8cf5n4zhrx5lkralcnokkmbhq 2406466 2406465 2026-04-27T18:01:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P91675 and previous config saved to /var/cache/conftool/dbconfig/20260427-180124-fceratto.json 2406466 wikitext text/x-wiki cnoutyvkop4ks943kywnocy9l42jz1o 2406468 2406466 2026-04-27T18:06:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P91676 and previous config saved to /var/cache/conftool/dbconfig/20260427-180635-fceratto.json 2406468 wikitext text/x-wiki h8bqmmz6qyc6zsv36tk4x40a7c2o4mn 2406469 2406468 2026-04-27T18:11:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P91677 and previous config saved to /var/cache/conftool/dbconfig/20260427-181132-fceratto.json 2406469 wikitext text/x-wiki pw5nejhjtgdjoda49h1uxcqxjp9j9y7 2406470 2406469 2026-04-27T18:16:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P91678 and previous config saved to /var/cache/conftool/dbconfig/20260427-181643-fceratto.json 2406470 wikitext text/x-wiki rh9lr7xdd8b77ryudxw98rosnvck24j 2406471 2406470 2026-04-27T18:21:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T419635)', diff saved to https://phabricator.wikimedia.org/P91679 and previous config saved to /var/cache/conftool/dbconfig/20260427-182140-fceratto.json 2406471 wikitext text/x-wiki 8ta9d33dpywx8b79e9r43ljh9o1nrz2 2406472 2406471 2026-04-27T18:21:47Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance 2406472 wikitext text/x-wiki jk4sififmfkiz82yk9no567fk5oan0a 2406473 2406472 2026-04-27T18:21:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 (T419635)', diff saved to https://phabricator.wikimedia.org/P91680 and previous config saved to /var/cache/conftool/dbconfig/20260427-182154-fceratto.json 2406473 wikitext text/x-wiki rwvbp9kckxc5gnnsr3md80rtus2jpn7 2406474 2406473 2026-04-27T18:24:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T419635)', diff saved to https://phabricator.wikimedia.org/P91681 and previous config saved to /var/cache/conftool/dbconfig/20260427-182404-fceratto.json 2406474 wikitext text/x-wiki 75eoa8hr247m6ek5wvqwxpgo18l4kss 2406475 2406474 2026-04-27T18:24:44Z Stashbot 7414 jasmine@dns1004: START - running authdns-update 2406475 wikitext text/x-wiki 5llhe4ykql6y9cagjahmt0143u2w1i8 2406476 2406475 2026-04-27T18:26:23Z Stashbot 7414 jasmine@dns1004: END - running authdns-update 2406476 wikitext text/x-wiki c2nfkkmcy0ud0qwoqjt5gcqx9u3gqkc 2406477 2406476 2026-04-27T18:26:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 (T419961)', diff saved to https://phabricator.wikimedia.org/P91682 and previous config saved to /var/cache/conftool/dbconfig/20260427-182652-fceratto.json 2406477 wikitext text/x-wiki 0v4kii3f9i7n6nn68h5lh8op9bm3idp 2406478 2406477 2026-04-27T18:34:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P91685 and previous config saved to /var/cache/conftool/dbconfig/20260427-183413-fceratto.json 2406478 wikitext text/x-wiki 8avx0leexkewk8np1qrc5ecurnjxdxe 2406479 2406478 2026-04-27T18:44:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P91686 and previous config saved to /var/cache/conftool/dbconfig/20260427-184421-fceratto.json 2406479 wikitext text/x-wiki qd9nv6jb5to36083ta7g6m3j9o1kxfs 2406480 2406479 2026-04-27T18:54:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T419635)', diff saved to https://phabricator.wikimedia.org/P91687 and previous config saved to /var/cache/conftool/dbconfig/20260427-185429-fceratto.json 2406480 wikitext text/x-wiki gmm1w86m08d4uwjmdymucsm0w4n33v3 2406481 2406480 2026-04-27T18:54:37Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1210.eqiad.wmnet with reason: Maintenance 2406481 wikitext text/x-wiki pmzz1vnx27axkel0l1qths4igy5v776 2406482 2406481 2026-04-27T18:54:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 (T419635)', diff saved to https://phabricator.wikimedia.org/P91688 and previous config saved to /var/cache/conftool/dbconfig/20260427-185444-fceratto.json 2406482 wikitext text/x-wiki cbr5l6xu2xog2u8azpxbsu06eq0a4um 2406483 2406482 2026-04-27T18:56:03Z Stashbot 7414 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1277693|Add 2FA enforcement demotion config for phase 1 groups]] 2406483 wikitext text/x-wiki 35tu04al9ghlnv8hxoj21azf4j2vg1a 2406484 2406483 2026-04-27T18:57:43Z Stashbot 7414 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1277693|Add 2FA enforcement demotion config for phase 1 groups]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406484 wikitext text/x-wiki svcckvf9ck2ia7p5lrz900nh3vwngg3 2406486 2406484 2026-04-27T19:00:01Z Stashbot 7414 jasmine_: "Restarting pybal on the backup LVS servers in eqiad" 2406486 wikitext text/x-wiki o3zkd6ozzksc1yje7rlbarbny2jzs7r 2406487 2406486 2026-04-27T19:00:28Z Stashbot 7414 alexsanford@deploy1003: alexsanford: Continuing with deployment 2406487 wikitext text/x-wiki k5mzy4l4r534bl3wihmqe0qrj4qokix 2406490 2406487 2026-04-27T19:05:06Z Stashbot 7414 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277693|Add 2FA enforcement demotion config for phase 1 groups]] (duration: 09m 03s) 2406490 wikitext text/x-wiki ntzq8uy1ht4udyhlajt5nn44f0yekk2 2406491 2406490 2026-04-27T19:06:55Z Stashbot 7414 jasmine_: "Restarting pybal on primary LVS servers in eqiad" 2406491 wikitext text/x-wiki 3osysqkzz5snezntf8y4cw9rg4od6f3 2406492 2406491 2026-04-27T19:19:44Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.253.0" for 2 host(s) 2406492 wikitext text/x-wiki ctj3ibo0kvttt67bnxo3clq413n9rax 2406493 2406492 2026-04-27T19:20:27Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ganeti1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406493 wikitext text/x-wiki 0dto22fx1k24d8rpyhmc25zqckb2yhm 2406494 2406493 2026-04-27T19:21:36Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.253.0" completed for 2 hosts 2406494 wikitext text/x-wiki sowkvveeffaymp7jgxzs3uyg3sb5ibl 2406495 2406494 2026-04-27T19:31:58Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406495 wikitext text/x-wiki qd6gjj7fnq1m18o2d3xwm5gosow2zz8 2406496 2406495 2026-04-27T19:32:04Z Stashbot 7414 rzl: root@apt1002:~# reprepro --noskipold --restrict vopsbot update bookworm-wikimedia 2406496 wikitext text/x-wiki j61k3i6pwr59h2jl3vurxxp4fcl5iof 2406497 2406496 2026-04-27T19:33:15Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ganeti1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406497 wikitext text/x-wiki 1lkwjwsk2imuksh2v30kyfjrmwm7hc9 2406498 2406497 2026-04-27T19:41:36Z Stashbot 7414 jasmine_: "Restarting pybal on the backup LVS servers in codfw" 2406498 wikitext text/x-wiki 2vxvi61gjjoujbn3rzf6s56mxsuhyie 2406499 2406498 2026-04-27T19:45:10Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406499 wikitext text/x-wiki q8a45fvz9kdnwpjdyxwq80zq520ed7t 2406500 2406499 2026-04-27T19:45:50Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ganeti1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406500 wikitext text/x-wiki 0qoq7f3nwd52zc5kw7vvfj2gx6qzxes 2406501 2406500 2026-04-27T19:49:45Z Stashbot 7414 jasmine_: "Restarting pybal on primary LVS servers in codfw" 2406501 wikitext text/x-wiki hp8ili754myosr0tsko8xdzfpl943ik 2406502 2406501 2026-04-27T20:02:09Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406502 wikitext text/x-wiki ea2b42lb3kjbi5gjy2ew7e1cgyye7d8 2406503 2406502 2026-04-27T20:02:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ganeti1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406503 wikitext text/x-wiki hcngq5k4e5299rii8oq43wp3q122grm 2406504 2406503 2026-04-27T20:03:18Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406504 wikitext text/x-wiki hep4zzos050szd5i30ipb2hodwrg0fk 2406505 2406504 2026-04-27T20:04:06Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ganeti1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406505 wikitext text/x-wiki omtzhypjiitzedem73hsm08z1mkkm5k 2406506 2406505 2026-04-27T20:10:18Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:1277656|JS SDK: Add aliases for compatibility with existing experiment code (T419513)]] 2406506 wikitext text/x-wiki n39bvigx0yjgaw4cy53f2leu0i2ln03 2406507 2406506 2026-04-27T20:11:54Z Stashbot 7414 dancy@deploy1003: dancy, sfaci: Backport for [[gerrit:1277656|JS SDK: Add aliases for compatibility with existing experiment code (T419513)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406507 wikitext text/x-wiki i78no4ssl7gg0mdokeg4c405njf56q6 2406508 2406507 2026-04-27T20:13:04Z Stashbot 7414 dancy@deploy1003: dancy, sfaci: Continuing with deployment 2406508 wikitext text/x-wiki qsymxwevxwc8jo7f99rmym8hgx5f4qx 2406509 2406508 2026-04-27T20:14:37Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2406509 wikitext text/x-wiki 0xhaxo9y6q98pd7d4u6nucw1eqlirny 2406510 2406509 2026-04-27T20:16:56Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277656|JS SDK: Add aliases for compatibility with existing experiment code (T419513)]] (duration: 06m 38s) 2406510 wikitext text/x-wiki he304yl3ihuev3rh6psek24f3lhx3dl 2406511 2406510 2026-04-27T20:17:39Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti1055.eqiad.wmnet with OS bookworm 2406511 wikitext text/x-wiki 3ajojni7kc7i5j23s0dyag6off0mhu0 2406512 2406511 2026-04-27T20:30:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1210: Repooling 2406512 wikitext text/x-wiki l6c4du3nlqgq9y0dhtxv089e92mubm5 2406514 2406512 2026-04-27T20:45:25Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1210: Repooling 2406514 wikitext text/x-wiki n5oeqhlvv0chje8wg5okr79k9ij2o0r 2406515 2406514 2026-04-27T20:47:06Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1055.eqiad.wmnet with reason: host reimage 2406515 wikitext text/x-wiki r6pzi2fb489vk3u5ym7or1e93b1iqfb 2406516 2406515 2026-04-27T20:54:26Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1055.eqiad.wmnet with reason: host reimage 2406516 wikitext text/x-wiki atkw59l27wpjx7eqqjdwx99m120x0mt 2406517 2406516 2026-04-27T21:13:40Z Stashbot 7414 jasmine@deploy1003: conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=sophroid 2406517 wikitext text/x-wiki askg4sc2vzdlak51o48qo25zqaavfkb 2406518 2406517 2026-04-27T21:13:55Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2406518 wikitext text/x-wiki tpxdp117v6z0zf022gkf4kylsjn64k0 2406519 2406518 2026-04-27T21:14:19Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2406519 wikitext text/x-wiki 8q808mqm9s6nir46w6emgd7co732elx 2406520 2406519 2026-04-27T21:14:22Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1055.eqiad.wmnet with OS bookworm 2406520 wikitext text/x-wiki 3qiamfpwlt4zp2y2lhjq7i0judb3yf9 2406521 2406520 2026-04-27T21:14:50Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti1056.eqiad.wmnet with OS bookworm 2406521 wikitext text/x-wiki jmcd6xf3sy10v1w0jgga0zijybfdw2h 2406522 2406521 2026-04-27T21:14:58Z Stashbot 7414 jasmine@dns1004: START - running authdns-update 2406522 wikitext text/x-wiki alev0x5u1zwqs0s4acqyopah4zxqveh 2406523 2406522 2026-04-27T21:16:34Z Stashbot 7414 jasmine@dns1004: END - running authdns-update 2406523 wikitext text/x-wiki eq8f3mbx0arch5essgfwjw3ro6qsrmm 2406524 2406523 2026-04-27T21:27:08Z Stashbot 7414 alexsanford: Deployed security fix for T422306 2406524 wikitext text/x-wiki 9f08e7zdlglq80t6ad1snrt36vt4b4v 2406525 2406524 2026-04-27T21:29:48Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1056.eqiad.wmnet with reason: host reimage 2406525 wikitext text/x-wiki dcs0mq3e7ap5c678egl5emlff0ua1uj 2406526 2406525 2026-04-27T21:35:43Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1056.eqiad.wmnet with reason: host reimage 2406526 wikitext text/x-wiki 3ahxowvgltjtbqrt2mvxbpufek20pxd 2406527 2406526 2026-04-27T21:39:48Z Stashbot 7414 maryum: Deployed security fix for T422676 2406527 wikitext text/x-wiki arhkq67l2b44zqa07dnx02kno24oc3t 2406528 2406527 2026-04-27T21:54:00Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2406528 wikitext text/x-wiki i2b4qnbb6miabxcjns2xf7xiero54j3 2406529 2406528 2026-04-27T21:54:24Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2406529 wikitext text/x-wiki tlzum8w49qu0p96uplv364fjpz9dh77 2406530 2406529 2026-04-27T21:54:26Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1056.eqiad.wmnet with OS bookworm 2406530 wikitext text/x-wiki a1y7wfe0evtdhk6jhnprioaecu6qmn7 2406531 2406530 2026-04-27T21:54:44Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti1057.eqiad.wmnet with OS bookworm 2406531 wikitext text/x-wiki 1fets1vumhgyhhgyfazfdknqfk4bxv0 2406533 2406531 2026-04-27T22:09:08Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1057.eqiad.wmnet with reason: host reimage 2406533 wikitext text/x-wiki 7uk3m64hb0rw64wyimzfj5fuaw1znu7 2406537 2406533 2026-04-27T22:16:03Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1057.eqiad.wmnet with reason: host reimage 2406537 wikitext text/x-wiki 1xhkhlho0l9nu5fcfwhn5bxr6q9nscr 2406540 2406537 2026-04-27T22:34:43Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2406540 wikitext text/x-wiki s7aul0hlqlxjfcnhj5ge2j8jxhlqmcu 2406541 2406540 2026-04-27T22:35:32Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2406541 wikitext text/x-wiki h8c2zad3ln0djbswd53yjwvig6pdvkd 2406542 2406541 2026-04-27T22:35:34Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1057.eqiad.wmnet with OS bookworm 2406542 wikitext text/x-wiki cr202v3spkgemfzyxf8gvdoxmu6gn0a 2406543 2406542 2026-04-27T22:36:32Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti1058.eqiad.wmnet with OS bookworm 2406543 wikitext text/x-wiki 9csphowcji25094hbln4du68sp44o2c 2406546 2406543 2026-04-27T22:51:30Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1058.eqiad.wmnet with reason: host reimage 2406546 wikitext text/x-wiki c52fx813scq9sohrfp90o4kz8q4i1pr 2406547 2406546 2026-04-27T23:00:03Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1058.eqiad.wmnet with reason: host reimage 2406547 wikitext text/x-wiki 41y4uhla3m1i27ngr0tf8zpk9do8u6h 2406548 2406547 2026-04-27T23:18:03Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2406548 wikitext text/x-wiki m28ca0f2nytm1x6sgjvmjww7qq3j5wc 2406549 2406548 2026-04-27T23:25:09Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2406549 wikitext text/x-wiki aaedp73p0fdh7tg5zi3k8l30noaj18u 2406550 2406549 2026-04-27T23:25:12Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1058.eqiad.wmnet with OS bookworm 2406550 wikitext text/x-wiki 75j9thifsw9ymeho32qxom8v6k6qxe3 2406556 2406550 2026-04-28T02:01:12Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2406556 wikitext text/x-wiki 2b51misvfjgg8637trgziko2hlvb5uf 2406557 2406556 2026-04-28T02:07:27Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 15s) 2406557 wikitext text/x-wiki 49hbcsevgrakxjxjrgo7niu4p471o94 2406559 2406557 2026-04-28T03:03:18Z Stashbot 7414 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.46.0-wmf.26 refs T423877 2406559 wikitext text/x-wiki ddrmhbtcmh89j9ceux61vfeqjepvegc 2406560 2406559 2026-04-28T03:38:53Z Stashbot 7414 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.46.0-wmf.26 refs T423877 (duration: 35m 34s) 2406560 wikitext text/x-wiki 7zstz8n0uq582pi7hhktugw47hee56h 2406561 2406560 2026-04-28T05:03:24Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s6 T424522 2406561 wikitext text/x-wiki qmfj24dik8nyni1yrpsah3hjafxj8xn 2406562 2406561 2026-04-28T05:03:33Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1201 with weight 0 T424522', diff saved to https://phabricator.wikimedia.org/P91697 and previous config saved to /var/cache/conftool/dbconfig/20260428-050328-marostegui.json 2406562 wikitext text/x-wiki 2lcguwbstghqqt3thy5x8vjuw7jg5f5 2406564 2406562 2026-04-28T05:06:55Z Stashbot 7414 marostegui: Starting s6 eqiad failover from db1173 to db1201 - T424522 2406564 wikitext text/x-wiki tk1do1cxgr0j641pv3y0v6s0q2mkpo8 2406565 2406564 2026-04-28T05:07:46Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1201 to s6 primary and set section read-write T424522', diff saved to https://phabricator.wikimedia.org/P91699 and previous config saved to /var/cache/conftool/dbconfig/20260428-050742-marostegui.json 2406565 wikitext text/x-wiki 3pce662kfxonh58gz7ujkbi59bp8891 2406566 2406565 2026-04-28T05:08:45Z Stashbot 7414 marostegui@dns1005: START - running authdns-update 2406566 wikitext text/x-wiki 59r50uslsi5xnazjlgh226s0hnq4gd3 2406567 2406566 2026-04-28T05:09:42Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1173 T424522', diff saved to https://phabricator.wikimedia.org/P91700 and previous config saved to /var/cache/conftool/dbconfig/20260428-050938-marostegui.json 2406567 wikitext text/x-wiki dihh707okp4flgqn09k397agkc6py0e 2406568 2406567 2026-04-28T05:10:07Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1173: Repooling after switchover 2406568 wikitext text/x-wiki g3il52af1gapsx66fllqaxdlnk3ibyf 2406569 2406568 2026-04-28T05:10:16Z Stashbot 7414 marostegui@dns1005: END - running authdns-update 2406569 wikitext text/x-wiki 127z56z0biem12n6j6ca9oq8zsn6hke 2406570 2406569 2026-04-28T05:10:29Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1173: Repooling after switchover 2406570 wikitext text/x-wiki rkyey17lgfifbiy3p0mmhqtt9zgpde8 2406571 2406570 2026-04-28T05:10:40Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1173: Repooling after switchover 2406571 wikitext text/x-wiki jbxc4efnytthxjyzsjsw4qr17r8nvr7 2406572 2406571 2026-04-28T05:26:16Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 14 hosts with reason: Sanitarium: reimage to Debian Trixie 2406572 wikitext text/x-wiki m11jav5s40ncahtvtm30oztir7me463 2406573 2406572 2026-04-28T05:26:48Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1161.eqiad.wmnet with reason: Reimage to Trixie 2406573 wikitext text/x-wiki axkqmo2xodss4su8e6bhkht912wxk5p 2406574 2406573 2026-04-28T05:26:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1161: Reimage to Trixie 2406574 wikitext text/x-wiki opam5easm6wjhs0vgpor7jr2f718svs 2406575 2406574 2026-04-28T05:36:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1161: Reimage to Trixie 2406575 wikitext text/x-wiki dbnd9yfamz1ops9rta4j9i0nvy18c35 2406576 2406575 2026-04-28T05:37:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1161.eqiad.wmnet with OS trixie 2406576 wikitext text/x-wiki rddmnyzh3uw9ls37bxw5rpf6g7rh62g 2406577 2406576 2026-04-28T05:42:10Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2192.codfw.wmnet with reason: Reimage to Trixie 2406577 wikitext text/x-wiki hl12ailisork7i4imue9wjwg9bfj3kq 2406578 2406577 2026-04-28T05:42:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2192: Reimage to Trixie 2406578 wikitext text/x-wiki h0d2yd97d2h49qxkai5fg5927lnfgyt 2406579 2406578 2026-04-28T05:42:59Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2192: Reimage to Trixie 2406579 wikitext text/x-wiki tktu6i98wc6tjhm1dw8zpjpd1a1gwzj 2406580 2406579 2026-04-28T05:45:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2192.codfw.wmnet with OS trixie 2406580 wikitext text/x-wiki 67qr94sd2emo837kmz1rps5z9cgsrx7 2406581 2406580 2026-04-28T05:50:03Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1161.eqiad.wmnet with reason: host reimage 2406581 wikitext text/x-wiki 8ld9pp05nzysz3dz2op9tklvv1otufx 2406582 2406581 2026-04-28T05:55:47Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1161.eqiad.wmnet with reason: host reimage 2406582 wikitext text/x-wiki f3je7uhbjo2ucn8ivprykvi4nw7rgd0 2406583 2406582 2026-04-28T05:56:07Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1173: Repooling after switchover 2406583 wikitext text/x-wiki r9xm6mwpw31g558k35cqyqujj60i0w8 2406584 2406583 2026-04-28T06:00:31Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1173.eqiad.wmnet with reason: Reimage to Trixie 2406584 wikitext text/x-wiki hgyqw4f55vzvjxgo54hj0uthf6fi0av 2406585 2406584 2026-04-28T06:00:36Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1173: Reimage to Trixie 2406585 wikitext text/x-wiki tlt8w9ea7yk9xd2bsdvxsxykash2pqr 2406586 2406585 2026-04-28T06:01:03Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1173: Reimage to Trixie 2406586 wikitext text/x-wiki 00j5hhoah7ckbkl430sg4ygz5o9i3ha 2406587 2406586 2026-04-28T06:02:13Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1173.eqiad.wmnet with OS trixie 2406587 wikitext text/x-wiki 13rl5jswnlss6y3iet6pl15h8vsloi4 2406588 2406587 2026-04-28T06:07:30Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2192.codfw.wmnet with reason: host reimage 2406588 wikitext text/x-wiki od7of6tdaf5kewsc0bknbokis9am1g4 2406589 2406588 2026-04-28T06:11:01Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ncredir5001.eqsin.wmnet 2406589 wikitext text/x-wiki 2c5nrpn10yg151169597vke15xe3vjo 2406590 2406589 2026-04-28T06:11:38Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2192.codfw.wmnet with reason: host reimage 2406590 wikitext text/x-wiki o8d0v4x4j746de3ilkqt4i4u44jir4t 2406591 2406590 2026-04-28T06:17:01Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2406591 wikitext text/x-wiki g2004d48kf31uoagsqtet7mzqbc6f2k 2406592 2406591 2026-04-28T06:17:52Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1173.eqiad.wmnet with reason: host reimage 2406592 wikitext text/x-wiki ssd03f6cvg8cgwaxta09a6pix0hl95i 2406593 2406592 2026-04-28T06:18:35Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1161.eqiad.wmnet with OS trixie 2406593 wikitext text/x-wiki lqc036xwe7j8j8w2rzyarwqbqoic6wt 2406594 2406593 2026-04-28T06:21:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1161: after reimage to trixie 2406594 wikitext text/x-wiki n0ps35pfs56486cbq0sfp87ufs8fhj8 2406595 2406594 2026-04-28T06:23:56Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1173.eqiad.wmnet with reason: host reimage 2406595 wikitext text/x-wiki gbytil6udbl14kx2m1cdh3dt1jprd4h 2406596 2406595 2026-04-28T06:27:41Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ncredir5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406596 wikitext text/x-wiki r6c7lk0dnvk2ty4aeo9dqq9t0z9huuy 2406597 2406596 2026-04-28T06:28:19Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ncredir5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406597 wikitext text/x-wiki ifuu4747jrlt19bgzbs3vhbqq9x9rid 2406598 2406597 2026-04-28T06:28:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406598 wikitext text/x-wiki qc68ixdbot4ey986lkuqz5ljar0l0v8 2406599 2406598 2026-04-28T06:28:24Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ncredir5001.eqsin.wmnet 2406599 wikitext text/x-wiki mn6z32hfmcmnoi60xjbkfq6uvz8vmp4 2406600 2406599 2026-04-28T06:28:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ncredir5002.eqsin.wmnet 2406600 wikitext text/x-wiki lxiqorargm00b3rrmpqn9rq1g70b6ax 2406601 2406600 2026-04-28T06:33:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2406601 wikitext text/x-wiki fxpsjjlvqjyrdy1hae51fydd3aoakib 2406602 2406601 2026-04-28T06:36:28Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2192.codfw.wmnet with OS trixie 2406602 wikitext text/x-wiki 4sfqzxgwp8j63yx92ektd3n7g4sra7z 2406603 2406602 2026-04-28T06:38:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2192: after reimage to trixie 2406603 wikitext text/x-wiki csx3u736dxlzl4pdyg6a1v81gcm7qio 2406604 2406603 2026-04-28T06:39:04Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ncredir5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406604 wikitext text/x-wiki camau8vpu6oeelzovqvbdd931z9k4uf 2406605 2406604 2026-04-28T06:41:51Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ncredir5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406605 wikitext text/x-wiki mw4n8jf06hlldc45tpincydothvlmcg 2406606 2406605 2026-04-28T06:41:54Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406606 wikitext text/x-wiki 7849oxdx20ihq6sp1vam61ygvrohwfv 2406607 2406606 2026-04-28T06:41:57Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ncredir5002.eqsin.wmnet 2406607 wikitext text/x-wiki ts6gw9sfc7sklk19wacyd0htjnuyjxq 2406608 2406607 2026-04-28T06:47:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5005.eqsin.wmnet 2406608 wikitext text/x-wiki 9s9wz7kbsxz1zqj26u06gso7hd6wdxv 2406609 2406608 2026-04-28T06:49:14Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5005.eqsin.wmnet 2406609 wikitext text/x-wiki feo65nfwl4q4nw1882s6um2aiiah959 2406610 2406609 2026-04-28T06:49:28Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1173.eqiad.wmnet with OS trixie 2406610 wikitext text/x-wiki oddj4p7ex9dh9xaol6a36ni0fhwautq 2406611 2406610 2026-04-28T06:51:00Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of install5003.wikimedia.org to plain 2406611 wikitext text/x-wiki ec2o7c8wcfmt5784hq40gk1cgk4eoh9 2406612 2406611 2026-04-28T06:51:18Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1173: after reimage to trixie 2406612 wikitext text/x-wiki c2621i64h0r0il9lhvft6d0m3eff2dd 2406613 2406612 2026-04-28T06:52:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install5003.wikimedia.org to plain 2406613 wikitext text/x-wiki n1obawyhmdu5estgb2qxmtb8hk8bo5j 2406614 2406613 2026-04-28T06:54:38Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5002.eqsin.wmnet to plain 2406614 wikitext text/x-wiki cy4gufrkib4zkqm622krk2aqtjqm2uc 2406615 2406614 2026-04-28T06:57:07Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus5002.eqsin.wmnet to plain 2406615 wikitext text/x-wiki jlbi4nv88m99dsrheu2u4sv52hfei1m 2406616 2406615 2026-04-28T07:04:18Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum5001.eqsin.wmnet to plain 2406616 wikitext text/x-wiki aibcjeoan7qrlmgnxcyz6ayy32v77bj 2406617 2406616 2026-04-28T07:05:12Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum5001.eqsin.wmnet to plain 2406617 wikitext text/x-wiki gu2sc0j5h3sd2sk54wv24lrs2mhjmln 2406618 2406617 2026-04-28T07:07:12Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: after reimage to trixie 2406618 wikitext text/x-wiki rdz2npxl6deqrv6l0t71bfn3dj0ugi8 2406619 2406618 2026-04-28T07:13:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of durum5002.eqsin.wmnet to plain 2406619 wikitext text/x-wiki q5ihvb3r4t04nugvxyoz6h20aqc8zk7 2406620 2406619 2026-04-28T07:14:08Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum5002.eqsin.wmnet to plain 2406620 wikitext text/x-wiki 88hhjp3wxixsx8c36qbgx65qaw472cd 2406621 2406620 2026-04-28T07:24:14Z Stashbot 7414 jayme: switching cfss-issuer instances on all clusters to use discovery2026 - T420993 2406621 wikitext text/x-wiki 4matjytw6mqqt5lkhokf2bngs7nmb7w 2406622 2406621 2026-04-28T07:24:22Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2192: after reimage to trixie 2406622 wikitext text/x-wiki oyoudnnbxm8s6yq9ei7u1eesqkino3u 2406623 2406622 2026-04-28T07:24:32Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2406623 wikitext text/x-wiki ek773vnvi6obu1k44lxcfp9cfwx9cby 2406624 2406623 2026-04-28T07:24:34Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2406624 wikitext text/x-wiki k2k142dnqlcedc7p9lukkassblygaea 2406625 2406624 2026-04-28T07:24:39Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2406625 wikitext text/x-wiki 44df63wfddwz5bd7gm4rxhkxa1ssx8i 2406626 2406625 2026-04-28T07:24:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of hcaptcha-proxy5001.wikimedia.org to plain 2406626 wikitext text/x-wiki 1dw7akt7tehtjzxi3j146yeeycqg3jg 2406627 2406626 2026-04-28T07:24:46Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2406627 wikitext text/x-wiki 9aorvoorfja9wph7j8ilnmiyuifyqsa 2406628 2406627 2026-04-28T07:24:49Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2406628 wikitext text/x-wiki d6akxuljjwy2vu33bgxrzke2dx6m3n8 2406629 2406628 2026-04-28T07:24:53Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2406629 wikitext text/x-wiki 3j5vinkbuxzlqznxlkk5ucs3undbivb 2406630 2406629 2026-04-28T07:24:58Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2406630 wikitext text/x-wiki pnao60imh0pna8h9syts6bvktxmcgqn 2406631 2406630 2026-04-28T07:25:04Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2406631 wikitext text/x-wiki a3dcejnlqe4tivdtc0zjv4f3qlui4tf 2406632 2406631 2026-04-28T07:25:12Z Stashbot 7414 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2406632 wikitext text/x-wiki kocmz08tcn9dqoxvndsihy3qxs4vc0g 2406633 2406632 2026-04-28T07:25:14Z Stashbot 7414 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2406633 wikitext text/x-wiki fydxstbex1ng72yp9fw7mue673mpcdr 2406634 2406633 2026-04-28T07:25:17Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2406634 wikitext text/x-wiki l4npa1ntub4ggqxawzwli3xc6igif1p 2406635 2406634 2026-04-28T07:25:21Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2406635 wikitext text/x-wiki qdp9naa9brp6mvto4p833xnevhpx0nm 2406636 2406635 2026-04-28T07:25:25Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2406636 wikitext text/x-wiki 3ma4548fzowg43lztya19s4nu67m0ou 2406637 2406636 2026-04-28T07:25:31Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2406637 wikitext text/x-wiki q946mu32e9s97sgt1pi0zw7jz9bz69e 2406638 2406637 2026-04-28T07:25:35Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2406638 wikitext text/x-wiki mt1004znymd8z80xv33vbzor8c0jfdv 2406639 2406638 2026-04-28T07:25:39Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2406639 wikitext text/x-wiki aonkongk7nfy2hknmjzktalyf85o2y4 2406640 2406639 2026-04-28T07:25:42Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2406640 wikitext text/x-wiki 7pwf8cnu2q647va0zk7kyt9xg61hfx5 2406641 2406640 2026-04-28T07:25:48Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2406641 wikitext text/x-wiki 3dif4oanrgak1xiyss6bs18zkwawz9d 2406642 2406641 2026-04-28T07:29:31Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of hcaptcha-proxy5001.wikimedia.org to plain 2406642 wikitext text/x-wiki c5x4e4g08a4a1ugniub4sxc4em12sqq 2406643 2406642 2026-04-28T07:36:42Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1173: after reimage to trixie 2406643 wikitext text/x-wiki lf6u5q8cugrerxh04y6nmnzkswbi4k2 2406644 2406643 2026-04-28T07:36:50Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of hcaptcha-proxy5002.wikimedia.org to plain 2406644 wikitext text/x-wiki 8u6jh8ifighnynxo00gss7jl5u368ev 2406645 2406644 2026-04-28T07:37:44Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of hcaptcha-proxy5002.wikimedia.org to plain 2406645 wikitext text/x-wiki t0cbiatcmtaaddkul9gs3jecibfsed5 2406646 2406645 2026-04-28T07:54:35Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply 2406646 wikitext text/x-wiki 6i8rrdw2u4soqjpvi7u5q4jk9klke6p 2406647 2406646 2026-04-28T07:54:58Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance 2406647 wikitext text/x-wiki o7i2k8do77zufc93ts5qo07ik5eno5b 2406648 2406647 2026-04-28T07:55:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2150 (T419961)', diff saved to https://phabricator.wikimedia.org/P91720 and previous config saved to /var/cache/conftool/dbconfig/20260428-075504-fceratto.json 2406648 wikitext text/x-wiki bvi652ztbzbarv4tiqtc3s90jm3p6ug 2406649 2406648 2026-04-28T07:55:53Z Stashbot 7414 jayme: started renewal of certificates on codfw kubernetes clusters - T420993 2406649 wikitext text/x-wiki 6pv5jb3ok5nwiex8w3tf3ldgoefv4xh 2406650 2406649 2026-04-28T07:56:18Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply 2406650 wikitext text/x-wiki bqbnlnn7xty9ljwu8df3il5gzfy7qhy 2406652 2406650 2026-04-28T08:01:40Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply 2406652 wikitext text/x-wiki dq9c5bzl46exm6zkq0czgss2u0fbxg1 2406653 2406652 2026-04-28T08:02:01Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply 2406653 wikitext text/x-wiki efnbg61izmzp9ycfbydcsvxukr00j1x 2406654 2406653 2026-04-28T08:03:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T419961)', diff saved to https://phabricator.wikimedia.org/P91721 and previous config saved to /var/cache/conftool/dbconfig/20260428-080337-fceratto.json 2406654 wikitext text/x-wiki 5r28ecc5vfbqsmcrj2xb4rc1j351tyv 2406655 2406654 2026-04-28T08:13:13Z Stashbot 7414 moritzm: installing e2fsprogs updates from trixie point release 2406655 wikitext text/x-wiki snqeogvuobj65bmrum22npc7h1vnd5g 2406656 2406655 2026-04-28T08:13:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P91722 and previous config saved to /var/cache/conftool/dbconfig/20260428-081346-fceratto.json 2406656 wikitext text/x-wiki 0mtqx9m2th4zv7h1dm9wgltqrhtithk 2406661 2406656 2026-04-28T08:23:38Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2150: Repooling 2406661 wikitext text/x-wiki f82hmn5dcwmkbibdqiyfwzyfo6jje3n 2406662 2406661 2026-04-28T08:23:47Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2150: Repooling 2406662 wikitext text/x-wiki fdp16hu2lp8myd0hddqr2xwlsrjisp1 2406663 2406662 2026-04-28T08:27:50Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance 2406663 wikitext text/x-wiki s1fqs8fqvhba4src8gjk7gjaxu0kolg 2406664 2406663 2026-04-28T08:27:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 (T419635)', diff saved to https://phabricator.wikimedia.org/P91724 and previous config saved to /var/cache/conftool/dbconfig/20260428-082756-fceratto.json 2406664 wikitext text/x-wiki esxe2yp55c2co1uj21gcdzb7vznxufi 2406667 2406664 2026-04-28T08:29:31Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance 2406667 wikitext text/x-wiki sygkyhigabrpaed26b5rm2dkfcso68m 2406668 2406667 2026-04-28T08:29:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2159 (T419961)', diff saved to https://phabricator.wikimedia.org/P91725 and previous config saved to /var/cache/conftool/dbconfig/20260428-082937-fceratto.json 2406668 wikitext text/x-wiki ghukm2t0ur05lkafxb1zkz0e4i05ngf 2406670 2406668 2026-04-28T08:31:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T419635)', diff saved to https://phabricator.wikimedia.org/P91726 and previous config saved to /var/cache/conftool/dbconfig/20260428-083127-fceratto.json 2406670 wikitext text/x-wiki m5rqv03yjnjxz9tmcw4uj1onpxngzyq 2406673 2406670 2026-04-28T08:37:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T419961)', diff saved to https://phabricator.wikimedia.org/P91727 and previous config saved to /var/cache/conftool/dbconfig/20260428-083706-fceratto.json 2406673 wikitext text/x-wiki c03d3d9d58k1inqo7hw2k4ki8gh8q5l 2406675 2406673 2026-04-28T08:40:39Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2238.codfw.wmnet with reason: Reimage to Trixie 2406675 wikitext text/x-wiki 3busajm2x1yc1ntavedvisp5vktysvy 2406676 2406675 2026-04-28T08:40:44Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2238: Reimage to Trixie 2406676 wikitext text/x-wiki n3g6quyoigo4cje52calutobu049qm1 2406677 2406676 2026-04-28T08:41:03Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2238: Reimage to Trixie 2406677 wikitext text/x-wiki 0pv49dumck35myxq7vhlt8kjylt8vgn 2406678 2406677 2026-04-28T08:41:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P91729 and previous config saved to /var/cache/conftool/dbconfig/20260428-084135-fceratto.json 2406678 wikitext text/x-wiki 8w80buo0gddklbutvdhw9o66xv2z4ri 2406680 2406678 2026-04-28T08:42:21Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2238.codfw.wmnet with OS trixie 2406680 wikitext text/x-wiki 5vc3mx8sjvdzd81h030k83xqj0mvepi 2406681 2406680 2026-04-28T08:42:25Z Stashbot 7414 jayme: started renewal of certificates on eqiad kubernetes clusters - T420993 2406681 wikitext text/x-wiki hd72283598mnj5181zmdiurjf5vrnci 2406683 2406681 2026-04-28T08:44:43Z Stashbot 7414 moritzm: migrate Ganeti clusters to the new discovery2026 intermediate, starting for the edges T420993 2406683 wikitext text/x-wiki gkccqfxyhrmlh9lsykksqxv11ntz9ni 2406685 2406683 2026-04-28T08:47:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P91730 and previous config saved to /var/cache/conftool/dbconfig/20260428-084714-fceratto.json 2406685 wikitext text/x-wiki 7yw50r8wmdch017q4p90xxy2tmae6gu 2406686 2406685 2026-04-28T08:51:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P91731 and previous config saved to /var/cache/conftool/dbconfig/20260428-085142-fceratto.json 2406686 wikitext text/x-wiki c4p9gitlsbhkv1wuoc1eli480km8s9h 2406687 2406686 2026-04-28T08:53:36Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2406687 wikitext text/x-wiki m3olzki2q31la9wpjtr70g0zn5x7jyh 2406690 2406687 2026-04-28T08:57:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P91732 and previous config saved to /var/cache/conftool/dbconfig/20260428-085722-fceratto.json 2406690 wikitext text/x-wiki 63szetqcmpzw41am34t4e9wmam7ol1q 2406691 2406690 2026-04-28T08:58:04Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2238.codfw.wmnet with reason: host reimage 2406691 wikitext text/x-wiki 0awzvncget29e2barvwxzn7ptoxyfb1 2406694 2406691 2026-04-28T09:01:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T419635)', diff saved to https://phabricator.wikimedia.org/P91733 and previous config saved to /var/cache/conftool/dbconfig/20260428-090150-fceratto.json 2406694 wikitext text/x-wiki t1f83fxd13ysaryo18yykcq5uuulxmo 2406695 2406694 2026-04-28T09:02:09Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance 2406695 wikitext text/x-wiki aqmsx4nko28m9qv80jnu0q9emvrppf4 2406696 2406695 2026-04-28T09:02:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 (T419635)', diff saved to https://phabricator.wikimedia.org/P91734 and previous config saved to /var/cache/conftool/dbconfig/20260428-090215-fceratto.json 2406696 wikitext text/x-wiki 01d69vok5tx498w172ol0dt6hpjinpb 2406698 2406696 2026-04-28T09:04:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2238.codfw.wmnet with reason: host reimage 2406698 wikitext text/x-wiki h7qaur7al0ebzrvprziyxqwdizv33a4 2406699 2406698 2026-04-28T09:04:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T419635)', diff saved to https://phabricator.wikimedia.org/P91735 and previous config saved to /var/cache/conftool/dbconfig/20260428-090446-fceratto.json 2406699 wikitext text/x-wiki 7v6p73vduwzxwkwgnau6bdclo8vtq02 2406702 2406699 2026-04-28T09:07:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T419961)', diff saved to https://phabricator.wikimedia.org/P91736 and previous config saved to /var/cache/conftool/dbconfig/20260428-090730-fceratto.json 2406702 wikitext text/x-wiki pj3jmgxv0u36yowakx8m66iu9v3w3pq 2406703 2406702 2026-04-28T09:07:53Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance 2406703 wikitext text/x-wiki 6os88ghbhvazvntp2g7qxnixk611xet 2406704 2406703 2026-04-28T09:08:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2168 (T419961)', diff saved to https://phabricator.wikimedia.org/P91737 and previous config saved to /var/cache/conftool/dbconfig/20260428-090759-fceratto.json 2406704 wikitext text/x-wiki m0vvifu637cj9pub6xq0hqca24lsjqt 2406705 2406704 2026-04-28T09:14:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P91738 and previous config saved to /var/cache/conftool/dbconfig/20260428-091454-fceratto.json 2406705 wikitext text/x-wiki n2e5l0c2g0pd8vubb89yayfkfygpms2 2406706 2406705 2026-04-28T09:15:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T419961)', diff saved to https://phabricator.wikimedia.org/P91739 and previous config saved to /var/cache/conftool/dbconfig/20260428-091527-fceratto.json 2406706 wikitext text/x-wiki 270w1hr3vfwx9zzpruxenrwn8o569n7 2406707 2406706 2026-04-28T09:21:32Z Stashbot 7414 moritzm: migrate eqiad/codfw Ganeti clusters to the new discovery2026 intermediate T420993 2406707 wikitext text/x-wiki 6qmxv9wc0brq9bfps497m74k0wy9y4y 2406708 2406707 2026-04-28T09:25:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P91740 and previous config saved to /var/cache/conftool/dbconfig/20260428-092502-fceratto.json 2406708 wikitext text/x-wiki hiw9vqicj7ztfxxfj5bkuhfcdgk0agc 2406709 2406708 2026-04-28T09:25:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P91741 and previous config saved to /var/cache/conftool/dbconfig/20260428-092534-fceratto.json 2406709 wikitext text/x-wiki akj5795pjuwvwuqrchn9e52zx19obx5 2406710 2406709 2026-04-28T09:28:27Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2238.codfw.wmnet with OS trixie 2406710 wikitext text/x-wiki 26k650kmthq34z2kwupinvw3pd9gk17 2406711 2406710 2026-04-28T09:32:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2238: after reimage to trixie 2406711 wikitext text/x-wiki h4opv1mtptnbiy7cp2idmfwcate2ny0 2406712 2406711 2026-04-28T09:35:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T419635)', diff saved to https://phabricator.wikimedia.org/P91743 and previous config saved to /var/cache/conftool/dbconfig/20260428-093510-fceratto.json 2406712 wikitext text/x-wiki 8bptarybe0emwn35yb03fkxzb0xqyrn 2406713 2406712 2026-04-28T09:35:29Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance 2406713 wikitext text/x-wiki ezz3uyitdir29wug731mw9djjf9h7mr 2406714 2406713 2026-04-28T09:35:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2178 (T419635)', diff saved to https://phabricator.wikimedia.org/P91744 and previous config saved to /var/cache/conftool/dbconfig/20260428-093535-fceratto.json 2406714 wikitext text/x-wiki 41gk5qugzdcmhsxgo0ahi4l4und800p 2406715 2406714 2026-04-28T09:35:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P91745 and previous config saved to /var/cache/conftool/dbconfig/20260428-093543-fceratto.json 2406715 wikitext text/x-wiki iumfw3cwz2eaxzbj7aoa6hnxyhyqu9b 2406716 2406715 2026-04-28T09:36:43Z Stashbot 7414 moritzm: installing openjdk-21 security updates 2406716 wikitext text/x-wiki kgyroz2ocb9kfz8ophieez5x997xauh 2406717 2406716 2026-04-28T09:40:42Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406717 wikitext text/x-wiki 248z95x79kevpgnss23nt1d8bz0en8i 2406718 2406717 2026-04-28T09:40:49Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406718 wikitext text/x-wiki drhg2hyuzj12qismwj1pla4pdfg8rsn 2406719 2406718 2026-04-28T09:41:02Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406719 wikitext text/x-wiki 1p4dpds5ik83g0r5ik40skz2ak3whuo 2406720 2406719 2026-04-28T09:41:05Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406720 wikitext text/x-wiki 603quzl3g1z3j5w40xd70vcll00j7bv 2406721 2406720 2026-04-28T09:41:35Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2406721 wikitext text/x-wiki irocgoko2yp7o1bv6yl5mq85k40pib9 2406722 2406721 2026-04-28T09:41:42Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2406722 wikitext text/x-wiki 713637sqpstzahbl6o9b9uiuv8nh02k 2406723 2406722 2026-04-28T09:41:53Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2406723 wikitext text/x-wiki 660j9m6t6e48r49o0tm25ieav8qzd7s 2406724 2406723 2026-04-28T09:41:56Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2406724 wikitext text/x-wiki 66dfes33wchs0pia4kzs3uvzbtxw5tl 2406725 2406724 2026-04-28T09:42:13Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2406725 wikitext text/x-wiki 4ulxyy8p2p7uybuniavsmlnp0oksjcy 2406726 2406725 2026-04-28T09:42:20Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2406726 wikitext text/x-wiki cu2blfm9bdnjh4liazwk1p9z8bplnf6 2406727 2406726 2026-04-28T09:42:31Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2406727 wikitext text/x-wiki 0jws2fc6gc97j84zkxlilkrqzprt65l 2406728 2406727 2026-04-28T09:42:35Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2406728 wikitext text/x-wiki 111eutb9cvmfiuahunjeqkh3nl7j9zs 2406729 2406728 2026-04-28T09:42:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T419635)', diff saved to https://phabricator.wikimedia.org/P91746 and previous config saved to /var/cache/conftool/dbconfig/20260428-094240-fceratto.json 2406729 wikitext text/x-wiki kzjqym7tp8d9ku05s4b9g3g7aiuijel 2406730 2406729 2026-04-28T09:45:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T419961)', diff saved to https://phabricator.wikimedia.org/P91747 and previous config saved to /var/cache/conftool/dbconfig/20260428-094551-fceratto.json 2406730 wikitext text/x-wiki m2i35rohurdo29igifgvmwldejm3xu9 2406731 2406730 2026-04-28T09:46:15Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance 2406731 wikitext text/x-wiki 32rqqgp2aja62nntftxprin1voebqen 2406732 2406731 2026-04-28T09:46:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2182 (T419961)', diff saved to https://phabricator.wikimedia.org/P91748 and previous config saved to /var/cache/conftool/dbconfig/20260428-094621-fceratto.json 2406732 wikitext text/x-wiki hp18iv1703xayo7gphvll1vy32xkuqb 2406734 2406732 2026-04-28T09:52:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P91750 and previous config saved to /var/cache/conftool/dbconfig/20260428-095248-fceratto.json 2406734 wikitext text/x-wiki p93jhilkdnsvr1vi606ek7k6ewsvne6 2406735 2406734 2026-04-28T09:53:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T419961)', diff saved to https://phabricator.wikimedia.org/P91751 and previous config saved to /var/cache/conftool/dbconfig/20260428-095340-fceratto.json 2406735 wikitext text/x-wiki g6zmjbht2l8hxph698wvr38tiglfgpj 2406736 2406735 2026-04-28T09:55:26Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2406736 wikitext text/x-wiki 32me55n9nxvoc1koeitu0a2xpumyhfv 2406737 2406736 2026-04-28T09:55:35Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2406737 wikitext text/x-wiki q2g8migvjhs4ezaic14ert1ld2oca2s 2406738 2406737 2026-04-28T09:56:10Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2406738 wikitext text/x-wiki j08x6qkummay0uicwejpl2w0eaa9yix 2406739 2406738 2026-04-28T09:57:17Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2406739 wikitext text/x-wiki fe45hzupxr8zrdlmxinuvccv2x9clyj 2406740 2406739 2026-04-28T10:02:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P91753 and previous config saved to /var/cache/conftool/dbconfig/20260428-100256-fceratto.json 2406740 wikitext text/x-wiki 0vihevcuh224kqkswhdq8nh4src2qbx 2406741 2406740 2026-04-28T10:03:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P91754 and previous config saved to /var/cache/conftool/dbconfig/20260428-100348-fceratto.json 2406741 wikitext text/x-wiki al9w7tj35mh2m0jeq159gdts01c1a6k 2406743 2406741 2026-04-28T10:08:37Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2406743 wikitext text/x-wiki o0uxvjrlefiactpdvb51ahkwwud1vuo 2406744 2406743 2026-04-28T10:08:44Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2406744 wikitext text/x-wiki ijyes6zbl8ju9adgtzzlnpu79w3t0k6 2406745 2406744 2026-04-28T10:13:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T419635)', diff saved to https://phabricator.wikimedia.org/P91755 and previous config saved to /var/cache/conftool/dbconfig/20260428-101304-fceratto.json 2406745 wikitext text/x-wiki 87co88n2iixo1n1mrkb73i8e2jsbpuw 2406746 2406745 2026-04-28T10:13:24Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2192.codfw.wmnet with reason: Maintenance 2406746 wikitext text/x-wiki b26s77v50mujqo3alcpmz4nw5uzevts 2406747 2406746 2026-04-28T10:13:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 (T419635)', diff saved to https://phabricator.wikimedia.org/P91756 and previous config saved to /var/cache/conftool/dbconfig/20260428-101330-fceratto.json 2406747 wikitext text/x-wiki kkz2gwe7ugxw6uwtj4ekj96oo7bmblk 2406748 2406747 2026-04-28T10:13:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P91757 and previous config saved to /var/cache/conftool/dbconfig/20260428-101356-fceratto.json 2406748 wikitext text/x-wiki 28a2zkp3iq155yw8giwf0mdin5n683o 2406749 2406748 2026-04-28T10:17:55Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2238: after reimage to trixie 2406749 wikitext text/x-wiki qm264s1hg5vjl3yihbn2pdruh1zrjsr 2406750 2406749 2026-04-28T10:19:07Z Stashbot 7414 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2406750 wikitext text/x-wiki eyls52h9rw7kyxgwcs6lzsbs7vm1xy9 2406751 2406750 2026-04-28T10:20:03Z Stashbot 7414 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2406751 wikitext text/x-wiki eajpudvuuqszuczndm724t7xa81hdnb 2406752 2406751 2026-04-28T10:20:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T419635)', diff saved to https://phabricator.wikimedia.org/P91759 and previous config saved to /var/cache/conftool/dbconfig/20260428-102011-fceratto.json 2406752 wikitext text/x-wiki m8z4srtld8tai8ou55ii2itxa23q9u5 2406753 2406752 2026-04-28T10:24:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T419961)', diff saved to https://phabricator.wikimedia.org/P91760 and previous config saved to /var/cache/conftool/dbconfig/20260428-102404-fceratto.json 2406753 wikitext text/x-wiki d369kqx5nkol0fp47pdf7g3m76lh4kp 2406754 2406753 2026-04-28T10:24:22Z Stashbot 7414 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2406754 wikitext text/x-wiki feybe1hx8wx03wfq8qqlgvuw9z55aqe 2406755 2406754 2026-04-28T10:25:08Z Stashbot 7414 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2406755 wikitext text/x-wiki f1pvu6irfa6bghczq114puiuvihru2w 2406756 2406755 2026-04-28T10:29:58Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2406756 wikitext text/x-wiki t5ke6nt125px0iecspqc7toe8yyy24c 2406757 2406756 2026-04-28T10:30:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P91761 and previous config saved to /var/cache/conftool/dbconfig/20260428-103019-fceratto.json 2406757 wikitext text/x-wiki dg5ojnxvo340d6sxwfcp5jkv7xo2wkl 2406758 2406757 2026-04-28T10:31:31Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2406758 wikitext text/x-wiki 7e380wesz75hug1hvn4e746gz5agpno 2406759 2406758 2026-04-28T10:32:11Z Stashbot 7414 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2406759 wikitext text/x-wiki 3n36o62n89huik2kwh5uqor8arl6704 2406760 2406759 2026-04-28T10:32:30Z Stashbot 7414 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2406760 wikitext text/x-wiki ouihfy65exj5joqe5kaxefagp0tvl71 2406762 2406760 2026-04-28T10:39:22Z Stashbot 7414 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2406762 wikitext text/x-wiki enkp20lfe5587k4kmkfjaw89x5hu2fn 2406763 2406762 2026-04-28T10:39:55Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2406763 wikitext text/x-wiki i4in5j0ewpjylkjns0tftjml2ldbadz 2406764 2406763 2026-04-28T10:40:00Z Stashbot 7414 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2406764 wikitext text/x-wiki j8mrgpqii3s4uomk99u7vryoc4uvryl 2406765 2406764 2026-04-28T10:40:24Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2406765 wikitext text/x-wiki 6wcmxba92kqbhqucaaaz1pbqxp2xja8 2406767 2406765 2026-04-28T10:40:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P91762 and previous config saved to /var/cache/conftool/dbconfig/20260428-104027-fceratto.json 2406767 wikitext text/x-wiki jakc8jzhkd8gzyabhnbosf92shnp8xm 2406768 2406767 2026-04-28T10:42:25Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2406768 wikitext text/x-wiki 1tsp3vmqq06r5pcaro9bjghs3mds0qs 2406769 2406768 2026-04-28T10:42:43Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2406769 wikitext text/x-wiki k2br6opaelhbk1ev61kewhi06iofwiz 2406774 2406769 2026-04-28T10:50:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T419635)', diff saved to https://phabricator.wikimedia.org/P91763 and previous config saved to /var/cache/conftool/dbconfig/20260428-105035-fceratto.json 2406774 wikitext text/x-wiki sbuob10dufljl0qaxb7ptl0g6jrz837 2406775 2406774 2026-04-28T10:50:54Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2201.codfw.wmnet with reason: Maintenance 2406775 wikitext text/x-wiki 3agjmj80k83xvbbe8o7mnoo77443d6x 2406778 2406775 2026-04-28T10:57:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2211.codfw.wmnet with reason: Maintenance 2406778 wikitext text/x-wiki 64ue87zbgeu5033msjtalc4meowzh9q 2406779 2406778 2026-04-28T10:57:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 (T419635)', diff saved to https://phabricator.wikimedia.org/P91764 and previous config saved to /var/cache/conftool/dbconfig/20260428-105733-fceratto.json 2406779 wikitext text/x-wiki 7biuf6yrxlxqa0gj9me2augwtyhqajx 2406780 2406779 2026-04-28T11:05:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T419635)', diff saved to https://phabricator.wikimedia.org/P91765 and previous config saved to /var/cache/conftool/dbconfig/20260428-110543-fceratto.json 2406780 wikitext text/x-wiki n8zi87i6pbhkcrd5q9jic0e7ugjb9w9 2406781 2406780 2026-04-28T11:11:13Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2192 with weight 0 T424521', diff saved to https://phabricator.wikimedia.org/P91766 and previous config saved to /var/cache/conftool/dbconfig/20260428-111110-marostegui.json 2406781 wikitext text/x-wiki 0kglbf0c8p6o4m82ryfarp15a109fwb 2406782 2406781 2026-04-28T11:11:32Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s5 T424521 2406782 wikitext text/x-wiki pmpniowdpadickfpwnfd9nk19p7gn56 2406785 2406782 2026-04-28T11:15:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P91767 and previous config saved to /var/cache/conftool/dbconfig/20260428-111551-fceratto.json 2406785 wikitext text/x-wiki plhyl4xv1v7fh4mp9byrudii9lvl51g 2406786 2406785 2026-04-28T11:16:36Z Stashbot 7414 marostegui: Starting s5 codfw failover from db2213 to db2192 - T424521 2406786 wikitext text/x-wiki q87v4jlfb6xppcnofwu3wd8hdi7migg 2406787 2406786 2026-04-28T11:17:02Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2192 to s5 primary T424521', diff saved to https://phabricator.wikimedia.org/P91768 and previous config saved to /var/cache/conftool/dbconfig/20260428-111658-marostegui.json 2406787 wikitext text/x-wiki 77tuvbutopvm9nih6gw8l4cyfik6ejy 2406788 2406787 2026-04-28T11:17:44Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2213 T424521', diff saved to https://phabricator.wikimedia.org/P91769 and previous config saved to /var/cache/conftool/dbconfig/20260428-111740-marostegui.json 2406788 wikitext text/x-wiki abo95tm4ws2azhtjg2yrmsw9pwh27n9 2406789 2406788 2026-04-28T11:20:41Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2213.codfw.wmnet with reason: Reimage to Trixie 2406789 wikitext text/x-wiki sp8vrnygjo8x8dj26ukuni19yxlrnto 2406790 2406789 2026-04-28T11:20:47Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2213: Reimage to Trixie 2406790 wikitext text/x-wiki sz471gut7qz0vdbp0vvc3q48p4soy0u 2406791 2406790 2026-04-28T11:20:55Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2213: Reimage to Trixie 2406791 wikitext text/x-wiki m4xmpwgbsp2djdrz7n7hs0spii7sq20 2406792 2406791 2026-04-28T11:22:54Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2213.codfw.wmnet with OS trixie 2406792 wikitext text/x-wiki pwvguasbhglj0bf6b5eqatt2euirrtu 2406794 2406792 2026-04-28T11:26:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P91770 and previous config saved to /var/cache/conftool/dbconfig/20260428-112558-fceratto.json 2406794 wikitext text/x-wiki 8myz1v3ec178ph2yre4uid1kjsxbl2a 2406797 2406794 2026-04-28T11:36:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T419635)', diff saved to https://phabricator.wikimedia.org/P91772 and previous config saved to /var/cache/conftool/dbconfig/20260428-113606-fceratto.json 2406797 wikitext text/x-wiki cakrfpzc9vkvm2tt6wtl8ekvgts897y 2406798 2406797 2026-04-28T11:36:17Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2223.codfw.wmnet with reason: Maintenance 2406798 wikitext text/x-wiki 11l66zw64y40e0h50ux54wijhr29gww 2406799 2406798 2026-04-28T11:36:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 (T419635)', diff saved to https://phabricator.wikimedia.org/P91773 and previous config saved to /var/cache/conftool/dbconfig/20260428-113623-fceratto.json 2406799 wikitext text/x-wiki aorisuqwtogmgjsvcj6za5o60pxn4db 2406801 2406799 2026-04-28T11:38:20Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply 2406801 wikitext text/x-wiki 9w927q6a2jn3203t93vl4jq04sg2dbx 2406802 2406801 2026-04-28T11:38:26Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply 2406802 wikitext text/x-wiki 8vuer1uabteczw4p4wc2wtp61hohe8p 2406804 2406802 2026-04-28T11:38:49Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply 2406804 wikitext text/x-wiki ia8nko20qwfhy6a349gg08b69j6i95w 2406805 2406804 2026-04-28T11:38:52Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply 2406805 wikitext text/x-wiki q6ed8nxzopqyhdvau7aish6e14gx0u2 2406806 2406805 2026-04-28T11:39:20Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2406806 wikitext text/x-wiki 53acke72w8cmu3b6ldmga865mgdt7ih 2406807 2406806 2026-04-28T11:39:49Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2406807 wikitext text/x-wiki r6n1ft75xi5wyoh3xrwqfz9e8204hyn 2406808 2406807 2026-04-28T11:39:51Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406808 wikitext text/x-wiki cay56fxgfha9x5so9iq3itrpltyn05d 2406809 2406808 2026-04-28T11:39:57Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2406809 wikitext text/x-wiki cs6lo3j9pvuj8x56v6m9xqqh9c5xow5 2406810 2406809 2026-04-28T11:39:59Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406810 wikitext text/x-wiki d9yiw3w9c3yqatge01ntxd2z197y9r8 2406811 2406810 2026-04-28T11:40:10Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406811 wikitext text/x-wiki 244aji5qtuy61hwm3q1xhi0b02o70uy 2406812 2406811 2026-04-28T11:40:13Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406812 wikitext text/x-wiki b9kwzp6ywkarbd21zxvupadt22eogme 2406813 2406812 2026-04-28T11:40:25Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2406813 wikitext text/x-wiki 4vu9e2rhamt86ben74t0uvp967zykdm 2406814 2406813 2026-04-28T11:40:27Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406814 wikitext text/x-wiki hd9zaed0skpqoplqd2xycdjk0s2s6fc 2406815 2406814 2026-04-28T11:40:34Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406815 wikitext text/x-wiki 5lxubr2hr30c39myaosr4ocnd5zl5dn 2406816 2406815 2026-04-28T11:43:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2213.codfw.wmnet with reason: host reimage 2406816 wikitext text/x-wiki a9z6mn70syynl33xeeafr70gbkq6rlx 2406817 2406816 2026-04-28T11:44:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T419635)', diff saved to https://phabricator.wikimedia.org/P91774 and previous config saved to /var/cache/conftool/dbconfig/20260428-114434-fceratto.json 2406817 wikitext text/x-wiki oqbh9u6pyuw46lc3ilpiesma9j1g61c 2406818 2406817 2026-04-28T11:49:20Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2213.codfw.wmnet with reason: host reimage 2406818 wikitext text/x-wiki 4g2bvf7d7a037w67vjks890890qvm4t 2406820 2406818 2026-04-28T11:54:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P91775 and previous config saved to /var/cache/conftool/dbconfig/20260428-115442-fceratto.json 2406820 wikitext text/x-wiki c1p2nfac1d43b6dflbvdegzptbefqld 2406821 2406820 2026-04-28T11:57:09Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2406821 wikitext text/x-wiki jgd1r24ziic26qfr5poy11r75m5pc3w 4 8341 2406558 83723 2026-04-28T02:10:07Z Star walker 40724 correct licence 2406558 wikitext text/x-wiki bpr5yfajn9xs0skx463hdshcvop3bde 0 11689 2406297 2398535 2026-04-27T14:41:05Z BRouberol-WMF 39141 /* Upgrade the Kafka inter-broker protocol version */ 2406297 wikitext text/x-wiki l9jrtjohss20fjpry2yq96adr82mvwg 2406307 2406297 2026-04-27T14:51:30Z KHerron (WMF) 10463 /* Upgrade the Kafka inter-broker protocol version */ add example one-liners 2406307 wikitext text/x-wiki 04ijet5oa1kpq0lof17nne4c4u06uq4 2406308 2406307 2026-04-27T14:51:43Z KHerron (WMF) 10463 /* Upgrade the Kafka inter-broker protocol version */ 2406308 wikitext text/x-wiki kkmtloc5d0zdj6hu9nv8sq7jlscrb4l 2406348 2406308 2026-04-27T15:21:20Z LToscano (WMF) 7191 /* Upgrade the Kafka inter-broker protocol version */ 2406348 wikitext text/x-wiki g4v8i49k2nb40ptf83vrv2h72u2ekrd 0 17290 2406193 2406084 2026-04-27T12:38:35Z Stashbot 7414 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 383 2406193 wikitext text/x-wiki cr5pzqx16eqbvl0stn0j4j9a6dh7a2g 2406294 2406193 2026-04-27T14:32:30Z Stashbot 7414 James_F: Zuul: Drop PHP 8.2 enforcement from MediaWiki things for master and REL1_46 for T358667 2406294 wikitext text/x-wiki 82vwhtmgsq08p0toi5gvm3eh2yoe0uz 2406394 2406294 2026-04-27T15:53:53Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/ReportIncident] Add TestKitchen phan dependency, for T424220 2406394 wikitext text/x-wiki hl9x3n7mmjoazzo8uqocyslo4jimpc6 2406447 2406394 2026-04-27T17:16:07Z Stashbot 7414 James_F: Docker: [mediawiki-phan-taint-check-demo] Re-platform to Trixie and so PHP 8.4 2406447 wikitext text/x-wiki t19x21fpxget6jq83qkk855daa21ejb 2406513 2406447 2026-04-27T20:45:17Z Stashbot 7414 James_F: Zuul: Restrict mw*-codehealth-patch jobs to master only, for T424573 2406513 wikitext text/x-wiki kcdim8gu2dcqwlhu37qcpettk3ujl1n 2406532 2406513 2026-04-27T21:57:18Z Stashbot 7414 bd808: Add web security group to deployment-deploy04 (T256168) 2406532 wikitext text/x-wiki mt7n8s0jzmd5cmcgap7er2mlra4rc24 2406535 2406532 2026-04-27T22:11:42Z Stashbot 7414 bd808: Beta Cluster MediaWiki update logs now available via https://beta-update.wmcloud.org/ (T256168) 2406535 wikitext text/x-wiki k7gx8mz4sjbv8f4pt5fhurtpq6efa2c 2406553 2406535 2026-04-28T00:03:50Z Stashbot 7414 bd808: Increase parallelism for wmf-beta-update-databases.py (T256168) 2406553 wikitext text/x-wiki 6nwo7h8m2fewcg3ng6rcehlfsj3hpe0 2406766 2406553 2026-04-28T10:40:26Z Stashbot 7414 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud to fix failure seen in mwext-node24-rundoc #1717 2406766 wikitext text/x-wiki 0yiweeem77xhxlo5xnvk338knajsq42 0 20365 2406554 2399910 2026-04-28T01:13:19Z Krinkle 44 /* Directly */ attempt to document how to target mw-experiment and mw-debug over HTTP. Add fixme note that targeting Apache/php-fpm doesn't work, only through local K8s and Envoy proxies 2406554 wikitext text/x-wiki 5uitgrqks2tmjj3um7jfyxxygzk358a 2406555 2406554 2026-04-28T01:25:20Z Krinkle 44 2406555 wikitext text/x-wiki 8e7vr8jut9mkr3xnn4d0roisj78yu7k 0 22139 2406534 2383038 2026-04-27T22:10:50Z Neil Shah-Quinn (WMF) 36601 Link to [[Data Platform/Data Lake|Data Lake]] rather than [[Data Platform/Systems/Cluster|Analytics cluster]] in the intro 2406534 wikitext text/x-wiki opz4tuw2u0kxgab3y29lmp5cle8rrsz 498 30942 2406195 2404859 2026-04-27T12:43:31Z Stashbot 7414 godog: upgrade spicerack on cloudcumin 2406195 wikitext text/x-wiki 2v4f5sn5mnizekbfx2r6s6avxnwt7zk 2406657 2406195 2026-04-28T08:18:21Z Stashbot 7414 wmbot~godog@r5: START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' 2406657 wikitext text/x-wiki q1miffh0a1llghwpedumgfdvteujk71 2406658 2406657 2026-04-28T08:18:50Z Stashbot 7414 wmbot~godog@r5: END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' 2406658 wikitext text/x-wiki k9xlktzno3u5rc1kyck8a1x9eqc7xh7 2406659 2406658 2026-04-28T08:19:11Z Stashbot 7414 wmbot~godog@r5: START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' 2406659 wikitext text/x-wiki pv1pw4rrzmc7c0gmxg2qryatoewt4lq 2406660 2406659 2026-04-28T08:19:23Z Stashbot 7414 wmbot~godog@r5: END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' 2406660 wikitext text/x-wiki klaj3kqufcw0sownsxhpp7ehdsw3i3f 2406665 2406660 2026-04-28T08:28:59Z Stashbot 7414 wmbot~godog@r5: START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' 2406665 wikitext text/x-wiki 5aq8v8272unh0wbdw8pnzcdcbpsoub7 2406666 2406665 2026-04-28T08:29:11Z Stashbot 7414 wmbot~godog@r5: END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' 2406666 wikitext text/x-wiki 14rpyilelymclukbvs5v183zqi9irlj 2406671 2406666 2026-04-28T08:32:31Z Stashbot 7414 wmbot~godog@r5: START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' 2406671 wikitext text/x-wiki 9ajkcxeovjtrc9co4afhqzhzsaku9ww 2406672 2406671 2026-04-28T08:32:47Z Stashbot 7414 wmbot~godog@r5: END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' 2406672 wikitext text/x-wiki 6efm732o72owv748zv852ixka2idoeq 2406679 2406672 2026-04-28T08:42:16Z Stashbot 7414 wmbot~godog@r5: START - Cookbook wmcs.openstack.rack_resources on cluster 'codfw1dev' 2406679 wikitext text/x-wiki ffe07fo1r0zhhc4ete8no4kogiuihjo 2406682 2406679 2026-04-28T08:42:48Z Stashbot 7414 wmbot~godog@r5: END (PASS) - Cookbook wmcs.openstack.rack_resources (exit_code=0) on cluster 'codfw1dev' 2406682 wikitext text/x-wiki i6dv7sxlemn5e6fjzuweag1zh4yhojt 498 55577 2406733 2390689 2026-04-28T09:46:57Z Stashbot 7414 volans: killed 22 zombies pods stuck in Terminating status since many days and with no running container/process 2406733 wikitext text/x-wiki re1m5abjowxphx0zrlhmhnbd8yj6hh9 0 440607 2406538 2377535 2026-04-27T22:23:21Z Neil Shah-Quinn (WMF) 36601 Add basic menu of options for running ETLs 2406538 wikitext text/x-wiki 4h3ygo2auh8knvb4sppfr74c1dyx0bu 2406544 2406538 2026-04-27T22:37:56Z Neil Shah-Quinn (WMF) 36601 /* ETL */ Add option of manual ETL using Pandas 2406544 wikitext text/x-wiki 9awnt58o6xlk4yqz5ym04nvxiox4mcy 2406545 2406544 2026-04-27T22:38:34Z Neil Shah-Quinn (WMF) 36601 /* Manually using Pandas */ Simplify wording 2406545 wikitext text/x-wiki rrx9w54t23qmtk0irohq95mwc7bk358 0 444523 2406539 2401083 2026-04-27T22:31:24Z Neil Shah-Quinn (WMF) 36601 Remove references to Hive as it is deprecated 2406539 wikitext text/x-wiki 7ngatsme8zm6thnnwylxqg7e4e9xadr 0 446051 2406324 2401326 2026-04-27T15:07:02Z Ottomata 88 /* EventStreams */ 2406324 wikitext text/x-wiki swrxe5z9da38k9kpxd47cprtmkng1sx 0 449160 2406208 2406072 2026-04-27T13:08:49Z Dexbot 30554 Bot: Updating the report 2406208 wikitext text/x-wiki a4umabke0cxhrinl0zypmjyzbz70nzh 2406303 2406208 2026-04-27T14:48:20Z Dexbot 30554 Bot: Updating the report 2406303 wikitext text/x-wiki 3xhkzrl9ziozo1hlrer0qhtkpoiveat 2406552 2406303 2026-04-28T00:01:44Z Dexbot 30554 Bot: Updating the report 2406552 wikitext text/x-wiki qv6fnu4pu7jr5zzcxiiazwlvn80bz0u 2406563 2406552 2026-04-28T05:05:25Z Dexbot 30554 Bot: Updating the report 2406563 wikitext text/x-wiki r1z7hy59q93mb6lhipawi1lk7crugac 2406651 2406563 2026-04-28T07:56:49Z Dexbot 30554 Bot: Updating the report 2406651 wikitext text/x-wiki 57ee7medv1kjqd2ueyo4eg1c3sp3pyz 2406669 2406651 2026-04-28T08:29:56Z Dexbot 30554 Bot: Updating the report 2406669 wikitext text/x-wiki jtp9qitcu8z67i5hk4ss0a60wwjx9pf 2406784 2406669 2026-04-28T11:12:28Z Dexbot 30554 Bot: Updating the report 2406784 wikitext text/x-wiki hq6qmggyuieaxqhcazaqgiier89pkk3 0 451605 2406742 2370957 2026-04-28T10:05:55Z CMooney (WMF) 24288 /* QoS Classes */ 2406742 wikitext text/x-wiki kizhyntrsrc69c6wupgdqwb5x9qoc5c 1 452704 2406216 2078892 2026-04-27T13:23:12Z A smart kitten 38172 /* Cross-referencing to an email I sent to wikitech-l regarding the status of this page */ new section 2406216 wikitext text/x-wiki 507154jbfnitnwxqbtri47i8kv8smew 0 453749 2406793 2377685 2026-04-28T11:25:11Z RCoccioli (WMF) 7416 /* Updating the image */ Add --no-ff for clarity and to override eventual local stricter configs 2406793 wikitext text/x-wiki qv0pxqhud02y36nppa0hh98nq3tt689 116 453906 2406551 2406019 2026-04-27T23:42:29Z Gitlabaccountapprovalbot 37332 @quinlan was approved. 2406551 wikitext text/x-wiki 0fhv811qdie8ocihkzz61cqjj6lku10 0 459699 2406485 2405402 2026-04-27T18:58:47Z Stashbot 7414 ejegg: standalone (IPN listener) SmashPig upgraded from 572b69da to 88a1bcba 2406485 wikitext text/x-wiki 4db00kepbqlilvfp81gbfu89une5728 2406488 2406485 2026-04-27T19:02:25Z Stashbot 7414 ejegg: payments-wiki upgraded from b1a352af to 5265089d 2406488 wikitext text/x-wiki jdy86ysu8x5h3cw3rekzzsdu6wkvx8s 2406489 2406488 2026-04-27T19:05:00Z Stashbot 7414 ejegg: fundraising civicrm upgraded from 3f8d49fa to be3bb76b 2406489 wikitext text/x-wiki 75zy2pxjabsgwsc8wuo871g9fpt2lym 0 459719 2406365 2404163 2026-04-27T15:34:09Z BTullis (WMF) 25295 /* Service Level Indicators (SLIs) */ Exclude Bulk indexing queries from the latency SLI 2406365 wikitext text/x-wiki col4dn65uus6qihn6p6auakvynw538x 0 459966 2406229 2396592 2026-04-27T13:45:09Z MPopov (WMF) 5891 Include example in intro 2406229 wikitext text/x-wiki taci08o69hjd7u0thcb3gmrs9lpk5xw 0 460059 2406536 2402422 2026-04-27T22:13:37Z Neil Shah-Quinn (WMF) 36601 Remove unnecessary headings 2406536 wikitext text/x-wiki iy1d0n5svkw95400rrmv3gp5jdi325z 2 460094 2406168 2026-04-27T12:09:52Z DChan (WMF) 54599 Initial draft 2406168 wikitext text/x-wiki 1ncmg7y041pnxgyl27sxok6a8l1nsrd 3 460095 2406413 2026-04-27T16:06:32Z StrikerBot 8475 Welcome to Toolforge! 2406413 wikitext text/x-wiki apyf5ur0c8y27apqj6xu8xos09qdx1d 3 460096 2406415 2026-04-27T16:07:36Z StrikerBot 8475 Welcome to Toolforge! 2406415 wikitext text/x-wiki 66n65akr55arow6ha9oiqr4vntirn47 2 460097 2406674 2026-04-28T08:40:09Z Effie Mouzeli (WMF) 12880 Created page with "TBA" 2406674 wikitext text/x-wiki kd1620ukml0ssm2lahetsf8y997ay3c 2406684 2406674 2026-04-28T08:46:40Z Effie Mouzeli (WMF) 12880 2406684 wikitext text/x-wiki 5tteuqvx4jcfz5sn3m93zyj6ep1a4vd 2406697 2406684 2026-04-28T09:03:10Z Effie Mouzeli (WMF) 12880 2406697 wikitext text/x-wiki cvclkhi3hqiq1x6qpryi1npjw4pe46j 2406700 2406697 2026-04-28T09:05:58Z Effie Mouzeli (WMF) 12880 2406700 wikitext text/x-wiki rrfrlpjlxjh57jjr7j95sk9fdyaxjgv 2406701 2406700 2026-04-28T09:06:35Z Effie Mouzeli (WMF) 12880 2406701 wikitext text/x-wiki 8j8a7q7gc12vmj1pl8c3xcnfbwqcsl7 498 460098 2406688 2026-04-28T08:56:06Z Stashbot 7414 volans@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 8 cores, 34816 ram 2406688 wikitext text/x-wiki i7vxe3l7jplz5pbcxgseruwvhzcreu8 2406689 2406688 2026-04-28T08:56:12Z Stashbot 7414 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 8 cores, 34816 ram 2406689 wikitext text/x-wiki 0vh0j0pshn403s45085b5tzvfd3k76i 3 460099 2406692 2026-04-28T08:59:40Z StrikerBot 8475 Welcome to Toolforge! 2406692 wikitext text/x-wiki mlkzcjtg2ahv9ph3cwl0guxsfxym7wu 2 460100 2406693 2026-04-28T09:00:04Z Effie Mouzeli (WMF) 12880 Created page with "{{Draft}} = SCROLL: <!-- replace with service name --> = ''Service Checklist for Readiness, Operations, Launch and Lifecycle'' {| class="wikitable" style="font-size: 90%;" | '''Service''' || <!-- service name --> |- | '''Owner''' || [[<!-- team page -->]] |- | '''SCROLL bearer''' || <!-- @sre-reviewer --> |- | '''Target''' || <!-- target launch date --> |- | '''Scroll Epic''' || [[phab:T<!-- task id -->|T<!-- task id -->]] |} == 1. Service summ..." 2406693 wikitext text/x-wiki qy3w6v94vrbt3k8s3lrld2v7s5fuynb 2406773 2406693 2026-04-28T10:50:34Z Effie Mouzeli (WMF) 12880 /* SCROLL: */ 2406773 wikitext text/x-wiki 30mrok1v8wt749la6suwt80btarezlb 2406776 2406773 2026-04-28T10:56:40Z Effie Mouzeli (WMF) 12880 2406776 wikitext text/x-wiki 9iemryuv0c6teirvclkomhzn65k9t3q 2406796 2406776 2026-04-28T11:34:29Z Effie Mouzeli (WMF) 12880 2406796 wikitext text/x-wiki 4wo08yjpo8rygpsjm6qbrs9o42r14zn 2406822 2406796 2026-04-28T11:57:46Z Effie Mouzeli (WMF) 12880 2406822 wikitext text/x-wiki gwr270wdq0ii641bsybrq65h7kab7an 2 460101 2406783 2026-04-28T11:12:05Z Effie Mouzeli (WMF) 12880 Created page with ".scroll-placeholder { color: #aaa; font-style: italic; } /* Status Small coloured badges for the Status column. Usage: <span class="scroll-status scroll-status-done">done</span> Values: done | in-progress | todo | blocker */ .scroll-status { display: inline-block; padding: 2px 9px; border-radius: 10px; font-size: 85%; font-weight: bold; white-space: nowrap; } .scroll-status-done { background: #d4edda; color: #2d6a3f; }..." 2406783 css text/css sj17j03yrhp4wvuertpjk1yt4xhyl99 2406795 2406783 2026-04-28T11:32:57Z Effie Mouzeli (WMF) 12880 Effie Mouzeli (WMF) changed the content model of the page [[User:Effie Mouzeli (WMF)/SCROLL/styles.css]] from "CSS" to "Sanitized CSS": need to test https://phabricator.wikimedia.org/T424178 (Production Readiness Checklist) 2406795 sanitized-css text/css sj17j03yrhp4wvuertpjk1yt4xhyl99 2406800 2406795 2026-04-28T11:37:57Z Effie Mouzeli (WMF) 12880 Protected "[[User:Effie Mouzeli (WMF)/SCROLL/styles.css]]" ([Edit=Allow only autoconfirmed users] (indefinite) [Move=Allow only autoconfirmed users] (indefinite)) 2406795 sanitized-css text/css sj17j03yrhp4wvuertpjk1yt4xhyl99 2406803 2406800 2026-04-28T11:38:40Z Effie Mouzeli (WMF) 12880 2406803 sanitized-css text/css ar6df8j7hqjo466vyth8n5cdg8m0gmx 2406819 2406803 2026-04-28T11:51:52Z Effie Mouzeli (WMF) 12880 2406819 sanitized-css text/css ktx2pa4apfg5mqb99i4zg67xz5btx4e