Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.8 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2430344 2429530 2026-06-24T16:28:34Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api 2430344 wikitext text/x-wiki tw6t72recvrgwqdxtprl5zfjpjhn8l3 2430346 2430344 2026-06-24T16:34:18Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api 2430346 wikitext text/x-wiki 0j46y6nehq8oayfp7807chyyi4rvov5 2430540 2430346 2026-06-25T08:40:50Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T429907) 2430540 wikitext text/x-wiki phn9giovi9mgmsic3r5olgatc2tl1yn 2430553 2430540 2026-06-25T08:54:16Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T429907) 2430553 wikitext text/x-wiki 5251ripcpiytg17wk3upwvel9zckr3o 0 4108 2430143 2430090 2026-06-24T12:31:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1304695]] to Wednesday, June 24 UTC afternoon backport window 2430143 wikitext text/x-wiki epzqwa2d61r6rrbyzbi4twx0nzaazgz 2430144 2430143 2026-06-24T12:32:59Z ScheduleDeploymentBot 37566 Add [[gerrit:1304001]] to Wednesday, June 24 UTC afternoon backport window 2430144 wikitext text/x-wiki 5o55pslhmx4rhq236wl64wj5hlipxlg 2430200 2430144 2026-06-24T13:03:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1305407]] to Wednesday, June 24 UTC afternoon backport window 2430200 wikitext text/x-wiki teq5k8i07jir3pcnywkvoayorg3q47f 2430201 2430200 2026-06-24T13:03:20Z ScheduleDeploymentBot 37566 Add [[gerrit:1305408]] to Wednesday, June 24 UTC afternoon backport window 2430201 wikitext text/x-wiki kdcsrcblubmwu1yag3bxzfwrzu4a0hd 2430211 2430201 2026-06-24T13:20:06Z ScheduleDeploymentBot 37566 Add [[gerrit:1304919]] to Wednesday, June 24 UTC afternoon backport window 2430211 wikitext text/x-wiki 0fe168szmojmv54ncyafhw7ibcy1ms0 2430457 2430211 2026-06-24T20:15:47Z ScheduleDeploymentBot 37566 Add [[gerrit:1305500]] to Wednesday, June 24 UTC late backport window 2430457 wikitext text/x-wiki kpq5a3h68ws789nquujj5aamrf0ow6h 498 6726 2430339 2429514 2026-06-24T16:05:32Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api 2430339 wikitext text/x-wiki 672pq1vnm2b4y392av9q9h9a16ge0uw 2430340 2430339 2026-06-24T16:10:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api 2430340 wikitext text/x-wiki 01zxi2tisckakf5t3wawanljgwdoied 2430350 2430340 2026-06-24T16:39:28Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T429907) 2430350 wikitext text/x-wiki lcefuk6gdzj3gxflganat9m610og16r 2430356 2430350 2026-06-24T16:51:15Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T429907) 2430356 wikitext text/x-wiki qc7513f16m65xtnrq0aovvzbza9rnd8 0 7919 2430116 2430113 2026-06-24T12:01:50Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 23 hosts with reason: Switch maintenance 2430116 wikitext text/x-wiki so2cudkf9wcg5tf3n4u5ianzw1n5pb1 2430117 2430116 2026-06-24T12:02:48Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a6-codfw,lsw1-a6-codfw IPv6,lsw1-a6-codfw.mgmt with reason: Switch maintenance 2430117 wikitext text/x-wiki 1h1rrrfaxng0ovskbztiwz6coo8xgeg 2430118 2430117 2026-06-24T12:02:54Z Stashbot 7414 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A6 2430118 wikitext text/x-wiki t3h67tzpfh2mo8ld5pbhnokxlhuksh0 2430119 2430118 2026-06-24T12:03:23Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2430119 wikitext text/x-wiki 1g3wzaggrld4l6rdcbvkxrruur79yj1 2430120 2430119 2026-06-24T12:03:56Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2155: rack depool 2430120 wikitext text/x-wiki 6q2649mnxdd06irz8xclze6v8qnddfc 2430121 2430120 2026-06-24T12:04:16Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2155: rack depool 2430121 wikitext text/x-wiki s6yu6pqc16nqsrrbt7upum4ft5p6w3p 2430122 2430121 2026-06-24T12:04:30Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2156: rack depool 2430122 wikitext text/x-wiki pyr2e1qa5sotvxashs6p8ybeem9rbeh 2430123 2430122 2026-06-24T12:05:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2156: rack depool 2430123 wikitext text/x-wiki clx86l8myu1slv97ixjyf769o0g154v 2430124 2430123 2026-06-24T12:05:12Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2430124 wikitext text/x-wiki izv6zzaog7offptepvti26q2ffjsu5w 2430125 2430124 2026-06-24T12:05:57Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2067,2072-2073,2114-2115,2124-2127,2256-2257].codfw.wmnet 2430125 wikitext text/x-wiki 72bd8br235cp82gw257e9nrn50nvahs 2430126 2430125 2026-06-24T12:06:24Z Stashbot 7414 dcausse: T423993: closing ttmserver indices in the cirrussearch opensearch cluster (eqiad & codfw) 2430126 wikitext text/x-wiki cdwp4uun5w9xnsazetfzznhodul5qhg 2430127 2430126 2026-06-24T12:07:21Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2430127 wikitext text/x-wiki h00g36xlbaybdisbv2etj15yz3gkcdh 2430128 2430127 2026-06-24T12:07:23Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet 2430128 wikitext text/x-wiki dgflxa5gtxakx9pb83m642nbp2tffyk 2430129 2430128 2026-06-24T12:07:26Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet 2430129 wikitext text/x-wiki 81yyqgcqwz8y7g99giwft9osoo5k1d0 2430130 2430129 2026-06-24T12:07:39Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet 2430130 wikitext text/x-wiki ic9hx9zdarj4stssy8w8w8lxsr3a4p5 2430131 2430130 2026-06-24T12:09:17Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2430131 wikitext text/x-wiki 2r9cyy4bc4mhwe1yzk7tk4o6o1d619d 2430132 2430131 2026-06-24T12:09:29Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2430132 wikitext text/x-wiki r4tw1enb6i22qm9y49736cw48vwcflo 2430133 2430132 2026-06-24T12:10:37Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2430133 wikitext text/x-wiki 7rq83tbh9lb2v3n66h6f6id9pfpugwu 2430134 2430133 2026-06-24T12:11:22Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2430134 wikitext text/x-wiki cqtbmerlxnd99ami1vqre9xbviyfmrm 2430135 2430134 2026-06-24T12:16:14Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . 2430135 wikitext text/x-wiki 5fh8va26lyr11eou7s6ni0ei0bbmpzs 2430136 2430135 2026-06-24T12:17:12Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2067,2072-2073,2114-2115,2124-2127,2256-2257].codfw.wmnet 2430136 wikitext text/x-wiki tgd13rikz6loxffo0fuiwuluikexrrs 2430137 2430136 2026-06-24T12:17:46Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet 2430137 wikitext text/x-wiki 4wsqvgdhfyud2yt7h7zvcjy05gokavu 2430138 2430137 2026-06-24T12:23:56Z Stashbot 7414 cgoubert@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Testing apiportalwiki deletion in beta - T418494 (duration: 98m 50s) 2430138 wikitext text/x-wiki 9rwbwxfvgs918mfl5cefhc67m9xc5t7 2430139 2430138 2026-06-24T12:26:58Z Stashbot 7414 cgoubert@deploy1003: Started scap sync-world: Backport for [[gerrit:1304845|Remove config related to the API Portal (T429372 T418494)]], [[gerrit:1305380|CommonSettings-labs: Remove api.wikimedia.beta.wmcloud.org (T429372 T418494)]] 2430139 wikitext text/x-wiki 16ktevu0lqhc1z85xtbgz1b3b1xcp14 2430140 2430139 2026-06-24T12:29:06Z Stashbot 7414 cgoubert@deploy1003: apaskulin, cgoubert: Backport for [[gerrit:1304845|Remove config related to the API Portal (T429372 T418494)]], [[gerrit:1305380|CommonSettings-labs: Remove api.wikimedia.beta.wmcloud.org (T429372 T418494)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430140 wikitext text/x-wiki hnuls7w4rrk068k5xx0vbco9srrs2a6 2430141 2430140 2026-06-24T12:31:11Z Stashbot 7414 cgoubert@deploy1003: apaskulin, cgoubert: Continuing with deployment 2430141 wikitext text/x-wiki 403d2y58x0t3fy5vv1fldhpyep54k1c 2430142 2430141 2026-06-24T12:31:24Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host db1208.eqiad.wmnet 2430142 wikitext text/x-wiki 03bkbjgx1k4hz0bt6lc995cmiovq3or 2430145 2430142 2026-06-24T12:33:57Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet 2430145 wikitext text/x-wiki bmi93s30un2vrr5t7ho5kwyglur6prf 2430146 2430145 2026-06-24T12:33:59Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet 2430146 wikitext text/x-wiki k08866lfuxwlcp055ladvjy52ysf5tp 2430147 2430146 2026-06-24T12:34:07Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet 2430147 wikitext text/x-wiki cx2kd1o0gzefdiomjfndpf664o9cz5i 2430148 2430147 2026-06-24T12:34:10Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet 2430148 wikitext text/x-wiki kkduuj08l4xu85dt7x65f8bvka0crod 2430149 2430148 2026-06-24T12:35:23Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2155: rack depool 2430149 wikitext text/x-wiki 3g74dclduvj9egz13yfus7uqkib1wen 2430150 2430149 2026-06-24T12:35:33Z Stashbot 7414 cgoubert@deploy1003: Finished scap sync-world: Backport for [[gerrit:1304845|Remove config related to the API Portal (T429372 T418494)]], [[gerrit:1305380|CommonSettings-labs: Remove api.wikimedia.beta.wmcloud.org (T429372 T418494)]] (duration: 08m 34s) 2430150 wikitext text/x-wiki s6vkrco19x4eod9gvchf04hxowph8u5 2430151 2430150 2026-06-24T12:36:00Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2067,2072-2073,2114-2115,2124-2127,2256-2257].codfw.wmnet 2430151 wikitext text/x-wiki rkzmrjncd1sd53ahmluy7swcy6hdr6k 2430152 2430151 2026-06-24T12:36:07Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2067,2072-2073,2114-2115,2124-2127,2256-2257].codfw.wmnet 2430152 wikitext text/x-wiki qx68b5wmzws7ls0pjmzkg0tek9iydxz 2430153 2430152 2026-06-24T12:36:43Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2156: rack depool 2430153 wikitext text/x-wiki 5y5481kq962vklezb3r514y4u5e7enl 2430154 2430153 2026-06-24T12:37:46Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430154 wikitext text/x-wiki cmwlfc66avvf8q146ryfipyxj22o4g1 2430155 2430154 2026-06-24T12:37:50Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@eqiad T429893 2430155 wikitext text/x-wiki 8oof6gq1dholqzq03qq71xfn8v32bjw 2430156 2430155 2026-06-24T12:38:06Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1238: Upgrading db1238.eqiad.wmnet 2430156 wikitext text/x-wiki ir9fkgqlicu2uv1ajue9wi1t602z0ov 2430157 2430156 2026-06-24T12:38:32Z Stashbot 7414 claime: Deleting apiportalwiki references in localuser table - T418494 2430157 wikitext text/x-wiki iaydklf7ogec8kavo6kv6sftygx0tvw 2430158 2430157 2026-06-24T12:38:46Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1238: Upgrading db1238.eqiad.wmnet 2430158 wikitext text/x-wiki pyqy6xsbkg7nwmqcps1ywwbkizlpft4 2430159 2430158 2026-06-24T12:38:55Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430159 wikitext text/x-wiki t8w8a5v6c3ejw1zw9kdhcfsu7thqmsy 2430160 2430159 2026-06-24T12:38:59Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@codfw T429893 2430160 wikitext text/x-wiki rvwm2j9isa7rni1tf36426utl120kyc 2430161 2430160 2026-06-24T12:39:16Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2206: Upgrading db2206.codfw.wmnet 2430161 wikitext text/x-wiki 3ckswh78ocrnjfe3da2oe2unx1uu7p3 2430162 2430161 2026-06-24T12:39:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2206: Upgrading db2206.codfw.wmnet 2430162 wikitext text/x-wiki kx3imkcdfcon0cirxx570tfu1exzvx0 2430163 2430162 2026-06-24T12:39:58Z Stashbot 7414 ayounsi@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2155: rack depool 2430163 wikitext text/x-wiki linruya3fsns3er9468hdmpauqgccw2 2430164 2430163 2026-06-24T12:40:42Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp2043.codfw.wmnet 2430164 wikitext text/x-wiki axkh2s6ub9gxidto7lj59uygamqjqz5 2430165 2430164 2026-06-24T12:40:44Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2043.codfw.wmnet 2430165 wikitext text/x-wiki gbve6uihcib72r1y2gpqne0pyjnazxc 2430166 2430165 2026-06-24T12:40:49Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp2044.codfw.wmnet 2430166 wikitext text/x-wiki p6ti2084u4bmssxo35002zpdcq17h2v 2430167 2430166 2026-06-24T12:40:51Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2044.codfw.wmnet 2430167 wikitext text/x-wiki iwc982lf4vzx2c53szz2ie94q4zaf43 2430168 2430167 2026-06-24T12:41:45Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.dns.netbox 2430168 wikitext text/x-wiki 8mal9q69930uhdnj6b339lvmic4s8zu 2430169 2430168 2026-06-24T12:42:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2155: repool after rack maintenance 2430169 wikitext text/x-wiki mi19ctgsdz62o02yny9hn24swp863ns 2430170 2430169 2026-06-24T12:44:36Z Stashbot 7414 claime: Deleting apiportalwiki references in localnames table - T418494 2430170 wikitext text/x-wiki spqbbss9lk7r24ivy4881uhjpb5mo1i 2430171 2430170 2026-06-24T12:44:47Z Stashbot 7414 fabfur: repooling cp2043 and cp2044 after reimage (T419825) 2430171 wikitext text/x-wiki 3atpacfnuq9g2smir4g7ltzfegig8z4 2430172 2430171 2026-06-24T12:45:08Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* 2430172 wikitext text/x-wiki 02fypkmait6dtuwdrc92xv4k9uf43cv 2430173 2430172 2026-06-24T12:45:12Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2044.* 2430173 wikitext text/x-wiki do51aub4c7sqg932qy2atzrclifafea 2430174 2430173 2026-06-24T12:46:26Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new VIP for dumps-nfs - filippo@cumin1003" 2430174 wikitext text/x-wiki 29o53tei3rnjy3trhq91tstxqraf704 2430175 2430174 2026-06-24T12:46:29Z Stashbot 7414 cwilliams@dns1005: START - running authdns-update 2430175 wikitext text/x-wiki el18wbuklu5pv7kg4n5ylsw8f7fyglt 2430176 2430175 2026-06-24T12:46:31Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new VIP for dumps-nfs - filippo@cumin1003" 2430176 wikitext text/x-wiki ikcgdfw21jqqh2eqyi1qa5we6wzlt5h 2430177 2430176 2026-06-24T12:46:33Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430177 wikitext text/x-wiki idnwflunemsw8ta4t52x2mxg63ogjz5 2430178 2430177 2026-06-24T12:46:54Z Stashbot 7414 fabfur: depooling cp3066 and cp3074 to reimage (T419825) 2430178 wikitext text/x-wiki c7rfv1898v5e71z8thd8ja1eyqorlqq 2430179 2430178 2026-06-24T12:46:59Z Stashbot 7414 claime: Setting globaluser gu_home_db to NULL for apiportalwiki globalusers - T418494 2430179 wikitext text/x-wiki e3jgpmp4v4khpqcv56hamxpkmjt9fnm 2430180 2430179 2026-06-24T12:47:15Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.dns.netbox 2430180 wikitext text/x-wiki p2tw698xxaf3e9jcc2b7o7obopjwqmx 2430181 2430180 2026-06-24T12:47:20Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* 2430181 wikitext text/x-wiki g1zmbyz3mi1vlixm8psffudtk96uum7 2430182 2430181 2026-06-24T12:47:23Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* 2430182 wikitext text/x-wiki fl09i0inmjht6e8bt0pngs1edppasf1 2430183 2430182 2026-06-24T12:47:28Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.* 2430183 wikitext text/x-wiki thwks8x408vph8ufunv96jykc0s9xt1 2430184 2430183 2026-06-24T12:47:40Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430184 wikitext text/x-wiki fthg5tgpis0q8a3xt98a5vofr9sf5gp 2430185 2430184 2026-06-24T12:48:26Z Stashbot 7414 claime: Deleting apiportalwiki references in GlobalUsage - T418494 2430185 wikitext text/x-wiki i3ixkha6ewojqrjvdr7h6fp3w52lsdz 2430186 2430185 2026-06-24T12:48:28Z Stashbot 7414 cwilliams@dns1005: END - running authdns-update 2430186 wikitext text/x-wiki 2mhmy3qgqrhczlt7mc7ybjk827opfku 2430187 2430186 2026-06-24T12:49:01Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.reimage for host cp3066.esams.wmnet with OS trixie 2430187 wikitext text/x-wiki 2zohlqe9anks64fhq1dyhg6y8cmcfbk 2430188 2430187 2026-06-24T12:49:03Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.reimage for host cp3074.esams.wmnet with OS trixie 2430188 wikitext text/x-wiki tkg2c0zrbt235st3o74aqgxkiuvyvle 2430190 2430188 2026-06-24T12:51:34Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new VIP for dumps-nfs - filippo@cumin1003" 2430190 wikitext text/x-wiki gkfq5iprab6gxuzq3ugtx1km6egqcz8 2430191 2430190 2026-06-24T12:51:38Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new VIP for dumps-nfs - filippo@cumin1003" 2430191 wikitext text/x-wiki ic8klmwwwofwhyi15qf1w6rcisg4dcx 2430192 2430191 2026-06-24T12:51:41Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430192 wikitext text/x-wiki 2qvcsx8b05ias50g9v5k9fnlinfc8sv 2430193 2430192 2026-06-24T12:53:27Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430193 wikitext text/x-wiki q1q8ac2syfvaxrcvfkuoz82foh7dtu5 2430194 2430193 2026-06-24T12:54:30Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 9 hosts with reason: maintenance 2430194 wikitext text/x-wiki too3n6qsoz6irk2vh8761sszpzch57p 2430195 2430194 2026-06-24T12:55:37Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430195 wikitext text/x-wiki m5os1c019ekin7l32a1qsmd5ma020hk 2430196 2430195 2026-06-24T12:59:02Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430196 wikitext text/x-wiki llck3lafe1ebz4kxbypsw1txz61gmqi 2430197 2430196 2026-06-24T12:59:18Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430197 wikitext text/x-wiki noeer7ubxql97lk6dc3qag03pm5ams3 2430198 2430197 2026-06-24T13:00:49Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1238.eqiad.wmnet with OS trixie 2430198 wikitext text/x-wiki occ0l04rm1n6j4kealtd3n08iivbb5m 2430199 2430198 2026-06-24T13:01:08Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430199 wikitext text/x-wiki axfatnv0xuvjy5bfc4kiwdxil5y0585 2430202 2430199 2026-06-24T13:06:25Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1305378|CheckUserGetUsersPager: Fix TypeError for numeric usernames (T429971)]] 2430202 wikitext text/x-wiki qb632h4n9hpxfuh0my9cgy0xnig50i4 2430203 2430202 2026-06-24T13:06:58Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2206.codfw.wmnet with OS trixie 2430203 wikitext text/x-wiki swi0z4hjwn2zuv9flgjto7s8owiou4m 2430204 2430203 2026-06-24T13:08:29Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1305378|CheckUserGetUsersPager: Fix TypeError for numeric usernames (T429971)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430204 wikitext text/x-wiki jq2j6akomgv1wrl0a0liy7hquu7ac3a 2430205 2430204 2026-06-24T13:11:24Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2430205 wikitext text/x-wiki 0v70vds77v9s4dauh1wreujx8q0l5bi 2430206 2430205 2026-06-24T13:14:00Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3074.esams.wmnet with reason: host reimage 2430206 wikitext text/x-wiki jsdv6ks0g2sqee1xcq9z8u80yrmkwpu 2430207 2430206 2026-06-24T13:14:56Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3066.esams.wmnet with reason: host reimage 2430207 wikitext text/x-wiki 848tr8kc9ag8hd0z6kaiqifm1ph75t5 2430208 2430207 2026-06-24T13:15:42Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305378|CheckUserGetUsersPager: Fix TypeError for numeric usernames (T429971)]] (duration: 09m 17s) 2430208 wikitext text/x-wiki pclphv8exx56uz3axsqc5dntzldqly2 2430209 2430208 2026-06-24T13:15:54Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1238.eqiad.wmnet with reason: host reimage 2430209 wikitext text/x-wiki s0c2y3kzvuz0fyfi5eyc3eaxyt0ux6o 2430210 2430209 2026-06-24T13:19:58Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3074.esams.wmnet with reason: host reimage 2430210 wikitext text/x-wiki idqp3gnmggw6x4bx7cs15vy3rcq9f0x 2430212 2430210 2026-06-24T13:21:49Z Stashbot 7414 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1302201|Disable parser survey for all wikis]] 2430212 wikitext text/x-wiki 9hqa7f98pzcpupxast80zzc4q0jiafe 2430213 2430212 2026-06-24T13:22:16Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2156: rack depool 2430213 wikitext text/x-wiki t1njkmed1wnnj9hymk38lfb9ispxugg 2430214 2430213 2026-06-24T13:23:53Z Stashbot 7414 jgiannelos@deploy1003: mbsantos, jgiannelos: Backport for [[gerrit:1302201|Disable parser survey for all wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430214 wikitext text/x-wiki lqf1ns7z2lk0t4ub4xhufx6zxw1layr 2430215 2430214 2026-06-24T13:24:43Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3066.esams.wmnet with reason: host reimage 2430215 wikitext text/x-wiki 086jgtz4i4ks1qwqc95e0kl1jchx16j 2430216 2430215 2026-06-24T13:26:16Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2206.codfw.wmnet with reason: host reimage 2430216 wikitext text/x-wiki fh89smoln4nbdpsnjpqtyh46nsf0049 2430217 2430216 2026-06-24T13:28:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2155: repool after rack maintenance 2430217 wikitext text/x-wiki 7kjh9r7kdlfhzevap782fum6sjtqtin 2430218 2430217 2026-06-24T13:28:47Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1238.eqiad.wmnet with reason: host reimage 2430218 wikitext text/x-wiki 2sx5sdnezutz43v86n8v4c9y98u0m18 2430219 2430218 2026-06-24T13:29:55Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430219 wikitext text/x-wiki qe8p7t5e84mfa8g3guy341ro491d79g 2430220 2430219 2026-06-24T13:32:11Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430220 wikitext text/x-wiki mw97duurrokrb5kbvu236yhv0a5ot07 2430221 2430220 2026-06-24T13:32:51Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2206.codfw.wmnet with reason: host reimage 2430221 wikitext text/x-wiki 4j3z3r7n6gbkmoubzqdi451znbf0176 2430222 2430221 2026-06-24T13:33:37Z Stashbot 7414 jgiannelos@deploy1003: mbsantos, jgiannelos: Continuing with deployment 2430222 wikitext text/x-wiki lvtx7j9q1gctziwml5co209fgeud8ro 2430223 2430222 2026-06-24T13:37:51Z Stashbot 7414 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302201|Disable parser survey for all wikis]] (duration: 16m 01s) 2430223 wikitext text/x-wiki ca5onq6i31g6klastopjl3k0pya8w3k 2430225 2430223 2026-06-24T13:45:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1238.eqiad.wmnet with OS trixie 2430225 wikitext text/x-wiki 2zv335ljarsnwf0n4hy9lf4ooxd6ks3 2430226 2430225 2026-06-24T13:46:53Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3074.esams.wmnet with OS trixie 2430226 wikitext text/x-wiki 5t79jh7cxgvl3nrsib15rekjmzcbsau 2430227 2430226 2026-06-24T13:50:23Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1304001|Fix autonym for Khasi (kha) in wmgExtraLanguageNames (T427917)]], [[gerrit:1304695|csbwiki: update logo, wordmark and tagline (T429126)]], [[gerrit:1305407|Handle the ConfirmEditGetGlobalInstanceFromContext hook (T429848)]], [[gerrit:1304919|hCaptcha: Enable for Special:Contact (T429848)]], [[gerrit:1305408|Create ConfirmEditGetGlobalInstanc 2430227 wikitext text/x-wiki 9pjo85bqsmb1xfeq7h1xqk6k7sjsuyr 2430228 2430227 2026-06-24T13:50:27Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2206.codfw.wmnet with OS trixie 2430228 wikitext text/x-wiki as92ilb6lvnu0zdlpapz6wqcycyq2jt 2430229 2430228 2026-06-24T13:51:25Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3066.esams.wmnet with OS trixie 2430229 wikitext text/x-wiki dc2gmf8edj0w41qkrim3zxqqz5slnfi 2430230 2430229 2026-06-24T13:52:30Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, valn-ilyo, anzx: Backport for [[gerrit:1304001|Fix autonym for Khasi (kha) in wmgExtraLanguageNames (T427917)]], [[gerrit:1304695|csbwiki: update logo, wordmark and tagline (T429126)]], [[gerrit:1305407|Handle the ConfirmEditGetGlobalInstanceFromContext hook (T429848)]], [[gerrit:1304919|hCaptcha: Enable for Special:Contact (T429848)]], [[gerrit:1305408|Create ConfirmEditGetGlobalIns 2430230 wikitext text/x-wiki dzagwwx2ufjhnpgqfywkj8zeg0leps6 2430231 2430230 2026-06-24T13:55:11Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, valn-ilyo, anzx: Continuing with deployment 2430231 wikitext text/x-wiki 24qpol265pehyyiwuoqyr2qwq0metqa 2430232 2430231 2026-06-24T13:59:36Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1304001|Fix autonym for Khasi (kha) in wmgExtraLanguageNames (T427917)]], [[gerrit:1304695|csbwiki: update logo, wordmark and tagline (T429126)]], [[gerrit:1305407|Handle the ConfirmEditGetGlobalInstanceFromContext hook (T429848)]], [[gerrit:1304919|hCaptcha: Enable for Special:Contact (T429848)]], [[gerrit:1305408|Create ConfirmEditGetGlobalInstan 2430232 wikitext text/x-wiki kxgmlyxrsf0aneb09xmo2c8pjwpwf4x 2430234 2430232 2026-06-24T14:01:37Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1238: Migration of db1238.eqiad.wmnet completed 2430234 wikitext text/x-wiki fqq4m1up32gmjxrg848rrbma1oyxe8p 2430235 2430234 2026-06-24T14:05:02Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp3066.esams.wmnet 2430235 wikitext text/x-wiki gw61e9k17kgabkc3mnp3h6pktdewjjk 2430237 2430235 2026-06-24T14:05:05Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp3066.esams.wmnet 2430237 wikitext text/x-wiki jaxe0w5gaj672vqh98tu0kcveres3i3 2430238 2430237 2026-06-24T14:05:09Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp3074.esams.wmnet 2430238 wikitext text/x-wiki pqsug45okv1iznedky5e4063ui8zdpn 2430239 2430238 2026-06-24T14:05:12Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp3074.esams.wmnet 2430239 wikitext text/x-wiki 3xcq9hedcbso4f24xpd3pqtclcfi2sx 2430240 2430239 2026-06-24T14:05:29Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2206: Migration of db2206.codfw.wmnet completed 2430240 wikitext text/x-wiki hvdc5md3wgd0ylseop5s6r6d7w11o5k 2430241 2430240 2026-06-24T14:06:36Z Stashbot 7414 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2430241 wikitext text/x-wiki azrq3n0gg2zjppvmrmytti008yqf8e5 2430242 2430241 2026-06-24T14:07:32Z Stashbot 7414 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2430242 wikitext text/x-wiki 8hmz3izh0gafyro1ygey7j72dlmlzof 2430243 2430242 2026-06-24T14:09:00Z Stashbot 7414 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2430243 wikitext text/x-wiki gyb8u52z7s73ip8djnwtmnxq45wn4yf 2430244 2430243 2026-06-24T14:09:16Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1305430|Handle the ConfirmEditGetGlobalInstanceFromContext hook (T429848)]], [[gerrit:1305431|Create ConfirmEditGetGlobalInstanceFromContext hook (T429848)]] 2430244 wikitext text/x-wiki mxppqo66oaahciuz9femmpwev8wxvyk 2430245 2430244 2026-06-24T14:09:35Z Stashbot 7414 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2430245 wikitext text/x-wiki 90wcfeqnr4k1yrffnlzg7ertbie3bcm 2430246 2430245 2026-06-24T14:09:43Z Stashbot 7414 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2430246 wikitext text/x-wiki p03q6czoe7wa28fkfb5e3zhfl4polxw 2430247 2430246 2026-06-24T14:10:11Z Stashbot 7414 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2430247 wikitext text/x-wiki boymk1hiqb1t54xci8bqiroa4ntf00y 2430248 2430247 2026-06-24T14:11:21Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1305430|Handle the ConfirmEditGetGlobalInstanceFromContext hook (T429848)]], [[gerrit:1305431|Create ConfirmEditGetGlobalInstanceFromContext hook (T429848)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430248 wikitext text/x-wiki j0d2szgfonywch9xppaqh2kesc622nk 2430249 2430248 2026-06-24T14:12:48Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2430249 wikitext text/x-wiki ejgdhya9od7z9lvuhnc7b7itnze7iym 2430250 2430249 2026-06-24T14:14:24Z Stashbot 7414 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2430250 wikitext text/x-wiki k52d349eqbfi36lfzh06gb4iiktnaa7 2430251 2430250 2026-06-24T14:15:05Z Stashbot 7414 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2430251 wikitext text/x-wiki n6ykvyrk77or64hqhhdpopz3o3g1bff 2430252 2430251 2026-06-24T14:15:33Z Stashbot 7414 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2430252 wikitext text/x-wiki iriiaecerakqywfs7e70eg6nmmtkbwa 2430253 2430252 2026-06-24T14:16:11Z Stashbot 7414 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2430253 wikitext text/x-wiki 7wliv709peph4pcs4ebv3yoz8j8sx3j 2430254 2430253 2026-06-24T14:16:19Z Stashbot 7414 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2430254 wikitext text/x-wiki lc4swylstpku82tsvg8qmc1903a6ar7 2430255 2430254 2026-06-24T14:17:00Z Stashbot 7414 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2430255 wikitext text/x-wiki ir0qdsie1jpyh7c5bgt5rp6p3d5k89v 2430256 2430255 2026-06-24T14:17:05Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305430|Handle the ConfirmEditGetGlobalInstanceFromContext hook (T429848)]], [[gerrit:1305431|Create ConfirmEditGetGlobalInstanceFromContext hook (T429848)]] (duration: 07m 48s) 2430256 wikitext text/x-wiki nsafuqjp6vjhqxfpiolnca7dhjga1t0 2430257 2430256 2026-06-24T14:17:31Z Stashbot 7414 Dreamy_Jazz: Afternoon UTC backport window done 2430257 wikitext text/x-wiki 2tmtg6wanqrwxoaf7wkhy4471efci5q 2430258 2430257 2026-06-24T14:24:56Z Stashbot 7414 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2430258 wikitext text/x-wiki ef32qnnvmpxefsewskfo7njlwj5h1on 2430259 2430258 2026-06-24T14:25:00Z Stashbot 7414 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2430259 wikitext text/x-wiki iofw0ywxb9o8zhi7mc7ifswy8lfynnh 2430260 2430259 2026-06-24T14:25:24Z Stashbot 7414 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2430260 wikitext text/x-wiki qj1hz4t95vialgzqir6hpp8q2uwgmm1 2430261 2430260 2026-06-24T14:25:28Z Stashbot 7414 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2430261 wikitext text/x-wiki svvmcyl622u19qtd81r9p5c33jm2cvd 2430262 2430261 2026-06-24T14:25:38Z Stashbot 7414 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2430262 wikitext text/x-wiki thd12fw2gqs8uva4iqnwvmcixv7g3qx 2430263 2430262 2026-06-24T14:25:55Z Stashbot 7414 fabfur: repooling cp3066 and cp3074 after reimage (T419825) 2430263 wikitext text/x-wiki 3zdgncevctx79m5aaac1e4tn9pgojzf 2430264 2430263 2026-06-24T14:26:05Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* 2430264 wikitext text/x-wiki 8qp9e6fcr0ro86gh4p7wi80vnsr2vjp 2430265 2430264 2026-06-24T14:26:08Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* 2430265 wikitext text/x-wiki 8eh8u9fguhh3lq2s447jr7mbijh4q08 2430266 2430265 2026-06-24T14:26:10Z Stashbot 7414 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2430266 wikitext text/x-wiki its61lpg7tyrp77s6k2oixrsab6t50i 2430267 2430266 2026-06-24T14:26:13Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.* 2430267 wikitext text/x-wiki ra0a81vg9oyeoswm56ahuw5es8hy58z 2430268 2430267 2026-06-24T14:26:15Z Stashbot 7414 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2430268 wikitext text/x-wiki jwe9fv2jbqxbohunqb2f4djzx5ph61l 2430269 2430268 2026-06-24T14:26:22Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp3066.esams.wmnet 2430269 wikitext text/x-wiki 1qcylj95wp4i6lj3fei9rfuzwx7vt77 2430270 2430269 2026-06-24T14:26:25Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp3066.esams.wmnet 2430270 wikitext text/x-wiki 2wmatmtpar0hzqemqkrh2r0mpe3irr3 2430271 2430270 2026-06-24T14:26:31Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp3074.esams.wmnet 2430271 wikitext text/x-wiki a9y0j38dxrsd2v7iw2qesya4uy68m4n 2430272 2430271 2026-06-24T14:26:34Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp3074.esams.wmnet 2430272 wikitext text/x-wiki 10l3nyw0b281wgmq3ff4q5ws5i1ep7d 2430273 2430272 2026-06-24T14:26:50Z Stashbot 7414 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2430273 wikitext text/x-wiki 31pairow1j3xs65o5xmyfp7ubjpx6kr 2430274 2430273 2026-06-24T14:34:34Z Stashbot 7414 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1305436|Restore the per-reader opt-out for the mobile image carousel (T419786)]], [[gerrit:1305437|Restore the per-reader opt-out for the mobile image carousel (T419786)]] 2430274 wikitext text/x-wiki p320fylzsh4uslrklbsoppr9jsd8vd2 2430275 2430274 2026-06-24T14:46:11Z Stashbot 7414 moritzm: installing postgresql security updates 2430275 wikitext text/x-wiki lsa77b3bvp7dsbwc3axd6lxn09a7d6p 2430276 2430275 2026-06-24T14:47:08Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1238: Migration of db1238.eqiad.wmnet completed 2430276 wikitext text/x-wiki sfah7cstn35st6v1cc41gl8vxysdyut 2430277 2430276 2026-06-24T14:47:10Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430277 wikitext text/x-wiki pl8l15qvh96gbfxvrfx8h7h0gdp12a2 2430278 2430277 2026-06-24T14:51:00Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2206: Migration of db2206.codfw.wmnet completed 2430278 wikitext text/x-wiki grvqm9htq4yzjhjaaq4vdj0g4jsr5n8 2430279 2430278 2026-06-24T14:51:03Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430279 wikitext text/x-wiki czn7g5qx8bo8rujdk3clwn4hspiwo8m 2430280 2430279 2026-06-24T14:53:42Z Stashbot 7414 mfossati@deploy1003: egardner, mfossati: Backport for [[gerrit:1305436|Restore the per-reader opt-out for the mobile image carousel (T419786)]], [[gerrit:1305437|Restore the per-reader opt-out for the mobile image carousel (T419786)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430280 wikitext text/x-wiki bl438x866uddak830w4dqr5b2yehlog 2430282 2430280 2026-06-24T14:57:20Z Stashbot 7414 mfossati@deploy1003: egardner, mfossati: Continuing with deployment 2430282 wikitext text/x-wiki 4v51w5kngol79h33wtmwt8e29u1cido 2430283 2430282 2026-06-24T15:09:20Z Stashbot 7414 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2430283 wikitext text/x-wiki mfqf7mxs6h6cgbbfn3abcamc1ilsdq6 2430284 2430283 2026-06-24T15:09:58Z Stashbot 7414 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2430284 wikitext text/x-wiki 0skauz6sdbxajlp7iqauzpp2e6blo3o 2430285 2430284 2026-06-24T15:10:04Z Stashbot 7414 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2430285 wikitext text/x-wiki ll7v935n57sjo9wbenmtbi9ehync6nt 2430286 2430285 2026-06-24T15:10:26Z Stashbot 7414 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305436|Restore the per-reader opt-out for the mobile image carousel (T419786)]], [[gerrit:1305437|Restore the per-reader opt-out for the mobile image carousel (T419786)]] (duration: 35m 52s) 2430286 wikitext text/x-wiki j2jd5i755pyc202avfqmyujyr3z9q6p 2430287 2430286 2026-06-24T15:10:29Z Stashbot 7414 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2430287 wikitext text/x-wiki d0shmb67evypnamc5oz1p9b3jvsfesp 2430288 2430287 2026-06-24T15:10:33Z Stashbot 7414 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2430288 wikitext text/x-wiki qah6v0gg6m77451jps7anvgs5lr6570 2430289 2430288 2026-06-24T15:10:58Z Stashbot 7414 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2430289 wikitext text/x-wiki 5cowh5aiv5wonirouvdkoikmnl951k4 2430290 2430289 2026-06-24T15:11:05Z Stashbot 7414 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2430290 wikitext text/x-wiki 4xkwpl6wdm58tjot309vszwzuva41xo 2430291 2430290 2026-06-24T15:11:23Z Stashbot 7414 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2430291 wikitext text/x-wiki 6yuwsifrf0vkj6eonh4sjc6igtimvcj 2430292 2430291 2026-06-24T15:11:29Z Stashbot 7414 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2430292 wikitext text/x-wiki izhqa7o2rw2vfxoeedy563e7rl7a0h0 2430293 2430292 2026-06-24T15:11:58Z Stashbot 7414 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2430293 wikitext text/x-wiki rbf93f2k5hyz4k6hkmmhfpzt7expmk3 2430294 2430293 2026-06-24T15:12:04Z Stashbot 7414 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2430294 wikitext text/x-wiki 2lqfmul0eyxzkz5xwxep4tr8ei6mbxm 2430296 2430294 2026-06-24T15:14:08Z Stashbot 7414 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2430296 wikitext text/x-wiki slkwmyiq9uvb8dk1twf328u4jj4m0n4 2430297 2430296 2026-06-24T15:15:16Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2430297 wikitext text/x-wiki j8v1hxv4uwrskb4bwhnfv5cw59yr752 2430298 2430297 2026-06-24T15:16:11Z Stashbot 7414 cgoubert@deploy1003: Started scap sync-world: Backport for [[gerrit:1305442|Update interwiki map (T429372 T418494)]] 2430298 wikitext text/x-wiki kakrfhzin0nxx2u4y1hx2pre39hs8ai 2430299 2430298 2026-06-24T15:20:40Z Stashbot 7414 cgoubert@deploy1003: cgoubert: Backport for [[gerrit:1305442|Update interwiki map (T429372 T418494)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430299 wikitext text/x-wiki b8txfmdd2fjy1b9bj2h9ckpy95psz3k 2430300 2430299 2026-06-24T15:21:13Z Stashbot 7414 cgoubert@deploy1003: cgoubert: Continuing with deployment 2430300 wikitext text/x-wiki 0e2oqd8m7llj50ldlza33l49mdp2i4i 2430301 2430300 2026-06-24T15:25:50Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2430301 wikitext text/x-wiki 5hrf1873fwl7gkqpu7m51v390rgkrqt 2430302 2430301 2026-06-24T15:25:56Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2430302 wikitext text/x-wiki fm90tctzwumngwjulp2cdjndibbi370 2430303 2430302 2026-06-24T15:29:51Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2430303 wikitext text/x-wiki gimkv1162lbo1maqqgf2qe8o44i9c8h 2430304 2430303 2026-06-24T15:29:57Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2430304 wikitext text/x-wiki dsp30km4lqmzjsunjglwlsyved9uu3t 2430305 2430304 2026-06-24T15:30:19Z Stashbot 7414 cgoubert@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305442|Update interwiki map (T429372 T418494)]] (duration: 14m 09s) 2430305 wikitext text/x-wiki 48mjw5dndwm9wdx6si2lrjet9qacizw 2430306 2430305 2026-06-24T15:30:36Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2430306 wikitext text/x-wiki 060eebw25d8b9m9ew9ykejujfe4w3xe 2430307 2430306 2026-06-24T15:30:42Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2430307 wikitext text/x-wiki 9ol87gjc2zdv75esrq95w5s94tqezsy 2430308 2430307 2026-06-24T15:31:51Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2430308 wikitext text/x-wiki myzh4ycne9p4pw0sb9e0opo6fbvrqbk 2430309 2430308 2026-06-24T15:31:57Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2430309 wikitext text/x-wiki q1j7okearqk3c10ffed0xotaiwbjnnx 2430310 2430309 2026-06-24T15:32:31Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2430310 wikitext text/x-wiki 0f3mrdxyhvrmyp0lgj6sj886mj9hinr 2430311 2430310 2026-06-24T15:32:37Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2430311 wikitext text/x-wiki gzy232dqgq5j2uo1d1qhea2b2almq2v 2430312 2430311 2026-06-24T15:33:48Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2430312 wikitext text/x-wiki em1o3uy3eed0v8dmmn0rfisn5um4ce4 2430313 2430312 2026-06-24T15:37:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430313 wikitext text/x-wiki 363ab6ifej6h1e2v3tlrpeakb1b4mw6 2430314 2430313 2026-06-24T15:38:30Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430314 wikitext text/x-wiki 30vc65d3mrydlvygx6jzr4uodqn4yao 2430315 2430314 2026-06-24T15:40:23Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@eqiad T429893 2430315 wikitext text/x-wiki 18q7g544esdgk5j4bq1p7t9dgi7f906 2430316 2430315 2026-06-24T15:40:37Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1241: Upgrading db1241.eqiad.wmnet 2430316 wikitext text/x-wiki inf7ekcppj0xee1y5mwecgzcq2i1p39 2430317 2430316 2026-06-24T15:40:52Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430317 wikitext text/x-wiki n1ag1umiuirmaazhk0eyb3d4v377n1d 2430318 2430317 2026-06-24T15:41:02Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@codfw T429893 2430318 wikitext text/x-wiki o9fsvxq8somz8eoqjvqy93seww591j0 2430319 2430318 2026-06-24T15:41:17Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2210: Upgrading db2210.codfw.wmnet 2430319 wikitext text/x-wiki 7rba4apqts5stmvgkhw0rkew9k51wt4 2430320 2430319 2026-06-24T15:41:25Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1241: Upgrading db1241.eqiad.wmnet 2430320 wikitext text/x-wiki owhmgeqy7906r4efau0lxhh5i7l3uvs 2430321 2430320 2026-06-24T15:41:35Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply 2430321 wikitext text/x-wiki lx5nhijeqrop55ggfk62t5n8j32crli 2430322 2430321 2026-06-24T15:41:45Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2210: Upgrading db2210.codfw.wmnet 2430322 wikitext text/x-wiki 1az8hu5jejxjjjns4kydb7sthdh7xek 2430323 2430322 2026-06-24T15:43:30Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430323 wikitext text/x-wiki 8n17uzbsmv43tfctbnlfllk56cmqfwd 2430324 2430323 2026-06-24T15:43:48Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1241.eqiad.wmnet with OS trixie 2430324 wikitext text/x-wiki hr5l3fxkh7aelcbiohxwzl3zi0g24a0 2430325 2430324 2026-06-24T15:46:53Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply 2430325 wikitext text/x-wiki p381u60jrvfnwu1vbojctuurtbzf4zt 2430326 2430325 2026-06-24T15:47:14Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply 2430326 wikitext text/x-wiki r9epmkdhbsfxsyz147gtdputni4zcgh 2430327 2430326 2026-06-24T15:49:25Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430327 wikitext text/x-wiki 0hpx18ti15n1vjmpfqdlphapcf9ontt 2430328 2430327 2026-06-24T15:52:17Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply 2430328 wikitext text/x-wiki fnku474wgm3rghp6q6s81x67cyr7xll 2430329 2430328 2026-06-24T15:52:24Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply 2430329 wikitext text/x-wiki klvbvwbc2gez52ty1uy1b2ocuzgb4kx 2430330 2430329 2026-06-24T15:52:51Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2430330 wikitext text/x-wiki bigo19wn43l83j8lzuo2rd157vp1wu2 2430331 2430330 2026-06-24T15:52:57Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2430331 wikitext text/x-wiki 684kn7rp4ccvunekqgymc7pixcdzczj 2430332 2430331 2026-06-24T15:52:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2210.codfw.wmnet with OS trixie 2430332 wikitext text/x-wiki l4o2s8pcnjr44rk8c7ql7svr5nhqnlp 2430333 2430332 2026-06-24T15:53:25Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2430333 wikitext text/x-wiki 62wx2z4fy6szlnqevxa61xb498zjpee 2430334 2430333 2026-06-24T15:53:31Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2430334 wikitext text/x-wiki czdhul42z5t00uujjkbp3c3pyqvw8fx 2430335 2430334 2026-06-24T15:54:32Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2430335 wikitext text/x-wiki 0dikngl7ex4fit2wc1rgtnkyceaslm8 2430336 2430335 2026-06-24T15:56:32Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2430336 wikitext text/x-wiki jt9d9vq045to6rixlehx018ekucdb4t 2430337 2430336 2026-06-24T15:58:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1241.eqiad.wmnet with reason: host reimage 2430337 wikitext text/x-wiki qquhqeralw5fjrjh3abo7d5zmk09v61 2430338 2430337 2026-06-24T16:04:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1241.eqiad.wmnet with reason: host reimage 2430338 wikitext text/x-wiki saiyy20fmlumo1li7rvnkrdukhre080 2430341 2430338 2026-06-24T16:12:10Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2210.codfw.wmnet with reason: host reimage 2430341 wikitext text/x-wiki 4x3agmj2hxz8gbxw8l0wdoks0wi5mc2 2430342 2430341 2026-06-24T16:19:47Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2210.codfw.wmnet with reason: host reimage 2430342 wikitext text/x-wiki 5i1p68kgomgzd5b4z25squ99zfuhxqa 2430343 2430342 2026-06-24T16:21:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1241.eqiad.wmnet with OS trixie 2430343 wikitext text/x-wiki jxdqrwetyqziqvllcd0sv8obmaa8ogo 2430345 2430343 2026-06-24T16:33:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1241: Migration of db1241.eqiad.wmnet completed 2430345 wikitext text/x-wiki gxvtj5zy58g7bckqvsyny7gvu323n3j 2430347 2430345 2026-06-24T16:37:11Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430347 wikitext text/x-wiki hjt2jvt4oyzeuwa0mio9szfd911jm94 2430348 2430347 2026-06-24T16:37:15Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2210.codfw.wmnet with OS trixie 2430348 wikitext text/x-wiki tljhno0uxishf151lvepl6lp9tgkxvf 2430349 2430348 2026-06-24T16:37:22Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430349 wikitext text/x-wiki 6rc5mj5vci64lujh1073dalew703n9j 2430352 2430349 2026-06-24T16:41:37Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430352 wikitext text/x-wiki c1xnbggaqr6qr23lopimzrfk4fp42kr 2430353 2430352 2026-06-24T16:41:46Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430353 wikitext text/x-wiki 9wt69ar4vmd4b8vg6z0nw3b81ajov16 2430355 2430353 2026-06-24T16:51:03Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2210: Migration of db2210.codfw.wmnet completed 2430355 wikitext text/x-wiki bq4fhh7rp31cekozmhh6c46cpp34cah 2430359 2430355 2026-06-24T17:05:54Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430359 wikitext text/x-wiki 06gj2eqfa4jrtinkwvo0hpe3pykpwyq 2430360 2430359 2026-06-24T17:06:11Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430360 wikitext text/x-wiki t21zfwz27rsppnaat4b9vvielexo21u 2430363 2430360 2026-06-24T17:19:05Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1241: Migration of db1241.eqiad.wmnet completed 2430363 wikitext text/x-wiki 8exfnvmgbiu54rbxumjvrt2qbb3m101 2430364 2430363 2026-06-24T17:19:07Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430364 wikitext text/x-wiki m959duiuanydfyep6ccl7djeixvfn49 2430366 2430364 2026-06-24T17:28:39Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T430072 2430366 wikitext text/x-wiki hr29d7btxvecv7x4y3fq5ij3ab39jby 2430369 2430366 2026-06-24T17:34:15Z Stashbot 7414 kamila@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2430369 wikitext text/x-wiki thcpypfst0tehx6ytyqnb7gzz9b2sac 2430370 2430369 2026-06-24T17:34:52Z Stashbot 7414 kamila@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2430370 wikitext text/x-wiki niyjaulwuc69c89h8058mj8iusdx5u2 2430371 2430370 2026-06-24T17:35:05Z Stashbot 7414 kamila@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2430371 wikitext text/x-wiki 2ovrxi21x5crusyg5korihjt80ll7zb 2430372 2430371 2026-06-24T17:35:40Z Stashbot 7414 kamila@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2430372 wikitext text/x-wiki iil8hw2xp86c67ep6uwcb6cmiips8bz 2430373 2430372 2026-06-24T17:36:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2210: Migration of db2210.codfw.wmnet completed 2430373 wikitext text/x-wiki 05rtyqyxmv78kg5jefh5s2mlwjez1kn 2430374 2430373 2026-06-24T17:36:36Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430374 wikitext text/x-wiki 9abgtmgfi4ug2ckn22daz5q5hssknxg 2430375 2430374 2026-06-24T17:36:46Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2430375 wikitext text/x-wiki 6kloc3zhpoosa2ozbiw2mkjl602l7le 2430377 2430375 2026-06-24T17:38:16Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T430072 2430377 wikitext text/x-wiki bgcnoeibq1p93zwipa5tinb489jhn0j 2430378 2430377 2026-06-24T17:38:23Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2430378 wikitext text/x-wiki m6knwtzkj8uykocjg1ahjrvmym5o2cv 2430380 2430378 2026-06-24T17:42:21Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T430072 2430380 wikitext text/x-wiki 30qdhpvagc4ihhuqfdt4wx0akgfpakg 2430381 2430380 2026-06-24T17:45:18Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2430381 wikitext text/x-wiki 0n64c062utt9lr3cv8mvt6ebgp00eal 2430382 2430381 2026-06-24T17:45:48Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2430382 wikitext text/x-wiki jcux28clvvinw4zafdg17r95ol9so00 2430383 2430382 2026-06-24T17:45:51Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2430383 wikitext text/x-wiki gzk21ygwq4o3rrjg0e5isk04pzabp4r 2430384 2430383 2026-06-24T17:46:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet 2430384 wikitext text/x-wiki 3yh0ry14cjni6bezykg0yogj8swju5l 2430385 2430384 2026-06-24T17:46:13Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1016.eqiad.wmnet 2430385 wikitext text/x-wiki h42ba1qmw8dbyheit3abmc2asb2phmp 2430386 2430385 2026-06-24T17:46:17Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2430386 wikitext text/x-wiki e8yqvbg1tg3ttrkbgyq9ta7wuozrwe4 2430387 2430386 2026-06-24T17:51:50Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T430072 2430387 wikitext text/x-wiki kuet8amn5tsl8c1f7v4wskis3sa8gsr 2430389 2430387 2026-06-24T17:52:53Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1015.eqiad.wmnet with OS bookworm 2430389 wikitext text/x-wiki sh4ltt6ozj2at05d7cpfz76tz14s2bk 2430390 2430389 2026-06-24T17:52:55Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1016.eqiad.wmnet with OS bookworm 2430390 wikitext text/x-wiki j8qyrypq5wx0tagi125l78ayp9oz7tk 2430391 2430390 2026-06-24T17:52:58Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1017.eqiad.wmnet with OS bookworm 2430391 wikitext text/x-wiki 6q80pp04uzwq3iwimnbk2tel24nanzo 2430392 2430391 2026-06-24T17:54:12Z Stashbot 7414 cdanis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2430392 wikitext text/x-wiki sv6dzswf3qhz6ni35i6j4z9p48urljm 2430393 2430392 2026-06-24T17:54:35Z Stashbot 7414 cdanis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2430393 wikitext text/x-wiki 5slweipt2hx1zk3ffxewut54kd30bf3 2430394 2430393 2026-06-24T17:55:38Z Stashbot 7414 cdanis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2430394 wikitext text/x-wiki mcvsz2l4u4jxzor7wegfce23mp90eqs 2430395 2430394 2026-06-24T17:55:51Z Stashbot 7414 cdanis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2430395 wikitext text/x-wiki 51o5boethlb49tzumrvovcjarfu3lql 2430397 2430395 2026-06-24T18:01:54Z Stashbot 7414 brennen: 1.47.0-wmf.8 train status (T423917): no current blockers, logs no worse than expected, rolling to group1 2430397 wikitext text/x-wiki 31niwphs8tt5ru5c55u8qx5078vzzik 2430399 2430397 2026-06-24T18:05:15Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430399 wikitext text/x-wiki nhae1v2t8iz0sna8t594rarkfmg7r00 2430400 2430399 2026-06-24T18:05:19Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@eqiad T429893 2430400 wikitext text/x-wiki 3ruyecqu1z10nna41ug0set6zcbwlh2 2430401 2430400 2026-06-24T18:05:35Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1242: Upgrading db1242.eqiad.wmnet 2430401 wikitext text/x-wiki fuzg22astcilgoa9thcph5x76c5mp7z 2430402 2430401 2026-06-24T18:06:06Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1242: Upgrading db1242.eqiad.wmnet 2430402 wikitext text/x-wiki mkbvfl8ilioc0ewp1gem15u7y5d3s2x 2430404 2430402 2026-06-24T18:08:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1017.eqiad.wmnet with reason: host reimage 2430404 wikitext text/x-wiki lvabrh5ldekab7ld4c7knvwpj6uya7u 2430405 2430404 2026-06-24T18:09:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1015.eqiad.wmnet with reason: host reimage 2430405 wikitext text/x-wiki dd7leolk51joelre7wci7qrod66i9c5 2430406 2430405 2026-06-24T18:10:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1016.eqiad.wmnet with reason: host reimage 2430406 wikitext text/x-wiki 41h1z3edylncflefzc9424h3yb7jtr8 2430407 2430406 2026-06-24T18:12:46Z Stashbot 7414 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.8 refs T423917 2430407 wikitext text/x-wiki j7tx360067i3ey5d78x8ts3jfxh9sct 2430408 2430407 2026-06-24T18:14:51Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1017.eqiad.wmnet with reason: host reimage 2430408 wikitext text/x-wiki quiecz0hgpitoek376kk2ktqs4li0e6 2430409 2430408 2026-06-24T18:16:01Z Stashbot 7414 kamila@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2430409 wikitext text/x-wiki axmpg0antepxv3wfkpnortw457kn5ht 2430410 2430409 2026-06-24T18:17:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1015.eqiad.wmnet with reason: host reimage 2430410 wikitext text/x-wiki d84xtv82lv84pmfz571cozaw4nhdqwx 2430411 2430410 2026-06-24T18:19:25Z Stashbot 7414 kamila@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2430411 wikitext text/x-wiki geth8apjqcw8zj8g0xhv05wig8yhcet 2430412 2430411 2026-06-24T18:19:51Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2430412 wikitext text/x-wiki ragfkh60o2fbhi2uv31qouea8kc6330 2430413 2430412 2026-06-24T18:21:55Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430413 wikitext text/x-wiki o5iudbi6dfisyhk7r6rp8iiwglh0uql 2430414 2430413 2026-06-24T18:21:59Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@eqiad T429893 2430414 wikitext text/x-wiki bff8ypi05kl8v8a4qwnw1n4qqylxwvj 2430415 2430414 2026-06-24T18:22:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1242: Upgrading db1242.eqiad.wmnet 2430415 wikitext text/x-wiki mek2fxtmbh1rwphu4srefwwppebzvjs 2430416 2430415 2026-06-24T18:22:07Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1016.eqiad.wmnet with reason: host reimage 2430416 wikitext text/x-wiki q6yd6ri9u8u381nzcuakdxlizckbdnn 2430417 2430416 2026-06-24T18:22:14Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1242: Upgrading db1242.eqiad.wmnet 2430417 wikitext text/x-wiki jcuehl23k3ur0akkj3uny71xz6z59ts 2430418 2430417 2026-06-24T18:22:44Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1242.eqiad.wmnet with OS trixie 2430418 wikitext text/x-wiki lk6fdl0kyvctcvsaegg7rssan1kkscv 2430419 2430418 2026-06-24T18:23:10Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430419 wikitext text/x-wiki g02ygxvdhy8bjdbiiwd05gts22xfbyu 2430420 2430419 2026-06-24T18:23:14Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@codfw T429893 2430420 wikitext text/x-wiki mi7gnxr0zv6boft5plma999u3jb74zh 2430421 2430420 2026-06-24T18:23:31Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2219: Upgrading db2219.codfw.wmnet 2430421 wikitext text/x-wiki kl31pnqkl9uo4xwb74ye3ga4whslt9u 2430422 2430421 2026-06-24T18:23:53Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2219: Upgrading db2219.codfw.wmnet 2430422 wikitext text/x-wiki 9rm5tbwml6ceb3h7uk1hhumiyn2wxh9 2430423 2430422 2026-06-24T18:25:22Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2219.codfw.wmnet with OS trixie 2430423 wikitext text/x-wiki 99z58wlzqk972pq65u4v7g3y3klhw2d 2430424 2430423 2026-06-24T18:31:36Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet with OS bookworm 2430424 wikitext text/x-wiki fh4dr69degue5s7uh2basnm2gxoy6bl 2430425 2430424 2026-06-24T18:35:02Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1015.eqiad.wmnet with OS bookworm 2430425 wikitext text/x-wiki l4jzrmdcgplhzito6fv5ruebnjqnfyy 2430426 2430425 2026-06-24T18:39:36Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet with OS bookworm 2430426 wikitext text/x-wiki 2wtzan3svanpvalk4b1kcxpno4kw5j9 2430427 2430426 2026-06-24T18:40:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1242.eqiad.wmnet with reason: host reimage 2430427 wikitext text/x-wiki orkw4pssoh17nf43jchafwq55bu9906 2430428 2430427 2026-06-24T18:40:20Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2430428 wikitext text/x-wiki 69pt094l5m2lywaq8f6gaejjhogowke 2430429 2430428 2026-06-24T18:43:02Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2430429 wikitext text/x-wiki rrf9ht58kcxemajix8r61ct598k1dtx 2430430 2430429 2026-06-24T18:43:46Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2430430 wikitext text/x-wiki pso71xfhxbq6xbakrvw0321akpdtcae 2430431 2430430 2026-06-24T18:44:05Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2219.codfw.wmnet with reason: host reimage 2430431 wikitext text/x-wiki o27mub0gixdvpf5eyc21j5v16x53a61 2430432 2430431 2026-06-24T18:44:22Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1242.eqiad.wmnet with reason: host reimage 2430432 wikitext text/x-wiki n12957kl3r6oh2qoyaelbwq8km3sx6t 2430433 2430432 2026-06-24T18:45:38Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2430433 wikitext text/x-wiki m3vrb900jky0bprsyw576gvmqla2vum 2430434 2430433 2026-06-24T18:46:47Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2430434 wikitext text/x-wiki bnw89upchh5y30mx7lkmgm1ni8a1jt4 2430435 2430434 2026-06-24T18:48:10Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2219.codfw.wmnet with reason: host reimage 2430435 wikitext text/x-wiki m92cpqx4c33vgv5fypfp44lti8wrp1b 2430436 2430435 2026-06-24T18:49:29Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2430436 wikitext text/x-wiki 4ql0hpjot7qkhuhn9o57fmwj2e1sb0c 2430437 2430436 2026-06-24T18:56:18Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2430437 wikitext text/x-wiki 2u5jqug4lbsbrguaop3iljuyykpuxz9 2430438 2430437 2026-06-24T18:58:36Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2430438 wikitext text/x-wiki 1qfi1grdgz3v09cw36vv6fdrebz73ws 2430439 2430438 2026-06-24T19:01:32Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1242.eqiad.wmnet with OS trixie 2430439 wikitext text/x-wiki 0rwug6ggure5wp11xri59ex9demymvh 2430440 2430439 2026-06-24T19:02:01Z Stashbot 7414 swfrench-wmf: applied latent admin_ng diffs for mw-pretrain - T427668 2430440 wikitext text/x-wiki 0jzkp8gx07lmck5pduwj4f0se1nmsmz 2430441 2430440 2026-06-24T19:02:23Z Stashbot 7414 swfrench-wmf: applied latent admin_ng diffs for allow-urldownloaders GlobalNetworkPolicy - T430045 T427282 2430441 wikitext text/x-wiki 12oz771ckk8zj7njuv5i3aar644hgeg 2430442 2430441 2026-06-24T19:05:50Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2219.codfw.wmnet with OS trixie 2430442 wikitext text/x-wiki 9k3v7btndt6f2cubc3znmumihjrkruh 2430443 2430442 2026-06-24T19:13:18Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2430443 wikitext text/x-wiki j3qjnwgsddge9b6cjtfikbcszy4ds3i 2430444 2430443 2026-06-24T19:14:05Z Stashbot 7414 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2430444 wikitext text/x-wiki i48pypgjcc26jjtlfkmhhh8cx6rdbsf 2430445 2430444 2026-06-24T19:14:15Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply 2430445 wikitext text/x-wiki t0g9j8vvvley1jzc66tbyeplgkl11pz 2430446 2430445 2026-06-24T19:15:14Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1242: Migration of db1242.eqiad.wmnet completed 2430446 wikitext text/x-wiki ev17fiefhapt2qwyursyel04sdk4itr 2430447 2430446 2026-06-24T19:15:17Z Stashbot 7414 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply 2430447 wikitext text/x-wiki s2n0qbct2pmn6y8ks5y1r0jjw4v2x7m 2430448 2430447 2026-06-24T19:18:18Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2219: Migration of db2219.codfw.wmnet completed 2430448 wikitext text/x-wiki 6z0tzhrvel5wcqegdxys03u610zxi6j 2430449 2430448 2026-06-24T19:18:26Z Stashbot 7414 jgreen@dns1004: START - running authdns-update 2430449 wikitext text/x-wiki r8l0rybocyrw38uvas3n4xujdy172uf 2430450 2430449 2026-06-24T19:20:23Z Stashbot 7414 jgreen@dns1004: END - running authdns-update 2430450 wikitext text/x-wiki oaq1oeazn9hgxq9wlvp0ob76xpp3zcr 2430451 2430450 2026-06-24T20:00:46Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1242: Migration of db1242.eqiad.wmnet completed 2430451 wikitext text/x-wiki dxmqclzflc1cjutjv6abt32do8cusif 2430452 2430451 2026-06-24T20:00:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430452 wikitext text/x-wiki rf5a6u2twjvneh4y9dje89l1wu23gqv 2430453 2430452 2026-06-24T20:03:48Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2219: Migration of db2219.codfw.wmnet completed 2430453 wikitext text/x-wiki m2x1ad1b2dtctrq72kjy6rx31rm7db8 2430454 2430453 2026-06-24T20:03:51Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430454 wikitext text/x-wiki 7b0ryeadq0smyt113vznsiyjcvlo5y8 2430455 2430454 2026-06-24T20:04:08Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430455 wikitext text/x-wiki 9hgprm70w44r6sfgguinnrqlqnrcw8r 2430456 2430455 2026-06-24T20:04:19Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430456 wikitext text/x-wiki g8iy4skhl7dy1wge90om8itnitc59tx 2430458 2430456 2026-06-24T20:18:28Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430458 wikitext text/x-wiki rfavx5jarew2eb8z2o1o2ttcafjno85 2430459 2430458 2026-06-24T20:18:38Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430459 wikitext text/x-wiki pjrff6c7syha05zvodfwp2rz75atgby 2430460 2430459 2026-06-24T20:19:35Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2430460 wikitext text/x-wiki otjgwtihdb95hy0834el8onuq0tqd43 2430461 2430460 2026-06-24T20:20:02Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2430461 wikitext text/x-wiki 3yiwd2ki6vguhplmon7ewquwmyra1m6 2430462 2430461 2026-06-24T20:20:04Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2430462 wikitext text/x-wiki 813l659wiyiahijnuit15f3il6yjbsc 2430463 2430462 2026-06-24T20:20:32Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2430463 wikitext text/x-wiki 9zlb6h5l4c7kak91riuap5hdjzb42w0 2430464 2430463 2026-06-24T20:25:12Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1305500|Add $wgParserMigrationEnableParsoid as unified/fine-grained config]] 2430464 wikitext text/x-wiki oyyl2pb1vhrfc9hchs2hsaukys304qt 2430465 2430464 2026-06-24T20:27:14Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1305500|Add $wgParserMigrationEnableParsoid as unified/fine-grained config]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430465 wikitext text/x-wiki g804kyvodtktmxww17e1hz9ww636wfm 2430466 2430465 2026-06-24T20:27:37Z Stashbot 7414 XioNoX: cr1-eqiad# set chassis fpc 1 pic 1 port 5 speed 100g - T429623 2430466 wikitext text/x-wiki gd5z87hdddv3vplp20wfnl8l4hcn2ft 2430467 2430466 2026-06-24T20:28:17Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430467 wikitext text/x-wiki 5ezuzw6zd1ccdo5l9a8p3cppwcksjmh 2430468 2430467 2026-06-24T20:28:32Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2430468 wikitext text/x-wiki jvnp9zpire37to075dnplszjo7kypal 2430469 2430468 2026-06-24T20:29:12Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2430469 wikitext text/x-wiki 55524kg7v0bl9pal3ucmahc9o92ad3w 2430470 2430469 2026-06-24T20:33:34Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305500|Add $wgParserMigrationEnableParsoid as unified/fine-grained config]] (duration: 08m 21s) 2430470 wikitext text/x-wiki 48qhrs8yvrgr7mlelm7hrre01nopohg 2430471 2430470 2026-06-24T20:34:08Z Stashbot 7414 XioNoX: draining one of eqiad-codfw transports for PIC bounce 2430471 wikitext text/x-wiki tcf7gyiyznlnn2pnitky65shlwbni2h 2430472 2430471 2026-06-24T20:37:17Z Stashbot 7414 XioNoX: bouncing cr1-eqiad FPC1 PIC1 - T429623 2430472 wikitext text/x-wiki 66uymo7pnlckqa76z8tzq00ehhmye1l 2430473 2430472 2026-06-24T20:56:50Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1305503|Drop $wmgEmergencyCaptcha (T429849)]] 2430473 wikitext text/x-wiki 15ijhn2xh18r1al82qi13hrzpll4e4m 2430474 2430473 2026-06-24T20:58:53Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1305503|Drop $wmgEmergencyCaptcha (T429849)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430474 wikitext text/x-wiki hbncl5u8qvuebtkpg70sip15kgik8v7 2430475 2430474 2026-06-24T20:59:09Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2430475 wikitext text/x-wiki jtqoct4r9jc0l6p70icwrbm2q1b09rh 2430476 2430475 2026-06-24T21:03:35Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305503|Drop $wmgEmergencyCaptcha (T429849)]] (duration: 06m 43s) 2430476 wikitext text/x-wiki scfcmyb0i3jair36uvtbxz6s67s92j1 2430483 2430476 2026-06-24T21:50:14Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2430483 wikitext text/x-wiki t5sg16xy7ekmwmka0fro1dgz194pjr6 2430484 2430483 2026-06-24T21:50:38Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2430484 wikitext text/x-wiki 57ybk1poddtw6bfr9cou5moh95ppisy 2430485 2430484 2026-06-24T22:59:33Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1305191|Restore menu tab underline style (T428519)]], [[gerrit:1305499|Reduce watchstar icon size (T426131)]], [[gerrit:1305444|Replace Tools button with vertical ellipsis (T429258)]] 2430485 wikitext text/x-wiki eafzd0wclbfhjbr6knssze5p3chlatl 2430486 2430485 2026-06-24T23:01:45Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1305191|Restore menu tab underline style (T428519)]], [[gerrit:1305499|Reduce watchstar icon size (T426131)]], [[gerrit:1305444|Replace Tools button with vertical ellipsis (T429258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430486 wikitext text/x-wiki hubpqy8r7aiauiqn4clk6lcdav6pe47 2430487 2430486 2026-06-24T23:02:23Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2430487 wikitext text/x-wiki 7242mrtzf038v55cr5zapcirmpgoetz 2430488 2430487 2026-06-24T23:06:54Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305191|Restore menu tab underline style (T428519)]], [[gerrit:1305499|Reduce watchstar icon size (T426131)]], [[gerrit:1305444|Replace Tools button with vertical ellipsis (T429258)]] (duration: 07m 21s) 2430488 wikitext text/x-wiki fos8c2a5nzo9wukt2xgczwfpu1nlvcy 2430489 2430488 2026-06-24T23:10:10Z Stashbot 7414 tgr_: doing an emergency backport for T430092 2430489 wikitext text/x-wiki ffdmielg00b17amzw9zqfz4kv9w97sl 2430490 2430489 2026-06-24T23:14:39Z Stashbot 7414 tgr@deploy1003: Started scap sync-world: Backport for [[gerrit:1305524|Fix B/C break for OAuth 1 format=json option]], [[gerrit:1305525|Fix B/C break for OAuth 1 format=json option, round #2]] 2430490 wikitext text/x-wiki e2ps6mrisbo6eybiwouxry18193ze1n 2430491 2430490 2026-06-24T23:16:38Z Stashbot 7414 tgr@deploy1003: tgr: Backport for [[gerrit:1305524|Fix B/C break for OAuth 1 format=json option]], [[gerrit:1305525|Fix B/C break for OAuth 1 format=json option, round #2]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430491 wikitext text/x-wiki 8xsnnwr021atn7k0n8djlrv85i4l5ys 2430492 2430491 2026-06-24T23:22:27Z Stashbot 7414 tgr@deploy1003: tgr: Continuing with deployment 2430492 wikitext text/x-wiki ovwxmarfjoni2xsw64bvk5p3mro6keb 2430493 2430492 2026-06-24T23:26:48Z Stashbot 7414 tgr@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305524|Fix B/C break for OAuth 1 format=json option]], [[gerrit:1305525|Fix B/C break for OAuth 1 format=json option, round #2]] (duration: 12m 08s) 2430493 wikitext text/x-wiki b4ow1puqeq9tiw6v2l85qs98ed6e8qr 2430494 2430493 2026-06-24T23:28:17Z Stashbot 7414 tgr_: emergency deploy done 2430494 wikitext text/x-wiki p1caipwve1fnlw6m90j4tctfaw0xbe1 2430496 2430494 2026-06-25T02:00:26Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2430496 wikitext text/x-wiki 894mlfk5rtp12etq6dv5wceurjb9lzp 2430497 2430496 2026-06-25T02:00:50Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 00m 24s) 2430497 wikitext text/x-wiki 8xr3jjt5hgcpn7xjpfozv1n9p703ujv 2430498 2430497 2026-06-25T02:44:57Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.network.cf 2430498 wikitext text/x-wiki apobrpknaqm7l0dl5e8jzq6xycqu3vl 2430499 2430498 2026-06-25T02:44:59Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.network.cf (exit_code=0) 2430499 wikitext text/x-wiki 5hzwx7zp7h3s1wz0ddc6mnq7opgynii 2430500 2430499 2026-06-25T05:13:43Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1217,1228,1290].eqiad.wmnet with reason: Primary switchover m2 T429929 2430500 wikitext text/x-wiki bo8w33j8nceuvg6we7pxhdku0pjxpuz 2430501 2430500 2026-06-25T05:17:43Z Stashbot 7414 marostegui: Failover m2 from db1228 to db1290 - T429929 2430501 wikitext text/x-wiki h4vpv22d5aysk3nogofhrlcg7xy2qu6 2430502 2430501 2026-06-25T05:35:13Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430502 wikitext text/x-wiki to0lni9gosrztzutdfstmdtiqst3d63 2430503 2430502 2026-06-25T05:35:15Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2430503 wikitext text/x-wiki 8tlyjwo1ejsms5w7zskz3mdi10xpx7s 2430504 2430503 2026-06-25T05:46:19Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade gitlab 2430504 wikitext text/x-wiki 3usvnu87ggl4bfk8m2tniyfdi0y0o5j 2430505 2430504 2026-06-25T05:58:04Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1228.eqiad.wmnet with reason: Reimage to Trixie 2430505 wikitext text/x-wiki 8lznbyoktz13n1tb4lkuzzdrdllgb0x 2430506 2430505 2026-06-25T06:01:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1228.eqiad.wmnet with OS trixie 2430506 wikitext text/x-wiki greoynnavn6v7dbkbp6bo30o7m7pecs 2430507 2430506 2026-06-25T06:11:33Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade gitlab 2430507 wikitext text/x-wiki 7ms1l9c26cee76yo2vr6hauuqfkjay7 2430508 2430507 2026-06-25T06:15:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1228.eqiad.wmnet with reason: host reimage 2430508 wikitext text/x-wiki 187fiwwinwo89tue2q18e1bbgcoe78k 2430509 2430508 2026-06-25T06:19:59Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1228.eqiad.wmnet with reason: host reimage 2430509 wikitext text/x-wiki kbhq8x86e6gwou9txfme42hufjhp22o 2430510 2430509 2026-06-25T06:40:54Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1228.eqiad.wmnet with OS trixie 2430510 wikitext text/x-wiki rkmarlnf04c49pifxs764ftyx1ml5n9 2430514 2430510 2026-06-25T07:04:56Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1026.eqiad.wmnet with reason: Catching up 2430514 wikitext text/x-wiki qku21ksn7q7gshr340v9decbfedf6im 2430515 2430514 2026-06-25T07:07:20Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: Cloning 2430515 wikitext text/x-wiki c6mtii9pgybnz7c06ror1nqd92wyqjc 2430516 2430515 2026-06-25T07:10:43Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2234.codfw.wmnet,db1250.eqiad.wmnet with reason: Upgrading 2430516 wikitext text/x-wiki ibyee7m4zrmgkssx600w4ymls2deyqp 2430517 2430516 2026-06-25T07:11:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2234.codfw.wmnet with OS trixie 2430517 wikitext text/x-wiki 058a8bmtx0bi7kej5rtlua2sdkqbrgl 2430518 2430517 2026-06-25T07:18:05Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Upgrading 2430518 wikitext text/x-wiki 5owr4kurm5dm6f2z7cj2qs22ek9cvyj 2430519 2430518 2026-06-25T07:21:01Z Stashbot 7414 dcausse: T423993: dropping ttmserver indices from the cirrussearch opensearch clusters 2430519 wikitext text/x-wiki s8fvgjxzi3ngg0lz6vilra35b06hdov 2430520 2430519 2026-06-25T07:25:00Z Stashbot 7414 moritzm: installing nginx security updates 2430520 wikitext text/x-wiki 731fh9zrn7fhkipjuvjvapmyc3mcws7 2430521 2430520 2026-06-25T07:29:02Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@86ab691] (releasing): T430110 Test on Jenkins secondary 2430521 wikitext text/x-wiki p3gp578p46ae7uyclje2bu3a3z4wejz 2430522 2430521 2026-06-25T07:29:34Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@86ab691] (releasing): T430110 Test on Jenkins secondary (duration: 00m 50s) 2430522 wikitext text/x-wiki mpjzsd3ygraejr5ko71di7w3bfztjkc 2430523 2430522 2026-06-25T07:35:17Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2234.codfw.wmnet with OS trixie 2430523 wikitext text/x-wiki bmm08gbab69swq6y0ok44bj5ebvc51p 2430524 2430523 2026-06-25T07:35:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2234.codfw.wmnet with OS trixie 2430524 wikitext text/x-wiki ebyna1m5ph6piljgu34x4ddl17exh6r 2430525 2430524 2026-06-25T07:41:56Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2160.codfw.wmnet with reason: Upgrading 2430525 wikitext text/x-wiki lz8t5be44jh7vd4x0o19xhvbj0weeey 2430526 2430525 2026-06-25T07:47:45Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.dns.netbox 2430526 wikitext text/x-wiki swgceyw8yzz63qypcho8ei6rj7lt9tm 2430527 2430526 2026-06-25T07:51:22Z Stashbot 7414 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on releases2003.codfw.wmnet with reason: T410849 2430527 wikitext text/x-wiki fpw9jap5q1yyzfsw7fa9xeqltoejpcj 2430528 2430527 2026-06-25T07:52:09Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Allocate IPs for cloudvirt1077 - filippo@cumin1003" 2430528 wikitext text/x-wiki ruzoudtbtgidyniory6lg4kxujxi0ab 2430529 2430528 2026-06-25T07:52:13Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Allocate IPs for cloudvirt1077 - filippo@cumin1003" 2430529 wikitext text/x-wiki 45dkov4g8ix62g35yz5kbtmixhpqohs 2430530 2430529 2026-06-25T07:52:15Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430530 wikitext text/x-wiki bxaqq35qe6wgbybl5kida59idmtrd96 2430531 2430530 2026-06-25T08:02:53Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=10; selector: name=clouddb1026.eqiad.wmnet 2430531 wikitext text/x-wiki i5ip8kf1hcap7whwg1qn9xdgt1320jg 2430532 2430531 2026-06-25T08:03:13Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1026.eqiad.wmnet,service=s1 2430532 wikitext text/x-wiki 65y0shbp6hn7yrun4a1sq1aisgnds5r 2430533 2430532 2026-06-25T08:03:48Z Stashbot 7414 marostegui: Pool clouddb1026:s1 with a bit of weight T409557 2430533 wikitext text/x-wiki phx79lsj7mq2ianq26ufptxh2nyqip2 2430534 2430533 2026-06-25T08:07:09Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ec879e3] (releasing): T430110 retry Jenkins secondary 2430534 wikitext text/x-wiki 8u7vtkzv0tf6s3n344krzbt0xqu8i48 2430536 2430534 2026-06-25T08:07:39Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ec879e3] (releasing): T430110 retry Jenkins secondary (duration: 00m 53s) 2430536 wikitext text/x-wiki 5ytod2b1nwt8dg0kfcv9l0evi2pwp8n 2430537 2430536 2026-06-25T08:10:13Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ec879e3] (releasing): T430110 deploy to Jenkins primary 2430537 wikitext text/x-wiki 8nxf7q9up5sqbki3conpjpav0a3c14h 2430538 2430537 2026-06-25T08:10:56Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ec879e3] (releasing): T430110 deploy to Jenkins primary (duration: 00m 52s) 2430538 wikitext text/x-wiki tt7jegs0iql65kcsu91iy0z0vnpzymt 2430541 2430538 2026-06-25T08:44:46Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1024-1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Reimaging db1221 2430541 wikitext text/x-wiki 76k0umvk7ce6v5zuw4uiplhlfcr73ut 2430542 2430541 2026-06-25T08:45:28Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=30; selector: name=clouddb1026.eqiad.wmnet 2430542 wikitext text/x-wiki 6kd714dy0m2t6vjc4r1j87ajnqulibc 2430543 2430542 2026-06-25T08:47:19Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2006.codfw.wmnet with OS trixie 2430543 wikitext text/x-wiki pk9ukrdrylfdi3tmzoaubene0okxl23 2430544 2430543 2026-06-25T08:47:44Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430544 wikitext text/x-wiki gztl77u5zs0fd90afxhxgar6zp51uqe 2430545 2430544 2026-06-25T08:47:47Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@eqiad T429893 2430545 wikitext text/x-wiki sqy2qglgixu8v3zrruohxjkh6ei5fdy 2430546 2430545 2026-06-25T08:48:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1221: Upgrading db1221.eqiad.wmnet 2430546 wikitext text/x-wiki ic11m4rcg81j3af6t6ahwrkib6wmh0f 2430547 2430546 2026-06-25T08:48:34Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1221: Upgrading db1221.eqiad.wmnet 2430547 wikitext text/x-wiki krr9lgcork6qoh2bj42if5kb5izblh6 2430548 2430547 2026-06-25T08:50:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1221.eqiad.wmnet with OS trixie 2430548 wikitext text/x-wiki 1e60p289ltez6lbj781m4lmubxn34yz 2430549 2430548 2026-06-25T08:52:24Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430549 wikitext text/x-wiki mouv5g7b3vxmzutefsmeid83lye34bn 2430550 2430549 2026-06-25T08:52:28Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@codfw T429893 2430550 wikitext text/x-wiki 9iby83gi0v01nn8eblf1oxaf34jbqc4 2430551 2430550 2026-06-25T08:52:44Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2236: Upgrading db2236.codfw.wmnet 2430551 wikitext text/x-wiki f56g2m81sisysf6fc4zxhb6dwiau3e9 2430552 2430551 2026-06-25T08:53:06Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2236: Upgrading db2236.codfw.wmnet 2430552 wikitext text/x-wiki 658kb02iod1xfbi1piwblt2v8m5tv9n 2430554 2430552 2026-06-25T08:54:53Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2236.codfw.wmnet with OS trixie 2430554 wikitext text/x-wiki 40g4fp2bv1m3l12tw7lp566yx05u4w7 2430555 2430554 2026-06-25T08:55:49Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2234.codfw.wmnet with OS trixie 2430555 wikitext text/x-wiki sttlc2jcnjq0w3bxevbaxdeow8vmblw 2430556 2430555 2026-06-25T08:56:49Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2430556 wikitext text/x-wiki 1kmz7viwrpddtqdawd91afmc8p1didx 2430557 2430556 2026-06-25T08:57:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2430557 wikitext text/x-wiki jtrml9toxjf7l08ijg6ji5so7xcos5m 2430558 2430557 2026-06-25T08:57:49Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2430558 wikitext text/x-wiki iqyqw6yhcmh7ko0bkn4xoyx6spcnagb 2430559 2430558 2026-06-25T08:58:17Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1305599|On AW article deletion, clear all AWArticleStore from sections and metadata (T429873)]], [[gerrit:1305600|AWStorage: Use global stash keys (T430060)]] 2430559 wikitext text/x-wiki 78f2qr904lxa2lx6cb415fwm9h2swl6 2430560 2430559 2026-06-25T08:58:45Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2430560 wikitext text/x-wiki 5dvuwvsm6ds8l4uq09a1i8uu2xq9kdr 2430561 2430560 2026-06-25T09:00:25Z Stashbot 7414 jforrester@deploy1003: jforrester: Backport for [[gerrit:1305599|On AW article deletion, clear all AWArticleStore from sections and metadata (T429873)]], [[gerrit:1305600|AWStorage: Use global stash keys (T430060)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430561 wikitext text/x-wiki cn43ezr9ysy0gqvahszm9mw21wwmx0d 2430562 2430561 2026-06-25T09:00:47Z Stashbot 7414 jforrester@deploy1003: jforrester: Continuing with deployment 2430562 wikitext text/x-wiki idau729qcl547uwv5ds58bsxf25xqds 2430564 2430562 2026-06-25T09:05:28Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2006.codfw.wmnet with reason: host reimage 2430564 wikitext text/x-wiki hmwsfhvu91w18e3uk2ypbslvnt520rb 2430565 2430564 2026-06-25T09:05:46Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305599|On AW article deletion, clear all AWArticleStore from sections and metadata (T429873)]], [[gerrit:1305600|AWStorage: Use global stash keys (T430060)]] (duration: 07m 29s) 2430565 wikitext text/x-wiki gxizmg8jun9o4b63ijcmragbvsm1b7f 2430566 2430565 2026-06-25T09:06:25Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1221.eqiad.wmnet with reason: host reimage 2430566 wikitext text/x-wiki g7c1oyox9wtn9v1ehwelb448acv37u4 2430567 2430566 2026-06-25T09:06:59Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1026.eqiad.wmnet 2430567 wikitext text/x-wiki 60xw37ngqawtebhaqwlsg26tqkxe20u 2430568 2430567 2026-06-25T09:08:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2006.codfw.wmnet with reason: host reimage 2430568 wikitext text/x-wiki mlwhsj1pv552mdi6belgf35wbyftgff 2430569 2430568 2026-06-25T09:11:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2236.codfw.wmnet with reason: host reimage 2430569 wikitext text/x-wiki 5jy3y9i3umbjzxhif87pgcx1ayypr5w 2430570 2430569 2026-06-25T09:11:58Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1221.eqiad.wmnet with reason: host reimage 2430570 wikitext text/x-wiki nueo1gl7s3y62rt96ari3ghdclgcbc9 2430571 2430570 2026-06-25T09:12:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2007.codfw.wmnet with OS trixie 2430571 wikitext text/x-wiki j2j1wpvchhncse6chu57e8s5vjzjlae 2430572 2430571 2026-06-25T09:13:39Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging2008.codfw.wmnet with OS trixie 2430572 wikitext text/x-wiki 7yew04y2t2pru7wddh0fthvgmou691j 2430573 2430572 2026-06-25T09:15:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2236.codfw.wmnet with reason: host reimage 2430573 wikitext text/x-wiki tiwg63uzi8v0h37gfavuaakzgcd2h0o 2430574 2430573 2026-06-25T09:26:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2430574 wikitext text/x-wiki nssymwy1oq5r2yvl42kxx7muc84ruer 2430575 2430574 2026-06-25T09:31:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1221.eqiad.wmnet with OS trixie 2430575 wikitext text/x-wiki p20tl4rusbcind75kxxwlnh40uaw75b 2430576 2430575 2026-06-25T09:31:03Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2007.codfw.wmnet with reason: host reimage 2430576 wikitext text/x-wiki ib7l5hssrvzwfgqv0qqij0ec5u54uba 2430577 2430576 2026-06-25T09:31:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2008.codfw.wmnet with reason: host reimage 2430577 wikitext text/x-wiki h8xe9oyrlgf8rza6ij24uoxul21hujr 2430578 2430577 2026-06-25T09:33:10Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2236.codfw.wmnet with OS trixie 2430578 wikitext text/x-wiki bnqu6tftrla0f5kouy5f7c6gf7wk9lk 2430579 2430578 2026-06-25T09:34:46Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2007.codfw.wmnet with reason: host reimage 2430579 wikitext text/x-wiki 9stx6itaa6jgnvu20yout28r12cpi7g 2430580 2430579 2026-06-25T09:35:52Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1305609|hCaptcha: Skip blocked-IP score collection for crawlers in VE and MF (T429755)]] 2430580 wikitext text/x-wiki ihcunlzfyiocvols7owrodev0lr8nyg 2430581 2430580 2026-06-25T09:37:57Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1305609|hCaptcha: Skip blocked-IP score collection for crawlers in VE and MF (T429755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430581 wikitext text/x-wiki 5mvwkue3h59my1d6p5srkr9lktu7thx 2430582 2430581 2026-06-25T09:38:34Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2008.codfw.wmnet with reason: host reimage 2430582 wikitext text/x-wiki cuk3ihu1nv0x4i7b4yrrmlekjdsk7vn 2430583 2430582 2026-06-25T09:39:59Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2430583 wikitext text/x-wiki 720rg5n5r9a8zptlcdvrh4xj5543ngj 2430584 2430583 2026-06-25T09:40:01Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2006.codfw.wmnet with OS trixie 2430584 wikitext text/x-wiki f0w94s6qqy878pkp3jj2ja3nl6s7szd 2430585 2430584 2026-06-25T09:40:20Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2430585 wikitext text/x-wiki gjvycer10njs1ve4md83kfyrgutvv88 2430587 2430585 2026-06-25T09:43:26Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1221: Migration of db1221.eqiad.wmnet completed 2430587 wikitext text/x-wiki huecv3frdeugajilvqoug8bkd14hd93 2430588 2430587 2026-06-25T09:44:39Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305609|hCaptcha: Skip blocked-IP score collection for crawlers in VE and MF (T429755)]] (duration: 08m 46s) 2430588 wikitext text/x-wiki qv8sq34xp0d1ijeafwax7w5pmlycje4 2430589 2430588 2026-06-25T09:45:57Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2236: Migration of db2236.codfw.wmnet completed 2430589 wikitext text/x-wiki nsefl03opiqt9blv42nyqm4p2ene8sq 2430590 2430589 2026-06-25T09:52:21Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 6648 2430590 wikitext text/x-wiki 3el16df0m8k0aeg3s6rsyziry68xxz0 2430591 2430590 2026-06-25T09:52:37Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6648 2430591 wikitext text/x-wiki gsyyzuhpvf5ukyp3iy589epv1n1nyuf 2430592 2430591 2026-06-25T09:53:16Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2430592 wikitext text/x-wiki ol215cfy297mruaexxlcu57s5tzfaww 2430593 2430592 2026-06-25T09:58:03Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2430593 wikitext text/x-wiki dp0pan4vyl33bb9b8rlor8y2h3ulddi 2430594 2430593 2026-06-25T09:58:06Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2007.codfw.wmnet with OS trixie 2430594 wikitext text/x-wiki l75puzd2vallm6vayvp653cl7frixlo 2430595 2430594 2026-06-25T09:58:10Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2430595 wikitext text/x-wiki ejun3qb4rxrfd3i4d7whz6o7e5763x7 2430596 2430595 2026-06-25T09:58:30Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2430596 wikitext text/x-wiki 5s0h5riqjzwicu28dv8e0lfcjxckfeh 2430597 2430596 2026-06-25T09:58:32Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2008.codfw.wmnet with OS trixie 2430597 wikitext text/x-wiki c8dgu1sxb22c9yas5gkcr29qajto6n1 2430598 2430597 2026-06-25T10:03:55Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-staging-master@codfw 2430598 wikitext text/x-wiki 66aoi4zkp4h3byuvc5tmbiypedhs0uh 2430600 2430598 2026-06-25T10:07:26Z Stashbot 7414 klausman@cumin2002: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: ml-staging-master@codfw 2430600 wikitext text/x-wiki 85sbfshrhcbz2eb6egf0gx11gnckmmi 2430601 2430600 2026-06-25T10:09:59Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-staging-master@codfw 2430601 wikitext text/x-wiki tdbq0sukkb351mntnn4vr7unhxn1vgn 2430602 2430601 2026-06-25T10:13:33Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2430602 wikitext text/x-wiki jm5a33ys03auiu0hg88phiijll5sys7 2430603 2430602 2026-06-25T10:14:21Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2430603 wikitext text/x-wiki 07673f41muhhgpwgzp9pkqkgqmznq3g 2430604 2430603 2026-06-25T10:14:24Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-staging-master@codfw 2430604 wikitext text/x-wiki l10iiohd3th21i3ggzwp878wk9354s6 2430605 2430604 2026-06-25T10:15:20Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Non-deployment scap run to populate new release values 2430605 wikitext text/x-wiki gp7io9rgonwbuzsp79n24yaqvzb2loj 2430606 2430605 2026-06-25T10:15:46Z Stashbot 7414 blake@deploy1003: Stopping before sync operations 2430606 wikitext text/x-wiki mfavojfkjv0vsmf93fq3js9j8ngwc0p 2430607 2430606 2026-06-25T10:16:44Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1026.eqiad.wmnet,service=s1 2430607 wikitext text/x-wiki skla8nwwshmze4y68goiq08efp9xn51 2430608 2430607 2026-06-25T10:28:57Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1221: Migration of db1221.eqiad.wmnet completed 2430608 wikitext text/x-wiki eadgukahkss8mypo7v6h6z2aigpvwsr 2430609 2430608 2026-06-25T10:29:00Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430609 wikitext text/x-wiki psm0m58flgmv0o69qc873ty3tvagi10 2430610 2430609 2026-06-25T10:31:28Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2236: Migration of db2236.codfw.wmnet completed 2430610 wikitext text/x-wiki q39gk8z9sk7uhgb8eobcl5044kuulae 2430611 2430610 2026-06-25T10:31:31Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430611 wikitext text/x-wiki nol5fjp4ch9e217yrsnf4grj78u52v0 2430612 2430611 2026-06-25T10:37:15Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host build2003.codfw.wmnet 2430612 wikitext text/x-wiki a6xxn5c2qfeszf703ifg8iy1i8hhhii 2430613 2430612 2026-06-25T10:37:18Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host build2003.codfw.wmnet 2430613 wikitext text/x-wiki 23ig7oz8w1w3gqx61u44g7tegdl5fqv 2430614 2430613 2026-06-25T10:38:56Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host build2003.codfw.wmnet 2430614 wikitext text/x-wiki 6dtg9qtncoelz4t7sr0euxxn19oc3la 2430615 2430614 2026-06-25T10:38:58Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host build2003.codfw.wmnet 2430615 wikitext text/x-wiki b12lfxq28aibq6nerwew6r6fezbbyfx 2430617 2430615 2026-06-25T11:05:03Z Stashbot 7414 James_F: jforrester@deploy1003: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-wikifunctions_usage.sql # T428667 2430617 wikitext text/x-wiki 6mizlwgru9kq09aigo1rnr3urqblad9 2430618 2430617 2026-06-25T11:05:10Z Stashbot 7414 James_F: jforrester@deploy1003: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-wikifunctions_usage_wikis.sql # T428667 2430618 wikitext text/x-wiki 6i4mu2r46j6422kkapy3vj88kril73g 2430619 2430618 2026-06-25T11:09:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host build2004.codfw.wmnet 2430619 wikitext text/x-wiki i3yldxlp3be20y72ea7eao651hwmmis 2430620 2430619 2026-06-25T11:09:17Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.netbox 2430620 wikitext text/x-wiki 0v6ypfxsfaagtc1dcstxz1fhkzbsnhd 2430621 2430620 2026-06-25T11:14:29Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . 2430621 wikitext text/x-wiki rch08kerksfs55blbrnn55vko0ztwwq 2430622 2430621 2026-06-25T11:14:40Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . 2430622 wikitext text/x-wiki 7jc65ed7imywt18dnffsuvy8ziwaxcg 2430623 2430622 2026-06-25T11:25:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM build2004.codfw.wmnet - jmm@cumin2003" 2430623 wikitext text/x-wiki t01bfa26zhmfbxvaoao2shdnpbjiszs 2430624 2430623 2026-06-25T11:25:39Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM build2004.codfw.wmnet - jmm@cumin2003" 2430624 wikitext text/x-wiki 5dxo31k6u65emeuyg7v4pted0ox16b6 2430625 2430624 2026-06-25T11:25:41Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430625 wikitext text/x-wiki iyavgvvm0n78tzqh3kxixgadjnsxx9o 2430626 2430625 2026-06-25T11:25:43Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache build2004.codfw.wmnet on all recursors 2430626 wikitext text/x-wiki h6xe5u7z38ajbd1p82cr318dpfbspso 2430627 2430626 2026-06-25T11:25:45Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) build2004.codfw.wmnet on all recursors 2430627 wikitext text/x-wiki m0cz79sjrdbhy1rvzub8e9hi4mabxtx 2430628 2430627 2026-06-25T11:26:15Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM build2004.codfw.wmnet - jmm@cumin2003" 2430628 wikitext text/x-wiki gh0gd8ho1gijz2svdu0sqez3slskf6x 2430629 2430628 2026-06-25T11:26:19Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM build2004.codfw.wmnet - jmm@cumin2003" 2430629 wikitext text/x-wiki e3gmpz0amtfzyy6l1i79fbfefm9t6zw 2430630 2430629 2026-06-25T11:28:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host build2004.codfw.wmnet with OS trixie 2430630 wikitext text/x-wiki 6px2kcicxzlf2dmqvo6udx2pr2vkl82 2430631 2430630 2026-06-25T11:48:39Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on build2004.codfw.wmnet with reason: host reimage 2430631 wikitext text/x-wiki awwkq4n0srk3yvrmxwz3ai7atpu2odz 2430632 2430631 2026-06-25T11:50:34Z Stashbot 7414 moritzm: installing harfbuzz security updates 2430632 wikitext text/x-wiki 1gq1uv5ir86dfa4uxo1559enlmct1at 2430633 2430632 2026-06-25T11:53:36Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on build2004.codfw.wmnet with reason: host reimage 2430633 wikitext text/x-wiki ihavfhp5aqecfip33qwu7gpd7vwq0lr 2430634 2430633 2026-06-25T11:57:22Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a7-codfw,lsw1-a7-codfw IPv6,lsw1-a7-codfw.mgmt with reason: Switch maintenance 2430634 wikitext text/x-wiki of3n04gd2894rbc994zgshrqodn14cj 0 17290 2430295 2429692 2026-06-24T15:12:08Z Stashbot 7414 dancy: sudo systemctl restart php8.3-fpm on deployment-jobrunner05 (Attempting to resolve logspam) 2430295 wikitext text/x-wiki bqam0lvaiwr5xscluvm6wexswesqkmw 2430365 2430295 2026-06-24T17:26:20Z Stashbot 7414 dancy: Set `profile::puppetserver::autosign: /usr/local/sbin/validatecloudvpsfqdn.py` in hiera config for deployment-puppetserver prefix (T429413) 2430365 wikitext text/x-wiki stn2c4878feau3qhwges5l0rwad1xzi 2430599 2430365 2026-06-25T10:07:09Z Stashbot 7414 James_F: Docker: Bump Node 24 / Node 26 to new releases 2430599 wikitext text/x-wiki nygk9wcagoe3rvll7j25zrghuc68czu 498 30942 2430511 2429237 2026-06-25T06:51:11Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T424802) 2430511 wikitext text/x-wiki r3fqqe4z9jlxv9i2ajbxx0lnns5mif3 2430512 2430511 2026-06-25T06:51:19Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T424802) 2430512 wikitext text/x-wiki k7bknsc3nxkm0qotcvsz0wxmdgljvjj 0 56257 2430357 2403377 2026-06-24T16:54:26Z JKilani-WMF 37651 /* apus */ Adding time reference and context to apus exclusion 2430357 wikitext text/x-wiki a3phslby87vxzwbh96hrjeg7j6vahz2 2430586 2430357 2026-06-25T09:41:55Z BJensen-WMF 48942 /* Upcoming Switches */ Update the dates and times for the next switchover. 2430586 wikitext text/x-wiki jtj7y6s05e6uxr2jthv0fvtqa13rvxl 0 91251 2430539 2249943 2026-06-25T08:29:22Z MMuhlenhoff (WMF) 5268 Document how to add/remove URL downloaders 2430539 wikitext text/x-wiki k1oe192yk0ooco2z2yq3k0u5xzonxp6 498 441911 2430396 2429302 2026-06-24T17:57:11Z Stashbot 7414 mutante: - created second webproxy codesearch-backend and pointed to port 3002 on instance codesearch10 to fix search 2430396 wikitext text/x-wiki juuajzfjtjm278s1aiihkz5ls8zhkva 0 442175 2430616 2355785 2026-06-25T11:04:05Z Jdforrester (WMF) 468 /* Deployment */ Give the command for adding a new x1 table. 2430616 wikitext text/x-wiki caujulv4ttolobaxjnztlvn50qylv5a 498 443628 2430233 2427675 2026-06-24T14:01:26Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28103726388 (https://github.com/cluebotng/component-configs/commits/3ae736249e7b8cae3c57261c55b221a4d62296f9) 2430233 wikitext text/x-wiki ebxj11ynwg28woikmmxqmslafuzdxp8 2430354 2430233 2026-06-24T16:50:44Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01) 2430354 wikitext text/x-wiki if4mlvzp9mxwmvvq7ks6czaap40korr 2430358 2430354 2026-06-24T17:01:07Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01) 2430358 wikitext text/x-wiki sn31iznmeb3q31jv38ynfjov3ndbihe 2430361 2430358 2026-06-24T17:12:27Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01) 2430361 wikitext text/x-wiki ls8vpu6ctwdlqgqc84n2qkqipw4pi6a 2430367 2430361 2026-06-24T17:29:55Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01) 2430367 wikitext text/x-wiki iaqwlb5ridj9yuobphiyvp0qryb86bm 2430376 2430367 2026-06-24T17:38:05Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b) 2430376 wikitext text/x-wiki f6z4vnki3vpohizj4qh58r9vtnjwc01 2430388 2430376 2026-06-24T17:52:39Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b) 2430388 wikitext text/x-wiki 1ccz7h362avd8x8mwtoi1vrxumxs7sm 2430398 2430388 2026-06-24T18:02:49Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b) 2430398 wikitext text/x-wiki b34uh947ftv336ybp4604gtx4zoydft 498 444707 2430403 2428667 2026-06-24T18:06:48Z Stashbot 7414 mutante: - shutting down instance releases-trixie to avoid nagging mails about broken puppet - known issue to be worked on 2430403 wikitext text/x-wiki cyb2xccui7z81pb08daiba5yyj6y2b0 0 445717 2430535 2333179 2026-06-25T08:07:24Z BTullis (WMF) 25295 /* Check metrics about Presto */ 2430535 wikitext text/x-wiki raiu6jugeiqph7pnbdodh8e14p6r9pc 0 445919 2430281 2401040 2026-06-24T14:54:21Z Filippo Giunchedi 2939 Re-read and rework some sections 2430281 wikitext text/x-wiki epi8lkugihbrnoz1ddj0mwo2x4dua5k 0 449160 2430495 2429970 2026-06-25T00:01:51Z Dexbot 30554 Bot: Updating the report 2430495 wikitext text/x-wiki bg15ozpz02vy82tdpv9ewkxdzqnovh7 498 452890 2430379 2427634 2026-06-24T17:39:56Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117368498 (https://github.com/cluebotng/component-configs/commits/2a2ca3b7a9ed1a932fd98d3ae6694fc8b386db5d) 2430379 wikitext text/x-wiki 6098r1f8ryt1jchcamkbk3u0c71m0od 116 453906 2430513 2430114 2026-06-25T07:00:48Z Gitlabaccountapprovalbot 37332 kdh8219 was rejected. 2430513 wikitext text/x-wiki cbooby29krpyf7nkmupjao8l7fuszx5 0 454559 2430224 2300471 2026-06-24T13:42:04Z MMuhlenhoff (WMF) 5268 /* How to roll out kernel updates */ Remove Buster, add Trixie for the kernel list 2430224 wikitext text/x-wiki 7lqcp1jyry3p2x0jmoab1yi8rkg9cxr 0 458894 2430189 2426567 2026-06-24T12:49:17Z Phuedx (WMF) 47850 Troubleshooting -> How do I…? 2430189 wikitext text/x-wiki nitnpwqkg76da6sut2qz0z9oi0d7q3e 2430362 2430189 2026-06-24T17:17:58Z Phuedx (WMF) 47850 How can I check the health of the TestKitchen MediaWiki extension? 2430362 wikitext text/x-wiki muoluyx8eld6si6gc2rc21za484nbel 2430563 2430362 2026-06-25T09:02:01Z Phuedx (WMF) 47850 How can I check the health of GrowthBook? 2430563 wikitext text/x-wiki po2wcr9qp59eghyxus43sgyrts9tfq0 498 459116 2430236 2331480 2026-06-24T14:05:04Z Stashbot 7414 wmbot~dcaro@tools-bastion-15: rotated deployment token 2430236 wikitext text/x-wiki h56kjoygao32pngj1dhqj5bwj7pncjz 0 459138 2430477 2424744 2026-06-24T21:11:31Z WBrown (WMF) 37696 /* FancyCaptcha fallback mechanism */ This has changed 2430477 wikitext text/x-wiki ftkr8p7tifmzlgcvokveavuskcr4rkk 2430478 2430477 2026-06-24T21:12:04Z WBrown (WMF) 37696 /* mediawiki-config */ These have been removed 2430478 wikitext text/x-wiki dkkaokk689g0p4xbj34yclu5rb5aztt 2430479 2430478 2026-06-24T21:13:35Z WBrown (WMF) 37696 /* Runbook */ + 2430479 wikitext text/x-wiki qya9id520mxvo6xozgn6qr9fes3d5ym 2430480 2430479 2026-06-24T21:16:17Z WBrown (WMF) 37696 /* Loading hCaptcha */ update 2430480 wikitext text/x-wiki hnn3u26hx58qm1zap3czwv19ws1wbh0 2430481 2430480 2026-06-24T21:20:16Z WBrown (WMF) 37696 /* Configuration */ correct this 2430481 wikitext text/x-wiki 4owintwu6hfz3167vi0o2noatlqik11 2430482 2430481 2026-06-24T21:22:02Z WBrown (WMF) 37696 /* Enable emergency CAPTCHA */ + 2430482 wikitext text/x-wiki n2svgwogmcgxw2mpb2sagat45n2o5vd 0 459699 2430368 2429560 2026-06-24T17:30:55Z Stashbot 7414 ejegg: fundraising civicrm upgraded from 7130d7ff to a90a5451 2430368 wikitext text/x-wiki i8juiw9oxkjut30j9b5kkfii3sfjb87 0 460268 2430351 2423982 2026-06-24T16:40:20Z JVanderhoop-WMF 41069 /* Overview */ fixed typo 2430351 wikitext text/x-wiki 16t98bbe4d16as543q2wd6lbl81kg4b