Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.46.0-wmf.26 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 0 3859 2406998 2395479 2026-04-28T17:04:12Z Krinkle 44 /* foreachwikiindblist */ sql foreach example 2406998 wikitext text/x-wiki sm8ffainoq4fnpnq56vdw8kvk1xei75 2407002 2406998 2026-04-28T17:07:44Z Krinkle 44 /* foreachwikiindblist */ 2407002 wikitext text/x-wiki 3uqyvoddned0vyfw4zk4fk7qk38zw8i 0 4108 2406863 2406772 2026-04-28T12:53:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1278442]] to Tuesday, April 28 UTC afternoon backport window 2406863 wikitext text/x-wiki s67ta30zyjpz4an6zfs1j3bwdzbqhrz 2406893 2406863 2026-04-28T13:16:30Z Sohom Datta 15330 /* {{Deployment_day|date=2026-04-28}} */ +add backport! 2406893 wikitext text/x-wiki iy1tid0y2ltb36za3w4r49omy1uk2qz 2406896 2406893 2026-04-28T13:17:17Z Sohom Datta 15330 /* {{Deployment_day|date=2026-04-28}} */ nope, config 2406896 wikitext text/x-wiki 7xj6eb9amid9rv2cak0eureoc8hxq3j 2406944 2406896 2026-04-28T15:14:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1278494]] to Tuesday, April 28 UTC late backport window 2406944 wikitext text/x-wiki g42sjdnwwvfejcqxg2d0o03ph1n7t45 2406961 2406944 2026-04-28T15:42:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1277701]] to Tuesday, April 28 UTC late backport window 2406961 wikitext text/x-wiki el9eizqg40kcxs22jwxs6hhkfclg01s 2406964 2406961 2026-04-28T15:48:45Z ScheduleDeploymentBot 37566 Add [[gerrit:1278498]] to Tuesday, April 28 UTC late backport window 2406964 wikitext text/x-wiki i2g6h4dgq6ikvfrwz4tal5y6cd4d2h7 2406972 2406964 2026-04-28T16:19:14Z ScheduleDeploymentBot 37566 Add [[gerrit:1278509]] to Wednesday, April 29 UTC morning backport window 2406972 wikitext text/x-wiki 7wgiqgvn0saht7h39bqz6mzmmi239d1 2407030 2406972 2026-04-28T18:29:54Z TCipriani (WMF) 4895 2407030 wikitext text/x-wiki ee649pge75hhp5c7wprisdskkb7jexs 2407032 2407030 2026-04-28T18:33:17Z ScheduleDeploymentBot 37566 Add [[gerrit:1277667]] to Monday, May 04 UTC late backport window 2407032 wikitext text/x-wiki 0o3pi7o5x1hwfygacer2i858b43ins3 2407038 2407032 2026-04-28T19:02:02Z ScheduleDeploymentBot 37566 Add [[gerrit:1278538]] to Tuesday, April 28 UTC late backport window 2407038 wikitext text/x-wiki pgyt1q6idrs89zdbskaxif2b2mwuc87 2407055 2407038 2026-04-28T20:50:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1278584]] to Wednesday, April 29 UTC afternoon backport window 2407055 wikitext text/x-wiki psoeufollkfhm3udrxi4xfqk66xh7z0 2407072 2407055 2026-04-28T22:45:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1270882]] to Tuesday, May 05 UTC afternoon backport window 2407072 wikitext text/x-wiki hvyd6t1u88mc1y7mjh3q16wlom3tdvl 2407073 2407072 2026-04-28T22:45:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1271969]] to Tuesday, May 05 UTC afternoon backport window 2407073 wikitext text/x-wiki c26zahl9ebimoapwfysm11r1jov2azp 2407082 2407073 2026-04-29T02:03:05Z ScheduleDeploymentBot 37566 Add [[gerrit:1265959]] to Monday, May 04 UTC morning backport window 2407082 wikitext text/x-wiki or5lynm1wsru65o2wkdrdx5715vulso 2407113 2407082 2026-04-29T05:40:05Z JKilani-WMF 37651 2407113 wikitext text/x-wiki hgm1vv8gb3vzr7hza9d92g3ugtnghh2 2407114 2407113 2026-04-29T05:43:14Z ScheduleDeploymentBot 37566 Add [[gerrit:1279078]] to Wednesday, April 29 UTC morning backport window 2407114 wikitext text/x-wiki 81qrfcim765frhhvhlt6rvfnhw95bdz 2407115 2407114 2026-04-29T05:44:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1279079]] to Wednesday, April 29 UTC morning backport window 2407115 wikitext text/x-wiki fvpkif3668fqr34vonn3tk47x2eyl46 2407117 2407115 2026-04-29T05:45:13Z APatro (WMF) 38793 /* {{Deployment_day|date=2026-04-29}} */ 2407117 wikitext text/x-wiki a9cbyk70569520ae078kq9z0055y0qv 2407146 2407117 2026-04-29T06:42:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1269464]] to Wednesday, April 29 UTC morning backport window 2407146 wikitext text/x-wiki 67fznpvif6ness75f9qw57erfuicnn5 2407217 2407146 2026-04-29T08:39:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1278380]] to Wednesday, April 29 UTC afternoon backport window 2407217 wikitext text/x-wiki r4ya9xiutirh6k6mdyi8tl4q14r5bdr 2407327 2407217 2026-04-29T11:35:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1279279]] to Wednesday, April 29 UTC afternoon backport window 2407327 wikitext text/x-wiki edxa9of17fdawgq5ioxv4kz7f57tc1y 2407328 2407327 2026-04-29T11:35:40Z ScheduleDeploymentBot 37566 Add [[gerrit:1279280]] to Wednesday, April 29 UTC afternoon backport window 2407328 wikitext text/x-wiki 79htx7io69guyiffqjovwzydh4m9z2t 2407329 2407328 2026-04-29T11:35:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1279269]] to Wednesday, April 29 UTC afternoon backport window 2407329 wikitext text/x-wiki ho5o301kjklj7koai5te5kmgowcnbpf 498 6726 2406962 2406770 2026-04-28T15:46:53Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system 2406962 wikitext text/x-wiki 21wz7k5n1qqujn3ndfo3dtuxrslpmnf 2406963 2406962 2026-04-28T15:47:06Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system 2406963 wikitext text/x-wiki tft3c8hkl1y68opc61wbhs4oxmxwyq5 2406969 2406963 2026-04-28T15:53:26Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system 2406969 wikitext text/x-wiki i7djqbhdboz6yx3ef60ulmn3cm574iz 2406970 2406969 2026-04-28T15:53:40Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system 2406970 wikitext text/x-wiki 7hncqkhwjyoi79f2taam7iw4asrkztc 2407039 2406970 2026-04-28T19:43:08Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api 2407039 wikitext text/x-wiki nsnv541bpw1hgra8tbctqw1zwqp66tr 2407040 2407039 2026-04-28T19:43:10Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api 2407040 wikitext text/x-wiki m5qe5s8hk0ai13jwu02dgtsdt4e6mbi 0 7195 2407045 2401391 2026-04-28T20:14:55Z BKing (WMF) 29017 /* Hostname prefixes */ Add wdqs-internal-main and wdqs-internal-scholarly 2407045 wikitext text/x-wiki dzmovri3tbkumayrcj7uhm181nalpdm 0 7814 2407261 2382496 2026-04-29T09:43:55Z BTullis (WMF) 25295 /* jumbo (eqiad) */ Update to reflect that MirrorMakers are in k8s now 2407261 wikitext text/x-wiki e5ghm0ehqgn2ifx6734fqbtk11f9ex1 0 7919 2406823 2406821 2026-04-28T12:04:41Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2406823 wikitext text/x-wiki kr2lohtez7wvwtbu3tks7ew5k9y7x0n 2406824 2406823 2026-04-28T12:04:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P91776 and previous config saved to /var/cache/conftool/dbconfig/20260428-120450-fceratto.json 2406824 wikitext text/x-wiki qqetevpgkdpfhwwp2wpcui4i1btuh32 2406825 2406824 2026-04-28T12:11:53Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2213.codfw.wmnet with OS trixie 2406825 wikitext text/x-wiki 0gn2nahji2d0fk6vxbo2pxydbx3sahq 2406826 2406825 2026-04-28T12:13:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts doh5001.wikimedia.org 2406826 wikitext text/x-wiki b60051jaoiqqcda3dm9n1g3m40twfm5 2406827 2406826 2026-04-28T12:15:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T419635)', diff saved to https://phabricator.wikimedia.org/P91777 and previous config saved to /var/cache/conftool/dbconfig/20260428-121458-fceratto.json 2406827 wikitext text/x-wiki h4e4ap2rex5g57fsywvvdu4a4hc9r11 2406828 2406827 2026-04-28T12:15:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2213: after reimage to trixie 2406828 wikitext text/x-wiki e17wo6zbzziu5cvtvjl9rlx12vyfn65 2406829 2406828 2026-04-28T12:15:17Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2228.codfw.wmnet with reason: Maintenance 2406829 wikitext text/x-wiki 3gvr6sskrrrypzs65s1pzylynopfwsx 2406830 2406829 2026-04-28T12:15:25Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2226.codfw.wmnet with reason: Reimage to Trixie 2406830 wikitext text/x-wiki mg53ednpdrdgjdqbby00ah7uzkafvvb 2406831 2406830 2026-04-28T12:15:27Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1259.eqiad.wmnet with reason: Reimage to Trixie 2406831 wikitext text/x-wiki m93ocnu1h8es29cpiayk5mn739hb2c9 2406832 2406831 2026-04-28T12:15:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2226: Reimage to Trixie 2406832 wikitext text/x-wiki r4o7sld87bwkky0y3h4ug1mxtnpqdn7 2406833 2406832 2026-04-28T12:15:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1259: Reimage to Trixie 2406833 wikitext text/x-wiki 57bs8pbs6pn1o25pm70b90fvuy9ekw7 2406834 2406833 2026-04-28T12:15:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 (T419635)', diff saved to https://phabricator.wikimedia.org/P91779 and previous config saved to /var/cache/conftool/dbconfig/20260428-121530-fceratto.json 2406834 wikitext text/x-wiki b8oh1hiztaco65uzounx7cblcbhqaqq 2406835 2406834 2026-04-28T12:15:59Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1259: Reimage to Trixie 2406835 wikitext text/x-wiki 905h17u2dyuy7gc30n7c9he6r8sj58p 2406836 2406835 2026-04-28T12:16:01Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2226: Reimage to Trixie 2406836 wikitext text/x-wiki iat0b8j848vbg0iwwh2efl6u35h9b76 2406837 2406836 2026-04-28T12:17:01Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406837 wikitext text/x-wiki 5gb72l8n6r01kf00u0gyabo6r8o09px 2406838 2406837 2026-04-28T12:17:05Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406838 wikitext text/x-wiki ry02lb8atx80febl06gxj9z8vj3r5ie 2406839 2406838 2026-04-28T12:18:05Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2226.codfw.wmnet with OS trixie 2406839 wikitext text/x-wiki 92nysniu7ut2sxm0yir0s4g31ump6p0 2406840 2406839 2026-04-28T12:18:22Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1259.eqiad.wmnet with OS trixie 2406840 wikitext text/x-wiki m8fuq2h04r7z9d65xx6fxj3m9kwsduc 2406841 2406840 2026-04-28T12:18:35Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2406841 wikitext text/x-wiki i04cuuj2bcu29u0uldmzrdyoilz9okq 2406842 2406841 2026-04-28T12:22:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T419635)', diff saved to https://phabricator.wikimedia.org/P91782 and previous config saved to /var/cache/conftool/dbconfig/20260428-122211-fceratto.json 2406842 wikitext text/x-wiki 5j9s685jlonv7da5yqr3affsaryxjuq 2406843 2406842 2026-04-28T12:22:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: doh5001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406843 wikitext text/x-wiki psfp0araw5xm1l9sh305adl1buuucbn 2406844 2406843 2026-04-28T12:22:55Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: doh5001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406844 wikitext text/x-wiki phka80obh1622jhhw2dy3gyw04rgd7f 2406845 2406844 2026-04-28T12:22:58Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406845 wikitext text/x-wiki akt9s13uqf4spdorbu2y0toptpg5bhc 2406846 2406845 2026-04-28T12:23:01Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts doh5001.wikimedia.org 2406846 wikitext text/x-wiki 6z0so85czrd2w0adkrp39dmrrkj51iq 2406847 2406846 2026-04-28T12:27:29Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2406847 wikitext text/x-wiki cqmzag9d6jru92g2m4p2edo574d41kn 2406848 2406847 2026-04-28T12:30:42Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2406848 wikitext text/x-wiki ed4ligalex8l9b0sal16e9q62n3zwag 2406849 2406848 2026-04-28T12:30:51Z Stashbot 7414 moritzm: installing openjdk-21 security updates 2406849 wikitext text/x-wiki gjcily61k3184huvteat054znwfqd1v 2406850 2406849 2026-04-28T12:32:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P91785 and previous config saved to /var/cache/conftool/dbconfig/20260428-123220-fceratto.json 2406850 wikitext text/x-wiki 0u3dwdyae4w349wek7ohe97zybwx0ns 2406851 2406850 2026-04-28T12:34:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2226.codfw.wmnet with reason: host reimage 2406851 wikitext text/x-wiki 9tb7vsm7gwffbuhnsq1fxp2o3i4c7m4 2406852 2406851 2026-04-28T12:34:55Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1259.eqiad.wmnet with reason: host reimage 2406852 wikitext text/x-wiki lli59q3o9qkhpv9it8m6hr2s7t3g36h 2406853 2406852 2026-04-28T12:38:08Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2226.codfw.wmnet with reason: host reimage 2406853 wikitext text/x-wiki 5i7kfzm9t8ii8faqyngzbq9evt1i7tw 2406854 2406853 2026-04-28T12:40:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance 2406854 wikitext text/x-wiki gua11s5bz0snzsgvj2b02wqzvcqxxbz 2406856 2406854 2026-04-28T12:40:36Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance 2406856 wikitext text/x-wiki 0glcoem1vrpu15j8fnzvdyi1k7wjvmc 2406857 2406856 2026-04-28T12:40:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1158 (T419961)', diff saved to https://phabricator.wikimedia.org/P91786 and previous config saved to /var/cache/conftool/dbconfig/20260428-124042-fceratto.json 2406857 wikitext text/x-wiki ijrmlfl8g5sixa1f2x59w4hxwj6nyje 2406858 2406857 2026-04-28T12:41:02Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1259.eqiad.wmnet with reason: host reimage 2406858 wikitext text/x-wiki 8oidhxiywjytoias0y50heo43vnhsth 2406859 2406858 2026-04-28T12:42:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P91787 and previous config saved to /var/cache/conftool/dbconfig/20260428-124228-fceratto.json 2406859 wikitext text/x-wiki f7yplmdyiuagcqaftx0dmdmpz8xz8nj 2406861 2406859 2026-04-28T12:49:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T419961)', diff saved to https://phabricator.wikimedia.org/P91789 and previous config saved to /var/cache/conftool/dbconfig/20260428-124859-fceratto.json 2406861 wikitext text/x-wiki 6byso0z3e1plltmvzqyawz5alf2l0qo 2406862 2406861 2026-04-28T12:52:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T419635)', diff saved to https://phabricator.wikimedia.org/P91790 and previous config saved to /var/cache/conftool/dbconfig/20260428-125236-fceratto.json 2406862 wikitext text/x-wiki 9rht7wjzei6milwlrij0e6u1e993pgu 2406864 2406862 2026-04-28T12:56:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts doh5002.wikimedia.org 2406864 wikitext text/x-wiki tlbj28wger4eo1vh74lfo0chestfvxl 2406865 2406864 2026-04-28T12:59:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P91791 and previous config saved to /var/cache/conftool/dbconfig/20260428-125907-fceratto.json 2406865 wikitext text/x-wiki 5igt6o8s0n835m2ko3wvcemw4jewc8a 2406866 2406865 2026-04-28T13:00:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2213: after reimage to trixie 2406866 wikitext text/x-wiki 2rw180hopb4rnlnw432vdxmgvwhhhkm 2406867 2406866 2026-04-28T13:01:29Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2406867 wikitext text/x-wiki jhoho7tb1ny00wql2floakb2wugm735 2406868 2406867 2026-04-28T13:01:32Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply 2406868 wikitext text/x-wiki 14t3w3qojd8yl8dem5u6ey7duubgeci 2406869 2406868 2026-04-28T13:01:44Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2226.codfw.wmnet with OS trixie 2406869 wikitext text/x-wiki biej7k9t5yxbocs5at41q57sra4jf55 2406870 2406869 2026-04-28T13:02:44Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply 2406870 wikitext text/x-wiki bzm4wkwwzoyc1ba4a1qy18e1ygltd54 2406871 2406870 2026-04-28T13:03:15Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply 2406871 wikitext text/x-wiki aez03s1eofw2hucv7321whvjnkdqh3h 2406872 2406871 2026-04-28T13:03:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1259.eqiad.wmnet with OS trixie 2406872 wikitext text/x-wiki bdcmiuxz4al877c6zihpzdjkkk254on 2406873 2406872 2026-04-28T13:04:05Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2226: after reimage to trixie 2406873 wikitext text/x-wiki qq2kru7ycbz7upwc31v24ax0h1fyukq 2406874 2406873 2026-04-28T13:05:06Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply 2406874 wikitext text/x-wiki 36qc9w0qhf4k2fc0a3q32pmspz95901 2406875 2406874 2026-04-28T13:05:23Z Stashbot 7414 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply 2406875 wikitext text/x-wiki 7q52oxv5hjvb5f040jkqa05wgcniu8v 2406876 2406875 2026-04-28T13:06:22Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: doh5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406876 wikitext text/x-wiki 7xt7j0ppbvn7vsq2xd5qa0mnvn2lo0q 2406877 2406876 2026-04-28T13:06:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: doh5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2406877 wikitext text/x-wiki 3b2kmne49v9oivhz378naznpk0pxa8o 2406878 2406877 2026-04-28T13:06:41Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406878 wikitext text/x-wiki ii25vytzresoggj1swgtkz4nmvg94jh 2406879 2406878 2026-04-28T13:06:44Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts doh5002.wikimedia.org 2406879 wikitext text/x-wiki 1gnxq1rj50d3x8e0v44lz8mnd9ahqou 2406880 2406879 2026-04-28T13:07:18Z Stashbot 7414 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply 2406880 wikitext text/x-wiki pbqs5jda1sg0b7l5rd5zr7u9n6qv2hu 2406881 2406880 2026-04-28T13:07:20Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1259: after reimage to trixie 2406881 wikitext text/x-wiki h4f2bdymsuz6csjm7oo7rn7rr7paqjk 2406882 2406881 2026-04-28T13:07:50Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2141.codfw.wmnet with reason: decom 2406882 wikitext text/x-wiki t9wktbunv3dgnmi0nich286bwbyde3b 2406883 2406882 2026-04-28T13:09:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P91795 and previous config saved to /var/cache/conftool/dbconfig/20260428-130915-fceratto.json 2406883 wikitext text/x-wiki nejvk1b620lpofos7z0opatxuqpoanc 2406885 2406883 2026-04-28T13:12:40Z Stashbot 7414 moritzm: remove ganeti5005 from eqsin cluster T421863 2406885 wikitext text/x-wiki gs460m3w3w7ylcwofh0ad6fc7ie7chm 2406886 2406885 2026-04-28T13:13:24Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts db2141.codfw.wmnet 2406886 wikitext text/x-wiki 0xs0ewv9zmnplqsb7i1kn2ucu1upykz 2406887 2406886 2026-04-28T13:13:40Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts dborch1001.wikimedia.org 2406887 wikitext text/x-wiki 52oqtvql2a4tt2t13krormzy4qj1zfs 2406889 2406887 2026-04-28T13:15:58Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply 2406889 wikitext text/x-wiki 2m21l11t9sy04lno8e8rokjgcqyehgk 2406890 2406889 2026-04-28T13:16:01Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply 2406890 wikitext text/x-wiki ir0fpb5ktzrt5rj1wj8yawj1v8k7m5f 2406891 2406890 2026-04-28T13:16:12Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2406891 wikitext text/x-wiki no1gui8l01itktgkxdysht8ba7kdr2z 2406892 2406891 2026-04-28T13:16:21Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1278381|Update action parameter for bulk blocking instrumented events (T420517)]], [[gerrit:1278442|Resources: Define required message for 'oojs-ui-windows' module (T424653)]] 2406892 wikitext text/x-wiki qlqwvhnfot5pt773jqd72z2vzuiplx3 2406894 2406892 2026-04-28T13:16:57Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2406894 wikitext text/x-wiki 307ku8r3ax9fp2sgacitxntpc3eva4h 2406895 2406894 2026-04-28T13:17:05Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2406895 wikitext text/x-wiki nuub92k1gl5pco3nl8t1nfgreo86di1 2406897 2406895 2026-04-28T13:17:47Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2406897 wikitext text/x-wiki 3zexotud89uhtd19k21501wuysg9bhm 2406898 2406897 2026-04-28T13:18:46Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2406898 wikitext text/x-wiki 04jmwj9pesc4kqk4f1kbnw5kc506wh8 2406899 2406898 2026-04-28T13:19:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T419961)', diff saved to https://phabricator.wikimedia.org/P91797 and previous config saved to /var/cache/conftool/dbconfig/20260428-131924-fceratto.json 2406899 wikitext text/x-wiki 2d3k2ynchs0o92qwpwhipptx58cao9q 2406900 2406899 2026-04-28T13:19:47Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance 2406900 wikitext text/x-wiki i27v7slirbk6yr4j8hfmept5bawk5jd 2406901 2406900 2026-04-28T13:19:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1170 (T419961)', diff saved to https://phabricator.wikimedia.org/P91798 and previous config saved to /var/cache/conftool/dbconfig/20260428-131953-fceratto.json 2406901 wikitext text/x-wiki 5ls6hujzdi07vzc4rnnkhwxyq88776i 2406902 2406901 2026-04-28T13:22:13Z Stashbot 7414 stran@deploy1003: dreamyjazz, stran: Backport for [[gerrit:1278381|Update action parameter for bulk blocking instrumented events (T420517)]], [[gerrit:1278442|Resources: Define required message for 'oojs-ui-windows' module (T424653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406902 wikitext text/x-wiki 6qhkjkardvryi3ppk1yghe50ujhxync 2406903 2406902 2026-04-28T13:23:56Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.dns.netbox 2406903 wikitext text/x-wiki 1evobee1hitcyj11upg07mimbyh8221 2406904 2406903 2026-04-28T13:25:29Z Stashbot 7414 stran@deploy1003: dreamyjazz, stran: Continuing with deployment 2406904 wikitext text/x-wiki 1jp93izjsldlqimplr0w0uo7bts9h33 2406906 2406904 2026-04-28T13:26:40Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406906 wikitext text/x-wiki ln7s5dttdnhjwq5l7hqj3i83ny4cvb1 2406907 2406906 2026-04-28T13:26:42Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2141.codfw.wmnet 2406907 wikitext text/x-wiki 4ujxo5nlo5v70btf9d22z6sa7lyh4p9 2406908 2406907 2026-04-28T13:26:51Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dborch1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2406908 wikitext text/x-wiki rg2wtgx1oh2pbiahlq3l3i6r0jh48py 2406909 2406908 2026-04-28T13:28:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T419961)', diff saved to https://phabricator.wikimedia.org/P91800 and previous config saved to /var/cache/conftool/dbconfig/20260428-132822-fceratto.json 2406909 wikitext text/x-wiki ctji61j92gxf5as77lzfd3oyztsmybp 2406910 2406909 2026-04-28T13:28:34Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dborch1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2406910 wikitext text/x-wiki tecsy0bf0xh7r78u6wdi4screaim47x 2406911 2406910 2026-04-28T13:28:37Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2406911 wikitext text/x-wiki k77hf8vesl68k6tzmbjw46gl1kp9h1u 2406912 2406911 2026-04-28T13:28:39Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dborch1001.wikimedia.org 2406912 wikitext text/x-wiki k65z7ejnlhtbuo8zqzi8t43up9m5nd3 2406913 2406912 2026-04-28T13:31:30Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278381|Update action parameter for bulk blocking instrumented events (T420517)]], [[gerrit:1278442|Resources: Define required message for 'oojs-ui-windows' module (T424653)]] (duration: 15m 10s) 2406913 wikitext text/x-wiki nsk3eo3isr076sxcjygn3cjz015ei0u 2406916 2406913 2026-04-28T13:35:01Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1278435|testwiki: allow sysops to add/remove electionadmin (T423962)]] 2406916 wikitext text/x-wiki lbif6qxo8t6uct6wkfq1833g2rmy6t2 2406917 2406916 2026-04-28T13:36:49Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, novemlinguae: Backport for [[gerrit:1278435|testwiki: allow sysops to add/remove electionadmin (T423962)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406917 wikitext text/x-wiki k84n4jnydqupl8q2tc910j64sr6t8yg 2406918 2406917 2026-04-28T13:38:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P91803 and previous config saved to /var/cache/conftool/dbconfig/20260428-133830-fceratto.json 2406918 wikitext text/x-wiki h6spxvki6jt04rr8lawo03epniwarnz 2406919 2406918 2026-04-28T13:38:51Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, novemlinguae: Continuing with deployment 2406919 wikitext text/x-wiki 30ivtobcob1mrn6x7h6qbj81ok22ce8 2406920 2406919 2026-04-28T13:42:38Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278435|testwiki: allow sysops to add/remove electionadmin (T423962)]] (duration: 07m 37s) 2406920 wikitext text/x-wiki lgcgu0uoaakt1eeo4dgy0aojo6zjdwn 2406921 2406920 2026-04-28T13:43:14Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2406921 wikitext text/x-wiki ajnr44w3fozmhqfh0sspt0jcb2n7jp5 2406922 2406921 2026-04-28T13:48:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P91804 and previous config saved to /var/cache/conftool/dbconfig/20260428-134838-fceratto.json 2406922 wikitext text/x-wiki ijo6hel0e80er3nohg3aqmfogi9q9nd 2406923 2406922 2026-04-28T13:49:03Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reimage for host phab2003.codfw.wmnet with OS bullseye 2406923 wikitext text/x-wiki tqpvmmiv7lpyvo9nqwe5bog7olosbpu 2406924 2406923 2026-04-28T13:49:31Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2226: after reimage to trixie 2406924 wikitext text/x-wiki fgf9kn9kmmcqkkilhe6uufto10n2hiz 2406925 2406924 2026-04-28T13:52:45Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1259: after reimage to trixie 2406925 wikitext text/x-wiki hrwduec5if95hoesrd16xtdrnpatv72 2406926 2406925 2026-04-28T13:54:58Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2406926 wikitext text/x-wiki 7sn6iahk9hpf4vnboyljqfccd8qvcf7 2406927 2406926 2026-04-28T13:55:01Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2406927 wikitext text/x-wiki qvkuq6mgw466qahd62wtubasazhs0i1 2406928 2406927 2026-04-28T13:58:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T419961)', diff saved to https://phabricator.wikimedia.org/P91807 and previous config saved to /var/cache/conftool/dbconfig/20260428-135847-fceratto.json 2406928 wikitext text/x-wiki a6w1ds94h8fv0zxldqozzatdrp9cnt3 2406931 2406928 2026-04-28T14:26:24Z Stashbot 7414 herron@cumin1003: START - Cookbook sre.kafka.change-confluent-distro-version Change Confluent distribution for Kafka A:kafka-logging-eqiad cluster: Change Confluent distribution. 2406931 wikitext text/x-wiki 5eo5oz9quvhv8w06z3uatxs0uvwz1ga 2406934 2406931 2026-04-28T14:37:59Z Stashbot 7414 aokoth@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host phab2003.codfw.wmnet with OS bullseye 2406934 wikitext text/x-wiki dk5gi2mcd9dqtndy9gxrxmw9taob4ml 2406935 2406934 2026-04-28T14:47:18Z Stashbot 7414 herron@cumin1003: END (PASS) - Cookbook sre.kafka.change-confluent-distro-version (exit_code=0) Change Confluent distribution for Kafka A:kafka-logging-eqiad cluster: Change Confluent distribution. 2406935 wikitext text/x-wiki t2qlnvi97p48nqtny43f15j22vw9zr1 2406937 2406935 2026-04-28T15:02:02Z Stashbot 7414 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator deploy 2406937 wikitext text/x-wiki inyzhe72xw24o32v16fyknt38ngtclz 2406938 2406937 2026-04-28T15:02:31Z Stashbot 7414 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy 2406938 wikitext text/x-wiki 9kyh8kkjozw64i47dnxxl9p5ybft1wu 2406939 2406938 2026-04-28T15:03:46Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@ce0b865]: deploy phab2002 for T424656 2406939 wikitext text/x-wiki drqkp2mg622x9j7j8byv2liubj2sq7i 2406940 2406939 2026-04-28T15:04:30Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@ce0b865]: deploy phab2002 for T424656 (duration: 00m 44s) 2406940 wikitext text/x-wiki 0h7gybhwazxragm93pw9s3tc8h42cto 2406941 2406940 2026-04-28T15:04:53Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@ce0b865]: deploy phab1004 for T424656 2406941 wikitext text/x-wiki epmw656bkzyuylh4fl3j3shlnt47cwj 2406942 2406941 2026-04-28T15:05:55Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@ce0b865]: deploy phab1004 for T424656 (duration: 00m 52s) 2406942 wikitext text/x-wiki kjmqtdsfpvie2of15o8k9hhm8p2rtja 2406943 2406942 2026-04-28T15:14:45Z Stashbot 7414 herron@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad 2406943 wikitext text/x-wiki 5qwfwbh82lc31ih2i23qye4u69idi46 2406946 2406943 2026-04-28T15:16:19Z Stashbot 7414 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2406946 wikitext text/x-wiki 49wrvfqrbucmfe1q3ubuhgmnvq38b92 2406947 2406946 2026-04-28T15:16:51Z Stashbot 7414 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2406947 wikitext text/x-wiki srnmp2ji8flub0p6xziww4pg3oeph92 2406948 2406947 2026-04-28T15:17:49Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2406948 wikitext text/x-wiki htzs9az7owk6b50aki8uzj6o48odxcj 2406949 2406948 2026-04-28T15:18:32Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2406949 wikitext text/x-wiki g8poo81yer52os4xfiyjabz6c9sdx0n 2406950 2406949 2026-04-28T15:18:57Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2406950 wikitext text/x-wiki j1a7sszs243sbl428moj70f9lxoz6fe 2406951 2406950 2026-04-28T15:19:05Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2406951 wikitext text/x-wiki nxqmwxbxomdk1r54wdimume71qeygmh 2406952 2406951 2026-04-28T15:19:14Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2406952 wikitext text/x-wiki 8uwue3o3ne4z30opbrcn9s31n71egok 2406953 2406952 2026-04-28T15:19:30Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2406953 wikitext text/x-wiki bqf9tcqe990ldlzp0zijfoc6rtuv2x5 2406954 2406953 2026-04-28T15:19:47Z Stashbot 7414 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1278476|EventStreamConfig - Declare .v1 streams for html content and feature counts (T423920)]] 2406954 wikitext text/x-wiki k4nbon95lpi3o3j2o4pfij9feowpli8 2406955 2406954 2026-04-28T15:21:37Z Stashbot 7414 otto@deploy1003: otto: Backport for [[gerrit:1278476|EventStreamConfig - Declare .v1 streams for html content and feature counts (T423920)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2406955 wikitext text/x-wiki f48gfjri0soa090j3zff4b5jskw77wt 2406956 2406955 2026-04-28T15:24:13Z Stashbot 7414 otto@deploy1003: otto: Continuing with deployment 2406956 wikitext text/x-wiki r3prw2349r1okqatvisl5iqipb3vz90 2406957 2406956 2026-04-28T15:28:08Z Stashbot 7414 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278476|EventStreamConfig - Declare .v1 streams for html content and feature counts (T423920)]] (duration: 08m 21s) 2406957 wikitext text/x-wiki qigsn63hqi8z4g5skebsg67kx3x4kt9 2406959 2406957 2026-04-28T15:34:15Z Stashbot 7414 herron@cumin1003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad 2406959 wikitext text/x-wiki 4w1uyxyjoy7j2gkambim8sh90la4wia 2406967 2406959 2026-04-28T15:52:49Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2406967 wikitext text/x-wiki 43es7xc5jry43fm1jgblsuu0u367cu0 2406973 2406967 2026-04-28T16:22:00Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply 2406973 wikitext text/x-wiki i5z1owmpqlju3r6zlbn38jj2kw75cn2 2406974 2406973 2026-04-28T16:22:07Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply 2406974 wikitext text/x-wiki ek72yvpapbtlx5loinyjnwwrtv2a4ic 2406975 2406974 2026-04-28T16:22:14Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply 2406975 wikitext text/x-wiki b6m1qfil0hmqcd2s1dsgjvqayy3wh2w 2406976 2406975 2026-04-28T16:22:19Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply 2406976 wikitext text/x-wiki r566dj5dv0u1km3mkifltr0sjj0gw8o 2406977 2406976 2026-04-28T16:22:32Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2406977 wikitext text/x-wiki hlwl8gsz7l37zkv8rub6s6ugh4tam04 2406978 2406977 2026-04-28T16:22:39Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2406978 wikitext text/x-wiki n72leh2oszb2b6yc79nyy6kpyvyoubf 2406979 2406978 2026-04-28T16:22:46Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2406979 wikitext text/x-wiki qov4isald3rxvh9yigs13cmu2xn0fd7 2406980 2406979 2026-04-28T16:22:50Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2406980 wikitext text/x-wiki arno12d23nb1e8osejk2fizve0cs0rm 2406981 2406980 2026-04-28T16:22:58Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2406981 wikitext text/x-wiki aitcugjm7jnnygzv0mdbtooeol1rzj6 2406982 2406981 2026-04-28T16:23:05Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2406982 wikitext text/x-wiki pki4fingv6h0sxrjre4x8a8bnn0guz3 2406983 2406982 2026-04-28T16:23:12Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2406983 wikitext text/x-wiki reaggkjrbxvgajzq5ht8gzx3rwf6t4g 2406984 2406983 2026-04-28T16:23:16Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2406984 wikitext text/x-wiki f8j4skh8jir1l3bivjrccpex3rtj7s3 2406985 2406984 2026-04-28T16:23:24Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406985 wikitext text/x-wiki lbpjbxpgsu9ke8mho5s5sba6yy83xob 2406986 2406985 2026-04-28T16:23:30Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406986 wikitext text/x-wiki mvsww2klr6ep0cml9uq2sk56domfng4 2406987 2406986 2026-04-28T16:23:36Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406987 wikitext text/x-wiki f1dfdnr7ufng0q1flbom7fk0ni5vqng 2406988 2406987 2026-04-28T16:23:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply 2406988 wikitext text/x-wiki 6mmi7llvwcp58fhc65n3943rodl7t8r 2406989 2406988 2026-04-28T16:23:49Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406989 wikitext text/x-wiki 5vhg8ivgyhfeh49pqn4si0i1umgth2g 2406990 2406989 2026-04-28T16:23:57Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406990 wikitext text/x-wiki cnghh9cspwgby6n1sgym7sc65rbxo93 2406991 2406990 2026-04-28T16:24:03Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406991 wikitext text/x-wiki 8a8pwsyhgwffb3b0lj978xpyh0ic38x 2406992 2406991 2026-04-28T16:24:07Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply 2406992 wikitext text/x-wiki c8bfhbol6xy00yisssr92ae36jwxt6w 2406993 2406992 2026-04-28T16:29:09Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.254.0" for 2 host(s) 2406993 wikitext text/x-wiki h1pqsesgkhrhdzhrs2o8xqd0pt1qhoe 2406994 2406993 2026-04-28T16:31:01Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.254.0" completed for 2 hosts 2406994 wikitext text/x-wiki to9pii4splehhzrz3l362b8siuh6y0n 2407003 2406994 2026-04-28T17:15:11Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb1014.eqiad.wmnet 2407003 wikitext text/x-wiki cbkodss1y0t7l94hqho0nzxq2d82fj6 2407004 2407003 2026-04-28T17:15:26Z Stashbot 7414 jasmine@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host rdb1014.eqiad.wmnet 2407004 wikitext text/x-wiki a96u1st5drhtdv655azs2rakph1ju19 2407005 2407004 2026-04-28T17:16:05Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb1014.eqiad.wmnet 2407005 wikitext text/x-wiki eakw40jyir86frvxof0jki64comktnr 2407006 2407005 2026-04-28T17:23:01Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1014.eqiad.wmnet 2407006 wikitext text/x-wiki 568wnx4fhwkrh0hf4tc2l96tc1gahox 2407007 2407006 2026-04-28T17:23:51Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2008.codfw.wmnet 2407007 wikitext text/x-wiki fof3eq05j1ux82vgkws6fg4fj9h8eaj 2407008 2407007 2026-04-28T17:30:35Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2008.codfw.wmnet 2407008 wikitext text/x-wiki 6vkadzrzai5ip5uea6di7j28w9k5pzt 2407009 2407008 2026-04-28T17:31:25Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb1013.eqiad.wmnet 2407009 wikitext text/x-wiki t6jjlu7rkr1qoic7n4qh5bzum9937tq 2407010 2407009 2026-04-28T17:37:55Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1013.eqiad.wmnet 2407010 wikitext text/x-wiki lllvps9ipsm89scvajnx206bk2nc8yk 2407011 2407010 2026-04-28T17:38:07Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2007.codfw.wmnet 2407011 wikitext text/x-wiki jnndib998ac3osc4myleau0j0kwwv6i 2407014 2407011 2026-04-28T17:45:55Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2007.codfw.wmnet 2407014 wikitext text/x-wiki mk5lk9vd4slljg36k74nedradhnhyo8 2407015 2407014 2026-04-28T17:46:56Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb1012.eqiad.wmnet 2407015 wikitext text/x-wiki 6zrjft2y3bsymi7d3dsgo6uucwf5ay1 2407016 2407015 2026-04-28T17:55:23Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1012.eqiad.wmnet 2407016 wikitext text/x-wiki pi0krte526o2k4c1aufvcs2qs0o2dou 2407017 2407016 2026-04-28T17:55:40Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2010.codfw.wmnet 2407017 wikitext text/x-wiki 07coxahv4pu0cejfvwtwi38eq1my9e9 2407018 2407017 2026-04-28T17:56:15Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reimage for host phab2003.codfw.wmnet with OS bookworm 2407018 wikitext text/x-wiki klb6wp1zxmvua9d79psgfalrrixudxu 2407019 2407018 2026-04-28T18:02:10Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2010.codfw.wmnet 2407019 wikitext text/x-wiki qa3laturt141anuwviax16zs66gzhtq 2407020 2407019 2026-04-28T18:02:35Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb1011.eqiad.wmnet 2407020 wikitext text/x-wiki kscafhjycjvnalf87haxuonpln409mz 2407022 2407020 2026-04-28T18:09:12Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1011.eqiad.wmnet 2407022 wikitext text/x-wiki 3ec6fge66mal4j1zbof8f91j2knx0ku 2407023 2407022 2026-04-28T18:11:26Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on phab2003.codfw.wmnet with reason: host reimage 2407023 wikitext text/x-wiki lethrnyv06b14d8v4u15ke4eccx039p 2407025 2407023 2026-04-28T18:15:13Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab2003.codfw.wmnet with reason: host reimage 2407025 wikitext text/x-wiki 2351qxxtgghzy8xoodze7jyimsmnmdu 2407026 2407025 2026-04-28T18:15:35Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2009.codfw.wmnet 2407026 wikitext text/x-wiki 8h3tjm2c8d0iu06m543e133wcldz602 2407027 2407026 2026-04-28T18:21:59Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2009.codfw.wmnet 2407027 wikitext text/x-wiki qsigy39pa9ife8txnouvy3pc7bji8qo 2407028 2407027 2026-04-28T18:27:52Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.decommission for hosts planet1004.eqiad.wmnet 2407028 wikitext text/x-wiki nyjdkklbwyjxij254rec3yl6psnaxvq 2407031 2407028 2026-04-28T18:32:41Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.dns.netbox 2407031 wikitext text/x-wiki ri04iua1xm6skqjiyqwk9iigty2ay0p 2407033 2407031 2026-04-28T18:33:33Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host phab2003.codfw.wmnet with OS bookworm 2407033 wikitext text/x-wiki 2ws3625qq8jh31vfmwdxqegk20iz2kd 2407034 2407033 2026-04-28T18:36:30Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: planet1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin2002" 2407034 wikitext text/x-wiki hepffls6kolsk8a1njec6hbu8hl2db7 2407035 2407034 2026-04-28T18:37:30Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: planet1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin2002" 2407035 wikitext text/x-wiki 80oef4fkai4ejbijcypvrhim36v1ote 2407036 2407035 2026-04-28T18:37:32Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407036 wikitext text/x-wiki 6wh1rdyz4xdyp0g8ygxc2vc348v4pug 2407037 2407036 2026-04-28T18:37:35Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts planet1004.eqiad.wmnet 2407037 wikitext text/x-wiki cl2lr9sbterl12ips38j8sstwcvbhm0 2407041 2407037 2026-04-28T20:04:20Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1278494|testwiki: Enable Article Guidance extension (T417200)]] 2407041 wikitext text/x-wiki 02wjuycyvk953tqv2v03n31nhoqjsu1 2407042 2407041 2026-04-28T20:06:12Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1278494|testwiki: Enable Article Guidance extension (T417200)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407042 wikitext text/x-wiki e9uqu03keblqqybanhob49ey25qjhbm 2407043 2407042 2026-04-28T20:10:57Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2407043 wikitext text/x-wiki ev34vm9ykwm9mv0ufx6utobbevz1ioe 2407044 2407043 2026-04-28T20:14:54Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278494|testwiki: Enable Article Guidance extension (T417200)]] (duration: 10m 34s) 2407044 wikitext text/x-wiki dybnmi0k5pvu3d656degkqaqtc8u2r9 2407046 2407044 2026-04-28T20:22:37Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1277701|cirrus: AB test query suggester variants (T407432)]], [[gerrit:1278498|ExtensionDistributor: mark 1.46 as development (T423262)]] 2407046 wikitext text/x-wiki nrh70kziznerrnibx7ptrqpfm42z0y7 2407047 2407046 2026-04-28T20:24:31Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson, macfan4000: Backport for [[gerrit:1277701|cirrus: AB test query suggester variants (T407432)]], [[gerrit:1278498|ExtensionDistributor: mark 1.46 as development (T423262)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407047 wikitext text/x-wiki cnhbs1mjqqa40rm08bp5dlhsvzpl1wo 2407048 2407047 2026-04-28T20:25:22Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson, macfan4000: Continuing with deployment 2407048 wikitext text/x-wiki tdfennzhbhao55y775rnr18o8h245xt 2407049 2407048 2026-04-28T20:29:15Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277701|cirrus: AB test query suggester variants (T407432)]], [[gerrit:1278498|ExtensionDistributor: mark 1.46 as development (T423262)]] (duration: 06m 35s) 2407049 wikitext text/x-wiki 7p6jen63yxyjzmg7q7d1lyq7ht2r0gl 2407051 2407049 2026-04-28T20:41:58Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1278538|ContentBranchNodeCheck: cope with null actions (T424416)]] 2407051 wikitext text/x-wiki tu9sa2jp3gb0az2z0mi0nnke4d8sj6g 2407052 2407051 2026-04-28T20:43:41Z Stashbot 7414 kemayo@deploy1003: kemayo: Backport for [[gerrit:1278538|ContentBranchNodeCheck: cope with null actions (T424416)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407052 wikitext text/x-wiki jvurf2qtw4tf8dgpxfapf0532x6ykxf 2407053 2407052 2026-04-28T20:44:36Z Stashbot 7414 kemayo@deploy1003: kemayo: Continuing with deployment 2407053 wikitext text/x-wiki jnb9hcevodxscu06b2dayu2yuj4ru1c 2407054 2407053 2026-04-28T20:48:28Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278538|ContentBranchNodeCheck: cope with null actions (T424416)]] (duration: 06m 36s) 2407054 wikitext text/x-wiki kg33n3ph1r9mbtzo7rf8jntnulcw4rj 2407056 2407054 2026-04-28T20:56:53Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1009.eqiad.wmnet with OS trixie 2407056 wikitext text/x-wiki kzwj0q2evdk3sueamgf4pjescefpjo3 2407058 2407056 2026-04-28T21:07:31Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.255.0" for 2 host(s) 2407058 wikitext text/x-wiki bqln6r51c0af4ilw4cb9k60bht1vay1 2407059 2407058 2026-04-28T21:09:22Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.255.0" completed for 2 hosts 2407059 wikitext text/x-wiki ko6ktu2yzxpv0zc693f07vuky3z4v1p 2407060 2407059 2026-04-28T21:16:46Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1009.eqiad.wmnet with reason: host reimage 2407060 wikitext text/x-wiki tw6nmqw4rphe6jmuga815syg0ohjtwa 2407062 2407060 2026-04-28T21:20:50Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1009.eqiad.wmnet with reason: host reimage 2407062 wikitext text/x-wiki 2ouwxoza7xupz61xzixfh6toxpt5li8 2407063 2407062 2026-04-28T21:40:52Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1278513|Provide support for upright in thumbnails for older browsers (T424596)]] 2407063 wikitext text/x-wiki 5161fcx56pv0ks87zlgu0umd3rlsca9 2407064 2407063 2026-04-28T21:42:42Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1278513|Provide support for upright in thumbnails for older browsers (T424596)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407064 wikitext text/x-wiki tn9bqx0e5qmft3ouz6ob8fs501rrje3 2407066 2407064 2026-04-28T21:47:41Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2407066 wikitext text/x-wiki na4i612di8n46jbfkmryfjqn9qb1s3k 2407070 2407066 2026-04-28T21:51:17Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudelastic1009.eqiad.wmnet with OS trixie 2407070 wikitext text/x-wiki tbmisj5ekif7t0lxy84hxohztks7d7c 2407071 2407070 2026-04-28T21:51:35Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278513|Provide support for upright in thumbnails for older browsers (T424596)]] (duration: 10m 43s) 2407071 wikitext text/x-wiki tliwq1e9e760eif8mv8zkds09rpyat7 2407080 2407071 2026-04-29T01:28:58Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.dns.admin DNS admin: depool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified] 2407080 wikitext text/x-wiki 1qgtugm0q1mgftov4sx5x40kzpzur74 2407081 2407080 2026-04-29T01:29:12Z Stashbot 7414 andrew@cumin2002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool ulsfo for service: upload-addrs [reason: no reason specified, no task ID specified] 2407081 wikitext text/x-wiki 9gx0qw7di44gbc2k6ez6aumkch9vrnd 2407084 2407081 2026-04-29T02:28:17Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1008.eqiad.wmnet with OS trixie 2407084 wikitext text/x-wiki 8isdf8u12iqnmx9ztft1v8z2kgm6r3t 2407085 2407084 2026-04-29T02:51:26Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1008.eqiad.wmnet with reason: host reimage 2407085 wikitext text/x-wiki ojdi8b4qx96cn7z1cyvtht2p5k7sb52 2407086 2407085 2026-04-29T02:55:30Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1008.eqiad.wmnet with reason: host reimage 2407086 wikitext text/x-wiki d599c13q11llbojo63ie4pamxlpepk8 2407087 2407086 2026-04-29T03:16:23Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudelastic1008.eqiad.wmnet with OS trixie 2407087 wikitext text/x-wiki jpnl2806egkdb0hhjplvqi6qn9mvz6q 2407089 2407087 2026-04-29T05:07:06Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s5 T424550 2407089 wikitext text/x-wiki ag951fawaz3751gony3y9jy71flx30i 2407090 2407089 2026-04-29T05:07:22Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1210 with weight 0 T424550', diff saved to https://phabricator.wikimedia.org/P91814 and previous config saved to /var/cache/conftool/dbconfig/20260429-050718-marostegui.json 2407090 wikitext text/x-wiki kg2w4hbfhc1x3l3rsbqep8mf1lm4o48 2407091 2407090 2026-04-29T05:08:12Z Stashbot 7414 marostegui: Starting s5 eqiad failover from db1230 to db1210 - T424550 2407091 wikitext text/x-wiki 6k7d1fx69xxbcpwd1dag757fc37smj3 2407093 2407091 2026-04-29T05:10:36Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set s5 eqiad as read-only for maintenance - T424550', diff saved to https://phabricator.wikimedia.org/P91815 and previous config saved to /var/cache/conftool/dbconfig/20260429-051032-marostegui.json 2407093 wikitext text/x-wiki nr7wrk3anj0733wbhn5lwmn2k9bw9w4 2407094 2407093 2026-04-29T05:10:59Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1210 to s5 primary and set section read-write T424550', diff saved to https://phabricator.wikimedia.org/P91816 and previous config saved to /var/cache/conftool/dbconfig/20260429-051054-marostegui.json 2407094 wikitext text/x-wiki sy78lajnpnmlass83inixbhpcx6tz0y 2407095 2407094 2026-04-29T05:11:39Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2407095 wikitext text/x-wiki 4wzyli3myfu7uag7wfn1hqa5fvg6xnq 2407096 2407095 2026-04-29T05:12:47Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1230 T424550', diff saved to https://phabricator.wikimedia.org/P91817 and previous config saved to /var/cache/conftool/dbconfig/20260429-051244-marostegui.json 2407096 wikitext text/x-wiki 7e3tsmpue0m4itmkph9qnw0gozx71pr 2407097 2407096 2026-04-29T05:13:07Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2407097 wikitext text/x-wiki n3wijhrnxy1rd2sljrs2p9gvob1kum9 2407098 2407097 2026-04-29T05:18:56Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2407098 wikitext text/x-wiki tjw184zw3vgghbl640ty30r3xzc0743 2407099 2407098 2026-04-29T05:20:32Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2407099 wikitext text/x-wiki tj0yw88t2z1lg2x72jau4wph2s7m4pj 2407100 2407099 2026-04-29T05:20:49Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1230.eqiad.wmnet with reason: Reimage to Trixie 2407100 wikitext text/x-wiki 1o6z93vxy54r7wlj5zibw6z0ls4luih 2407101 2407100 2026-04-29T05:20:54Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1230: Reimage to Trixie 2407101 wikitext text/x-wiki q9uydqguqgm2eu0um5ifr5ilx2qkv1r 2407102 2407101 2026-04-29T05:21:01Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1230: Reimage to Trixie 2407102 wikitext text/x-wiki m14a13pnfq6xqrxt0za2yuu6af8ywv1 2407103 2407102 2026-04-29T05:22:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1230.eqiad.wmnet with OS trixie 2407103 wikitext text/x-wiki qfb59ad8nof1zquf5dgpauscic3zv8z 2407104 2407103 2026-04-29T05:31:07Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2225.codfw.wmnet with reason: Reimage to Trixie 2407104 wikitext text/x-wiki 62eypibd62y71cp0s9at936jx70w5fx 2407105 2407104 2026-04-29T05:31:13Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2225: Reimage to Trixie 2407105 wikitext text/x-wiki 45sjdzth2uem7fvx9jy0rhy4r267u6j 2407106 2407105 2026-04-29T05:31:31Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2225: Reimage to Trixie 2407106 wikitext text/x-wiki jn43lz49tl7g0z7job3mjcdzjutf5w2 2407107 2407106 2026-04-29T05:31:36Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1254.eqiad.wmnet with reason: Reimage to Trixie 2407107 wikitext text/x-wiki svzfeyz81epq0qa3e1bhblg0ymsew2i 2407108 2407107 2026-04-29T05:31:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1254: Reimage to Trixie 2407108 wikitext text/x-wiki klj5slpna543ll7hq4iie4xrhokkm3j 2407109 2407108 2026-04-29T05:32:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1254: Reimage to Trixie 2407109 wikitext text/x-wiki 1hayhgqoo2jytt6cqbyjy73fgq2bkkc 2407110 2407109 2026-04-29T05:32:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2225.codfw.wmnet with OS trixie 2407110 wikitext text/x-wiki jw94u8l2e4o55pquzyo9757u947c187 2407111 2407110 2026-04-29T05:35:03Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1254.eqiad.wmnet with OS trixie 2407111 wikitext text/x-wiki kaaoukfpiruiv2zwbq6ik1o7e0026kk 2407112 2407111 2026-04-29T05:37:51Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1230.eqiad.wmnet with reason: host reimage 2407112 wikitext text/x-wiki e0vsnt2vc6zbv24wxpz9gzu606e0hm2 2407116 2407112 2026-04-29T05:44:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1230.eqiad.wmnet with reason: host reimage 2407116 wikitext text/x-wiki aemclk1gf6gg8beyc6vuh78e91nbxo0 2407119 2407116 2026-04-29T05:49:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2225.codfw.wmnet with reason: host reimage 2407119 wikitext text/x-wiki e4bnkinzxbat5zvqprg145c35i59zu2 2407120 2407119 2026-04-29T05:50:14Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1254.eqiad.wmnet with reason: host reimage 2407120 wikitext text/x-wiki 0pljgqxpvj0qjrju37wvj6p0984iq3v 2407121 2407120 2026-04-29T05:55:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2225.codfw.wmnet with reason: host reimage 2407121 wikitext text/x-wiki iyiitgi0ltosij7jbk4wcb0ayssdnes 2407122 2407121 2026-04-29T05:59:45Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1254.eqiad.wmnet with reason: host reimage 2407122 wikitext text/x-wiki e45v8jmti05195l992eb0j3qyrtufqy 2407123 2407122 2026-04-29T06:06:22Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1230.eqiad.wmnet with OS trixie 2407123 wikitext text/x-wiki s6j3uilqy806qhz7uztfc7fte91kkyn 2407124 2407123 2026-04-29T06:08:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1230: after reimage to trixie 2407124 wikitext text/x-wiki rff60untf5vr9jafhs9oxspzux0b7rj 2407125 2407124 2026-04-29T06:16:59Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1198.eqiad.wmnet with reason: Reimage to Trixie 2407125 wikitext text/x-wiki j8yf75dge1m68xe0r6m044f6suorfne 2407126 2407125 2026-04-29T06:17:04Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1198: Reimage to Trixie 2407126 wikitext text/x-wiki dc6e2bpb3eiwgcsbx1n1ui99rk9pgx0 2407127 2407126 2026-04-29T06:17:14Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2227.codfw.wmnet with reason: Reimage to Trixie 2407127 wikitext text/x-wiki evlbu8g8q9ir6rpa8u3q1k0x4c4a6s4 2407128 2407127 2026-04-29T06:17:20Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2227: Reimage to Trixie 2407128 wikitext text/x-wiki 0zub7kxhslgj1pyfruz2iuc7lflpa01 2407129 2407128 2026-04-29T06:17:38Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2227: Reimage to Trixie 2407129 wikitext text/x-wiki rqkgqad7va9wiomacm4ex9wunbsaygt 2407130 2407129 2026-04-29T06:18:02Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1198: Reimage to Trixie 2407130 wikitext text/x-wiki 722ju00ifpg6qc5gxz06year271rnkz 2407131 2407130 2026-04-29T06:18:18Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2225.codfw.wmnet with OS trixie 2407131 wikitext text/x-wiki ttg3pe3ckyk3p41ffzvx5norzuau50b 2407132 2407131 2026-04-29T06:19:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1198.eqiad.wmnet with OS trixie 2407132 wikitext text/x-wiki o1sui9er8jlzuou92rll3dugyoduq9w 2407133 2407132 2026-04-29T06:19:47Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2227.codfw.wmnet with OS trixie 2407133 wikitext text/x-wiki rjf87mf7vlgnnlokd447zv9eki7asts 2407134 2407133 2026-04-29T06:20:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2225: after reimage to trixie 2407134 wikitext text/x-wiki lq7uudv3jm6fxzsf8ektyrxe6lmkk34 2407135 2407134 2026-04-29T06:22:14Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1254.eqiad.wmnet with OS trixie 2407135 wikitext text/x-wiki itx2zlx1ibqkkwzqb3909nmzsddsqgl 2407136 2407135 2026-04-29T06:25:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1254: after reimage to trixie 2407136 wikitext text/x-wiki 8vjl7ea8x2gmy0n8mztky15vq0wlm5q 2407140 2407136 2026-04-29T06:33:52Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage 2407140 wikitext text/x-wiki 18uzruay11d8csr444og702ouqxuxwf 2407143 2407140 2026-04-29T06:37:12Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2227.codfw.wmnet with reason: host reimage 2407143 wikitext text/x-wiki i284kbsax8v6z5xrwxc1pr6xaci01oo 2407144 2407143 2026-04-29T06:37:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc2011.codfw.wmnet 2407144 wikitext text/x-wiki bcaiqy1vlgc9baalnyjh9ez323ul20m 2407145 2407144 2026-04-29T06:39:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage 2407145 wikitext text/x-wiki 5yk8mj33lei620tgbvhypoxkh555819 2407147 2407145 2026-04-29T06:42:48Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2407147 wikitext text/x-wiki 5nkstoh49qee1tco4dqbrute3i8iqif 2407148 2407147 2026-04-29T06:43:21Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2227.codfw.wmnet with reason: host reimage 2407148 wikitext text/x-wiki 7xzpp49nyfs5bosdg2tpy47nadgxvww 2407149 2407148 2026-04-29T06:53:40Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc2011.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2407149 wikitext text/x-wiki bp7p17w294wwpp1zcci2enarox77cd9 2407150 2407149 2026-04-29T06:53:57Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1230: after reimage to trixie 2407150 wikitext text/x-wiki skf4qb6njetw52bz7y4jwa9joqczrm0 2407151 2407150 2026-04-29T06:54:50Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc2011.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2407151 wikitext text/x-wiki 1ptifvvzdixgkzeekn0cqw97q0c8v7j 2407152 2407151 2026-04-29T06:54:53Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407152 wikitext text/x-wiki ixwpcsbsbsktbuuxwjuof0efmxzd9sf 2407153 2407152 2026-04-29T06:54:56Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc2011.codfw.wmnet 2407153 wikitext text/x-wiki h0ohdm4t4hn7t6c339abib50ban0hq2 2407154 2407153 2026-04-29T07:01:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1198.eqiad.wmnet with OS trixie 2407154 wikitext text/x-wiki ifkz4z4910o0jzhphga7522b5runsaq 2407155 2407154 2026-04-29T07:06:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2227.codfw.wmnet with OS trixie 2407155 wikitext text/x-wiki t14es6n4tc1y71u0w17jy9jzcn95ygj 2407156 2407155 2026-04-29T07:06:24Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2225: after reimage to trixie 2407156 wikitext text/x-wiki jruv5jrmjbg0wn6fksbp896osd033ol 2407157 2407156 2026-04-29T07:06:55Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1198: after reimage to trixie 2407157 wikitext text/x-wiki lqton2rdajui7sg3licnrqn1ptiv84u 2407158 2407157 2026-04-29T07:11:01Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1254: after reimage to trixie 2407158 wikitext text/x-wiki bzpjpfised9zuk3baglh8qmcgn5qdg7 2407159 2407158 2026-04-29T07:11:20Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2227: after reimage to trixie 2407159 wikitext text/x-wiki 7tz8yg2wuqwjishe4h8rsyr0weobjdk 2407160 2407159 2026-04-29T07:17:23Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1278509|Completion: fix near match field name (T417648)]], [[gerrit:1269464|search: add alt. completion indices to test keyword tokenizer (1/2) (T420427)]] 2407160 wikitext text/x-wiki jw58pw8v6mrny09e4cjcchzj6lty6to 2407161 2407160 2026-04-29T07:19:23Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1278509|Completion: fix near match field name (T417648)]], [[gerrit:1269464|search: add alt. completion indices to test keyword tokenizer (1/2) (T420427)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407161 wikitext text/x-wiki erusdl1igislpp5py42o78hzs1kvdrj 2407162 2407161 2026-04-29T07:20:35Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2407162 wikitext text/x-wiki 54qxqjecg8bbrsz26ruqbmkkbq9xh76 2407163 2407162 2026-04-29T07:23:00Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet 2407163 wikitext text/x-wiki gk5b7up4kxb8kyf9azmxsaex3c4kjed 2407164 2407163 2026-04-29T07:24:30Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278509|Completion: fix near match field name (T417648)]], [[gerrit:1269464|search: add alt. completion indices to test keyword tokenizer (1/2) (T420427)]] (duration: 07m 07s) 2407164 wikitext text/x-wiki k8smfesexvbt6ds362w31p68eol7qpe 2407165 2407164 2026-04-29T07:25:37Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1039.eqiad.wmnet 2407165 wikitext text/x-wiki biacap9yr1rfd51in8vvmvs4skyepum 2407166 2407165 2026-04-29T07:30:15Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet 2407166 wikitext text/x-wiki g4tgr431iqfjrh5g8cit1f5agy4ppvx 2407167 2407166 2026-04-29T07:30:18Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet 2407167 wikitext text/x-wiki jdtelrq22saud1b3ft14ahkfcymwtsc 2407168 2407167 2026-04-29T07:31:08Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2407168 wikitext text/x-wiki hvozvd92x0rg6mlvdqsd3x5uiqncids 2407169 2407168 2026-04-29T07:32:35Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2189.codfw.wmnet with reason: Reimage to Trixie 2407169 wikitext text/x-wiki 9kjz76gav9rfhmzhru0kfh79axtjvx5 2407170 2407169 2026-04-29T07:32:40Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Reimage to Trixie 2407170 wikitext text/x-wiki a2zwpceyvc7o95s6s6hikh59zw8lfm3 2407171 2407170 2026-04-29T07:32:43Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1233.eqiad.wmnet with reason: Reimage to Trixie 2407171 wikitext text/x-wiki r91zeppsn7nm5mwusyt6rb645o43mlx 2407172 2407171 2026-04-29T07:32:48Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1233: Reimage to Trixie 2407172 wikitext text/x-wiki a6fodw9k1seytzhyb45jjmu87kmi7yw 2407173 2407172 2026-04-29T07:33:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1233: Reimage to Trixie 2407173 wikitext text/x-wiki 82o8s40r84k992asgaun4yg3irs6jcx 2407174 2407173 2026-04-29T07:33:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: Reimage to Trixie 2407174 wikitext text/x-wiki j44fndv2uicr5u7aufr9bqh5m8yl31i 2407175 2407174 2026-04-29T07:34:18Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2189.codfw.wmnet with OS trixie 2407175 wikitext text/x-wiki 2525w7jmz9ggsexn47ho69is9j8ihmb 2407176 2407175 2026-04-29T07:34:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1233.eqiad.wmnet with OS trixie 2407176 wikitext text/x-wiki 9o1orwx7rvmm9x7ciq7qpllp2tdjhqj 2407177 2407176 2026-04-29T07:38:29Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1007.eqiad.wmnet with OS trixie 2407177 wikitext text/x-wiki 45qhhgt64g3ef3kkq89oq7ho3qvfgn3 2407178 2407177 2026-04-29T07:39:31Z Stashbot 7414 ryankemper: T422860 [cloudelastic] Restarted opensearch services on `cloudelastic1011` and `cloudelastic1012` (needed to pick up missing opensearch plugins, which have already been fixed in puppet) (note: this was done ~2h ago; logged in wrong channel) 2407178 wikitext text/x-wiki e8pinhgs0tpxj3cx6om8tscx90bd0j7 2407180 2407178 2026-04-29T07:52:07Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1233.eqiad.wmnet with reason: host reimage 2407180 wikitext text/x-wiki d2dhu0a0z1i6utv28g6wsdzfoxu8cmp 2407181 2407180 2026-04-29T07:52:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1198: after reimage to trixie 2407181 wikitext text/x-wiki n2v6p5903y0r4hp125m7wv82r1emytg 2407182 2407181 2026-04-29T07:52:45Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2407182 wikitext text/x-wiki mf9sg4v9fue36o9gccfks60kerbzqf8 2407183 2407182 2026-04-29T07:53:23Z Stashbot 7414 a-pizzata@deploy1003: Started deploy [analytics/refinery@d6a17a0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d6a17a0a] 2407183 wikitext text/x-wiki 22j2u5ya3jdvfnzja0ju5ob2al78ahp 2407184 2407183 2026-04-29T07:53:51Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2189.codfw.wmnet with reason: host reimage 2407184 wikitext text/x-wiki l4up6jfxh27tsa9k7thmuiuf3jymumv 2407185 2407184 2026-04-29T07:55:20Z Stashbot 7414 a-pizzata@deploy1003: Finished deploy [analytics/refinery@d6a17a0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d6a17a0a] (duration: 01m 57s) 2407185 wikitext text/x-wiki 0en94d4sevpdn4ubhmcdkjkkeu4nlt2 2407186 2407185 2026-04-29T07:55:33Z Stashbot 7414 a-pizzata@deploy1003: Started deploy [analytics/refinery@d6a17a0]: Regular analytics weekly train [analytics/refinery@d6a17a0a] 2407186 wikitext text/x-wiki dnqrofy2jbgldmmsl5ojr4dovbd7pyn 2407187 2407186 2026-04-29T07:55:57Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1175.eqiad.wmnet with reason: Reimage to Trixie 2407187 wikitext text/x-wiki dsmtjk27b9uex3hxz77krwckxfdtpfy 2407188 2407187 2026-04-29T07:56:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1175: Reimage to Trixie 2407188 wikitext text/x-wiki l591bejypzuauuvnbeedg7rnhr2b22l 2407189 2407188 2026-04-29T07:56:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1175: Reimage to Trixie 2407189 wikitext text/x-wiki modos5ne1zh1ipdpvav6kjxox9ai8ae 2407190 2407189 2026-04-29T07:56:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2227: after reimage to trixie 2407190 wikitext text/x-wiki 2j41o8zuv5accjchm1vajduow848hoo 2407191 2407190 2026-04-29T07:57:24Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2194.codfw.wmnet with reason: Reimage to Trixie 2407191 wikitext text/x-wiki mkt5d4fdfada7sr4tu2l9nz1xx406hz 2407192 2407191 2026-04-29T07:57:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2194: Reimage to Trixie 2407192 wikitext text/x-wiki h7qefq7o832cl7i0l3zpx7swibak01o 2407193 2407192 2026-04-29T07:57:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2194: Reimage to Trixie 2407193 wikitext text/x-wiki 5khdkjinntagzb27rjdnr0dqv9e1vxk 2407194 2407193 2026-04-29T07:58:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1175.eqiad.wmnet with OS trixie 2407194 wikitext text/x-wiki jmbxz3z2q5py4w0shv4akajujg5p1ti 2407195 2407194 2026-04-29T07:59:16Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1233.eqiad.wmnet with reason: host reimage 2407195 wikitext text/x-wiki 9m9ud69xyriwer14vxlv4rolt2gck6b 2407196 2407195 2026-04-29T07:59:25Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2194.codfw.wmnet with OS trixie 2407196 wikitext text/x-wiki ovewwmtyo41z8w9tgi7ojewl9pizuav 2407197 2407196 2026-04-29T07:59:45Z Stashbot 7414 a-pizzata@deploy1003: Finished deploy [analytics/refinery@d6a17a0]: Regular analytics weekly train [analytics/refinery@d6a17a0a] (duration: 04m 12s) 2407197 wikitext text/x-wiki oi1l3p0ahnckh7vdafzqyuj9mgm9zyf 2407198 2407197 2026-04-29T08:02:24Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2189.codfw.wmnet with reason: host reimage 2407198 wikitext text/x-wiki jrymui410kifjcvyvtysemo8lvyhgpe 2407199 2407198 2026-04-29T08:06:52Z Stashbot 7414 a-pizzata@deploy1003: Started deploy [analytics/refinery@d6a17a0] (thin): Regular analytics weekly train THIN [analytics/refinery@d6a17a0a] 2407199 wikitext text/x-wiki o6bcxdsw0r1f44pbirgocliyllood7g 2407200 2407199 2026-04-29T08:08:46Z Stashbot 7414 a-pizzata@deploy1003: Finished deploy [analytics/refinery@d6a17a0] (thin): Regular analytics weekly train THIN [analytics/refinery@d6a17a0a] (duration: 01m 54s) 2407200 wikitext text/x-wiki diaszufsnf5x0vzc44h64krj06tdpw1 2407201 2407200 2026-04-29T08:09:07Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2407201 wikitext text/x-wiki fbffelaf1jjxm3g2my12usrh94rj2pb 2407202 2407201 2026-04-29T08:14:12Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1175.eqiad.wmnet with reason: host reimage 2407202 wikitext text/x-wiki 4yhp1zjz9mc7naodsihrrgyjrffafwi 2407203 2407202 2026-04-29T08:16:53Z Stashbot 7414 Emperor: disable puppet in apus/codfw for TLS key rollover T424674 2407203 wikitext text/x-wiki f15vgpnzmcgylf60udn15ujt9wlgiib 2407204 2407203 2026-04-29T08:18:29Z Stashbot 7414 Emperor: re-enable puppet in apus/codfw for TLS key rollover T424674 (no change, incident took over) 2407204 wikitext text/x-wiki 9unwmrr5pyynvja8ru2l13x701ikfa4 2407205 2407204 2026-04-29T08:18:55Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1175.eqiad.wmnet with reason: host reimage 2407205 wikitext text/x-wiki d4p8434dlrw1dxobo276ac0lrw2liy4 2407206 2407205 2026-04-29T08:21:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2194.codfw.wmnet with reason: host reimage 2407206 wikitext text/x-wiki pv5cr2iou4mxdppmazpsa31dv4zn95w 2407207 2407206 2026-04-29T08:21:31Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1233.eqiad.wmnet with OS trixie 2407207 wikitext text/x-wiki hpn0yeof6veo0goa2ow5q1m8w5p24l2 2407208 2407207 2026-04-29T08:24:43Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2189.codfw.wmnet with OS trixie 2407208 wikitext text/x-wiki bt5anvgvdm1rnpgdg96jigmf3x84zfl 2407209 2407208 2026-04-29T08:24:47Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2194.codfw.wmnet with reason: host reimage 2407209 wikitext text/x-wiki 5184o5z5uwzsd8b2fw3h6d5s6o5t8c9 2407210 2407209 2026-04-29T08:24:50Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1233: after reimage to trixie 2407210 wikitext text/x-wiki a9wx6btk7wulsmt8scztfqyzjuw39db 2407211 2407210 2026-04-29T08:28:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2189: after reimage to trixie 2407211 wikitext text/x-wiki tpdat52wbfgxnkekz7lmc4mdpscpw2x 2407212 2407211 2026-04-29T08:29:57Z Stashbot 7414 elukey@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: sync 2407212 wikitext text/x-wiki d30qlttvp5b69jh46u6po6dzx474l68 2407213 2407212 2026-04-29T08:30:01Z Stashbot 7414 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: sync 2407213 wikitext text/x-wiki 1yh5oa2oqh5926vqddjzdj0kmjn1z3h 2407214 2407213 2026-04-29T08:37:35Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki Wikimedia_Apps/Team/Android/TriviaGame 'Wikimedia Apps/Team/Android/Which' came 'first? Game' 'Martin Urbanec (WMF)' '--reason=per [[:phab:T423845]]' # T423845 2407214 wikitext text/x-wiki apd7go94tfz5s246kz3ln6jss77pl7f 2407215 2407214 2026-04-29T08:38:05Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki Wikimedia_Apps/Team/Android/TriviaGame 'Wikimedia Apps/Team/Android/"Which came first?" Game' 'Martin Urbanec (WMF)' '--reason=per [[:phab:T423845]]' # T423845 2407215 wikitext text/x-wiki 0jr2baa3qnykrv18jghd7vboobu3z57 2407216 2407215 2026-04-29T08:38:56Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki Wikimedia_Apps/Team/Android/TriviaGame 'Wikimedia Apps/Team/Android/"Which came first?" Game' 'Martin Urbanec (WMF)' '--reason=per [[:phab:T423845]]' # T423845 2407216 wikitext text/x-wiki 66j2grlc5iv0jpwc62cq9b166lmitdv 2407218 2407216 2026-04-29T08:40:51Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1175.eqiad.wmnet with OS trixie 2407218 wikitext text/x-wiki egg1j0xftfn9j69hrhe4ggjbfmuqxe1 2407219 2407218 2026-04-29T08:42:18Z Stashbot 7414 ryankemper@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudelastic1007.eqiad.wmnet with OS trixie 2407219 wikitext text/x-wiki 75hc5jm8862lzqxwub3tknc8biv1qts 2407220 2407219 2026-04-29T08:45:14Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1175: after reimage to trixie 2407220 wikitext text/x-wiki rygmcidf8x6itij6psbmslln7ovb74u 2407221 2407220 2026-04-29T08:49:01Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2194.codfw.wmnet with OS trixie 2407221 wikitext text/x-wiki cowac2co1vxwl3up9wjronr2rkb88xu 2407222 2407221 2026-04-29T08:51:18Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance 2407222 wikitext text/x-wiki ptl27ge2kpsvyrk0ciaxqhu686m1798 2407223 2407222 2026-04-29T08:51:51Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2407223 wikitext text/x-wiki 7ioe4q8xf1zcksivkcmkrq9upph54k2 2407224 2407223 2026-04-29T08:54:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2194: after reimage to trixie 2407224 wikitext text/x-wiki pk8twl34xmwd35mdbmxfmcwqi9j0uzm 2407225 2407224 2026-04-29T08:56:48Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance 2407225 wikitext text/x-wiki 0lkohf1dhfu0vy4v0bn72lhj7p467np 2407226 2407225 2026-04-29T08:56:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1174 (T419961)', diff saved to https://phabricator.wikimedia.org/P91854 and previous config saved to /var/cache/conftool/dbconfig/20260429-085654-fceratto.json 2407226 wikitext text/x-wiki 0pb3h71kkyfdrecrdaja5ow6ekezjw6 2407228 2407226 2026-04-29T09:01:09Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm 2407228 wikitext text/x-wiki 1oagnhm19uyqk3ctb3ljm5y1sf4lk3b 2407229 2407228 2026-04-29T09:05:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T419961)', diff saved to https://phabricator.wikimedia.org/P91857 and previous config saved to /var/cache/conftool/dbconfig/20260429-090534-fceratto.json 2407229 wikitext text/x-wiki sq5n39rgqb3lblmvc1skfwfs89ev5f8 2407230 2407229 2026-04-29T09:10:13Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1233: after reimage to trixie 2407230 wikitext text/x-wiki orguoron6v333gcx6bu0144ktqfs57a 2407231 2407230 2026-04-29T09:13:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: after reimage to trixie 2407231 wikitext text/x-wiki aw8nicowciel1bj70z3l0ei452fcxj5 2407232 2407231 2026-04-29T09:15:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P91862 and previous config saved to /var/cache/conftool/dbconfig/20260429-091542-fceratto.json 2407232 wikitext text/x-wiki 7pgjx5k3kpodclldf78ll1mkw6zigo0 2407233 2407232 2026-04-29T09:19:47Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2175.codfw.wmnet with reason: Reimage to Trixie 2407233 wikitext text/x-wiki 5pn0ruytkao5kmgh3uvc9zoxah5lj4j 2407234 2407233 2026-04-29T09:19:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2175: Reimage to Trixie 2407234 wikitext text/x-wiki 03dgvct10ywm8piehvx90riic0v8q2d 2407235 2407234 2026-04-29T09:20:12Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1229.eqiad.wmnet with reason: Reimage to Trixie 2407235 wikitext text/x-wiki fgbdgar01c3oi2t4kgk6doxt47bo86d 2407236 2407235 2026-04-29T09:20:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1229: Reimage to Trixie 2407236 wikitext text/x-wiki fjou4278lvympguigxlhr73o48ohjxm 2407237 2407236 2026-04-29T09:20:21Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2175: Reimage to Trixie 2407237 wikitext text/x-wiki g1sla1zixsxqbuchc1mza29f0ogfb4k 2407238 2407237 2026-04-29T09:21:07Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1229: Reimage to Trixie 2407238 wikitext text/x-wiki jn5cz0qzgxxne745nokip6kvx84fo88 2407239 2407238 2026-04-29T09:21:55Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2175.codfw.wmnet with OS trixie 2407239 wikitext text/x-wiki 7b65a69z8im3pmx1uy5oorpunyi4a9s 2407240 2407239 2026-04-29T09:22:12Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1229.eqiad.wmnet with OS trixie 2407240 wikitext text/x-wiki 64w0o5pyxn6l075a1dooaa360h7uqrl 2407241 2407240 2026-04-29T09:25:43Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool ulsfo [reason: primary network link stable, no task ID specified] 2407241 wikitext text/x-wiki i44dk8bg9r1j573hsnpvol2vkfid7tz 2407242 2407241 2026-04-29T09:25:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P91866 and previous config saved to /var/cache/conftool/dbconfig/20260429-092551-fceratto.json 2407242 wikitext text/x-wiki pyr0p1iipw6j2u9h1u7i6i56ko21tdb 2407243 2407242 2026-04-29T09:26:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool ulsfo [reason: primary network link stable, no task ID specified] 2407243 wikitext text/x-wiki osach7tfqkxl4ob758s8kvzt4vwys2w 2407245 2407243 2026-04-29T09:30:39Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1175: after reimage to trixie 2407245 wikitext text/x-wiki 1cqzzlenovbefp0gctdclklawcq48mt 2407246 2407245 2026-04-29T09:34:12Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1166.eqiad.wmnet with reason: Reimage to Trixie 2407246 wikitext text/x-wiki resawub82yyc6u29voq6kibpizh28m9 2407247 2407246 2026-04-29T09:34:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1166: Reimage to Trixie 2407247 wikitext text/x-wiki 4erojqptzagmvglgfq2gnxj0ut9gb9c 2407248 2407247 2026-04-29T09:34:45Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1166: Reimage to Trixie 2407248 wikitext text/x-wiki 2vyceuivqytokyk9qjnh5ja2cwfvku7 2407249 2407248 2026-04-29T09:35:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1166.eqiad.wmnet with OS trixie 2407249 wikitext text/x-wiki soi1732tyk8ojkvbkfj6sdl8osndgzu 2407250 2407249 2026-04-29T09:36:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T419961)', diff saved to https://phabricator.wikimedia.org/P91869 and previous config saved to /var/cache/conftool/dbconfig/20260429-093557-fceratto.json 2407250 wikitext text/x-wiki 7i1itzhol2qjtzt5lbujb6xjpbkamr8 2407251 2407250 2026-04-29T09:36:19Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance 2407251 wikitext text/x-wiki kdjbuuq7d3p4jxe6z2xvkpbfz74k0qx 2407252 2407251 2026-04-29T09:36:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1191 (T419961)', diff saved to https://phabricator.wikimedia.org/P91870 and previous config saved to /var/cache/conftool/dbconfig/20260429-093624-fceratto.json 2407252 wikitext text/x-wiki 6fy9idjxmojgh0tnceh5xiw9yd04gih 2407253 2407252 2026-04-29T09:37:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1229.eqiad.wmnet with reason: host reimage 2407253 wikitext text/x-wiki mf2ik42vy8rwhnkkfv4zbufxoiobd22 2407254 2407253 2026-04-29T09:39:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2194: after reimage to trixie 2407254 wikitext text/x-wiki 7ilzq1h8y4xyfom0armqk3tf1msif81 2407255 2407254 2026-04-29T09:40:34Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Test noop upgrade on the replica 2407255 wikitext text/x-wiki hv4956c81maif2p1g1bv8wdfe73ornq 2407256 2407255 2026-04-29T09:41:17Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2190.codfw.wmnet with reason: Reimage to Trixie 2407256 wikitext text/x-wiki h286spaiewdxg3d48r2qandxhlkrex5 2407257 2407256 2026-04-29T09:41:23Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2190: Reimage to Trixie 2407257 wikitext text/x-wiki dxrhnyakwirk4m2v3vzfuk0w3mnd8ae 2407258 2407257 2026-04-29T09:41:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2190: Reimage to Trixie 2407258 wikitext text/x-wiki e4eyk2ay22y73mq7vczeo137o046pq4 2407259 2407258 2026-04-29T09:42:16Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1229.eqiad.wmnet with reason: host reimage 2407259 wikitext text/x-wiki auyaiuf6l8v9sn2xvq0f0qjmp9mt2pu 2407260 2407259 2026-04-29T09:43:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T419961)', diff saved to https://phabricator.wikimedia.org/P91873 and previous config saved to /var/cache/conftool/dbconfig/20260429-094333-fceratto.json 2407260 wikitext text/x-wiki j7pmhakmmuwug8w3mokurqz351xrhwn 2407262 2407260 2026-04-29T09:44:07Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Test noop upgrade on the replica 2407262 wikitext text/x-wiki 2mxxd7bg45ydlw2nifm0i1ciyihr5g4 2407263 2407262 2026-04-29T09:44:35Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2190.codfw.wmnet with reason: Reimage to Trixie 2407263 wikitext text/x-wiki d7x1ph0n2rtrg258dx686t1x6gj6h5g 2407264 2407263 2026-04-29T09:44:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2190: Reimage to Trixie 2407264 wikitext text/x-wiki hry4ymv9w2m8vyklrgt9b52hmu6usk0 2407265 2407264 2026-04-29T09:44:48Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2190: Reimage to Trixie 2407265 wikitext text/x-wiki q6k382srl95y4e2g5yfnlm5lxy1kp7u 2407266 2407265 2026-04-29T09:45:54Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2190.codfw.wmnet with OS trixie 2407266 wikitext text/x-wiki 6v4kf931by6ae6ef8ywqqjyjg6me1rt 2407267 2407266 2026-04-29T09:51:24Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1166.eqiad.wmnet with reason: host reimage 2407267 wikitext text/x-wiki p2reeicdqoxztib3v0sj9tgwak099x4 2407268 2407267 2026-04-29T09:52:53Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2407268 wikitext text/x-wiki 7ciwko1g94fta3vqdi0zx1vdx5mcb9t 2407269 2407268 2026-04-29T09:53:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P91874 and previous config saved to /var/cache/conftool/dbconfig/20260429-095341-fceratto.json 2407269 wikitext text/x-wiki 30y5vqnearkiuvrzxny7mlbjf64ky2t 2407270 2407269 2026-04-29T09:53:57Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2407270 wikitext text/x-wiki 8zggzustisz2r7i549oz3vi8e2s2yja 2407271 2407270 2026-04-29T09:54:33Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2175.codfw.wmnet with OS trixie 2407271 wikitext text/x-wiki 5e04m39ng98qpz3tmluu7pb8kbei86p 2407272 2407271 2026-04-29T09:57:33Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1166.eqiad.wmnet with reason: host reimage 2407272 wikitext text/x-wiki ry2oackezr6p53kesqleleh6at7lioi 2407273 2407272 2026-04-29T10:00:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2175.codfw.wmnet with OS trixie 2407273 wikitext text/x-wiki bntkwu4uhscjzdatk6g849ob3q8or20 2407274 2407273 2026-04-29T10:03:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P91875 and previous config saved to /var/cache/conftool/dbconfig/20260429-100349-fceratto.json 2407274 wikitext text/x-wiki dibtov4utg0injxcydx053j9c0sh6rq 2407275 2407274 2026-04-29T10:04:24Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2190.codfw.wmnet with reason: host reimage 2407275 wikitext text/x-wiki 25jtibqfr7zm0d8cktkc949nywt8dr2 2407276 2407275 2026-04-29T10:04:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1229.eqiad.wmnet with OS trixie 2407276 wikitext text/x-wiki 5tezm5qhc4b8xvo07jw32vqy10s9xwz 2407277 2407276 2026-04-29T10:07:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1229: after reimage to trixie 2407277 wikitext text/x-wiki s2d6v91wai9tpubpu6imyqybeen2n7p 2407278 2407277 2026-04-29T10:08:38Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2407278 wikitext text/x-wiki d7964alwmrv6mmh2wkdwg5c04uxzw9y 2407279 2407278 2026-04-29T10:09:15Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2407279 wikitext text/x-wiki 6enwbzlm27cv95t2bdwhfod7yi1lfi1 2407280 2407279 2026-04-29T10:09:20Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2190.codfw.wmnet with reason: host reimage 2407280 wikitext text/x-wiki 1acufo0361xd64p0cqpbcniuskbn3ap 2407281 2407280 2026-04-29T10:12:06Z Stashbot 7414 Emperor: disable puppet in apus/codfw rgws for TLS key rollover T424674 2407281 wikitext text/x-wiki sw1020l1qe10lxalcrutgihfjc1g81n 2407282 2407281 2026-04-29T10:12:58Z Stashbot 7414 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti5005.eqsin.wmnet with OS bookworm 2407282 wikitext text/x-wiki bd1ubdbukd3ugbdhhx4u1gbqwialf1d 2407283 2407282 2026-04-29T10:13:35Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5005.eqsin.wmnet with OS bookworm 2407283 wikitext text/x-wiki n5gbaxvm199zrzo6qicxmw54zhhewlb 2407284 2407283 2026-04-29T10:14:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T419961)', diff saved to https://phabricator.wikimedia.org/P91877 and previous config saved to /var/cache/conftool/dbconfig/20260429-101358-fceratto.json 2407284 wikitext text/x-wiki 70cfr7xu60ylxetncsi35b6r1jahx7k 2407285 2407284 2026-04-29T10:14:11Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw 2407285 wikitext text/x-wiki m7gepgwape3pooccmxujxxgtjflqs5r 2407286 2407285 2026-04-29T10:14:20Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance 2407286 wikitext text/x-wiki ej1ocgrj5thya79gaxqflof9l3tpqfw 2407287 2407286 2026-04-29T10:14:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1194 (T419961)', diff saved to https://phabricator.wikimedia.org/P91878 and previous config saved to /var/cache/conftool/dbconfig/20260429-101426-fceratto.json 2407287 wikitext text/x-wiki t51k1lia9lxeaqb8lfbf560jirkhklb 2407288 2407287 2026-04-29T10:15:36Z Stashbot 7414 Emperor: disable puppet in apus/eqiad rgws for TLS key rollover T424674 2407288 wikitext text/x-wiki 3a104gap9bgdna1hql4hjz8dgad3p6f 2407289 2407288 2026-04-29T10:17:27Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2175.codfw.wmnet with reason: host reimage 2407289 wikitext text/x-wiki 0bis0rqxeaf0pve8wsm49q7cizuu0tq 2407290 2407289 2026-04-29T10:20:01Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1166.eqiad.wmnet with OS trixie 2407290 wikitext text/x-wiki rda6mlhzn6jah2s42yacvkwj87gmbon 2407291 2407290 2026-04-29T10:21:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T419961)', diff saved to https://phabricator.wikimedia.org/P91879 and previous config saved to /var/cache/conftool/dbconfig/20260429-102142-fceratto.json 2407291 wikitext text/x-wiki bcmkdrft2m37z5t2eqe9jobybapzsk4 2407292 2407291 2026-04-29T10:22:33Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw 2407292 wikitext text/x-wiki j3hbx9u6a1inf7kcwamkgjildrgau6p 2407293 2407292 2026-04-29T10:23:37Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet 2407293 wikitext text/x-wiki jdn3x40o1k33d2y0q3sx8a16vdxnh7m 2407294 2407293 2026-04-29T10:23:40Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet 2407294 wikitext text/x-wiki 27qplgqewrbidyobsowwjcnnwrb4y36 2407295 2407294 2026-04-29T10:24:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1166: after reimage to trixie 2407295 wikitext text/x-wiki bvzkblwdg2vem8a4njfsjwotubxq8y7 2407296 2407295 2026-04-29T10:25:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2175.codfw.wmnet with reason: host reimage 2407296 wikitext text/x-wiki sk7a2noku54uxzjahihfst8ho71wey0 2407297 2407296 2026-04-29T10:30:03Z Stashbot 7414 moritzm: installing Envoy upgrades on chartmuseum* T410975 T419637 2407297 wikitext text/x-wiki qa75p8nj8y3umjqg172v52zsfv5d4vo 2407298 2407297 2026-04-29T10:31:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P91882 and previous config saved to /var/cache/conftool/dbconfig/20260429-103150-fceratto.json 2407298 wikitext text/x-wiki b56hu20fuce66e4vbvg3ig060xkawme 2407299 2407298 2026-04-29T10:32:07Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2190.codfw.wmnet with OS trixie 2407299 wikitext text/x-wiki 9hwhd6eb81vbhzom5e9ubxc8d64qyc0 2407300 2407299 2026-04-29T10:32:41Z Stashbot 7414 moritzm: installing Envoy upgrades on webperf* T410975 T419637 2407300 wikitext text/x-wiki orf7d599fluilf56eq7z7tjup9ifstg 2407301 2407300 2026-04-29T10:42:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P91885 and previous config saved to /var/cache/conftool/dbconfig/20260429-104158-fceratto.json 2407301 wikitext text/x-wiki n209xvbstwqlqrklfmmhghaxgi83m87 2407302 2407301 2026-04-29T10:45:02Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage 2407302 wikitext text/x-wiki q2vxobo52entwoz7v8ryob7otz0v2l4 2407303 2407302 2026-04-29T10:48:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2175.codfw.wmnet with OS trixie 2407303 wikitext text/x-wiki oex4wcwvk1oj35hz7ozjtsq047b88ck 2407304 2407303 2026-04-29T10:49:09Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2175: After reimage 2407304 wikitext text/x-wiki qw1773jpj0jn8n7sxviugx9e68jk6ic 2407305 2407304 2026-04-29T10:49:13Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2175: After reimage 2407305 wikitext text/x-wiki fdufzkj2stzomf8li3zbxzb7ffndx55 2407306 2407305 2026-04-29T10:49:25Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2175: After reimage 2407306 wikitext text/x-wiki cfa7z3o6afvo4no2erky4z1dl2qf7vd 2407307 2407306 2026-04-29T10:50:18Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5005.eqsin.wmnet with reason: host reimage 2407307 wikitext text/x-wiki 7c8ev7opf5z2wtugxcs4ndlkd4mo1dr 2407308 2407307 2026-04-29T10:52:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T419961)', diff saved to https://phabricator.wikimedia.org/P91887 and previous config saved to /var/cache/conftool/dbconfig/20260429-105206-fceratto.json 2407308 wikitext text/x-wiki jtdbraj0m5odujfhi8qoz3ou1i4nq6g 2407309 2407308 2026-04-29T10:52:28Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance 2407309 wikitext text/x-wiki djf2r8j6wqu4j827jhc5ameu4bcce7e 2407310 2407309 2026-04-29T10:52:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1202 (T419961)', diff saved to https://phabricator.wikimedia.org/P91888 and previous config saved to /var/cache/conftool/dbconfig/20260429-105234-fceratto.json 2407310 wikitext text/x-wiki i7bx4xsegy773lqyw51vee5itvv89dx 2407311 2407310 2026-04-29T10:53:22Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1229: after reimage to trixie 2407311 wikitext text/x-wiki ajz8i6kzxhy4ayrrvb4mprnv6oe8tpc 2407312 2407311 2026-04-29T10:54:03Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet 2407312 wikitext text/x-wiki h8hddzlsyebz9awxf8uudebwde12hwd 2407313 2407312 2026-04-29T10:54:19Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet 2407313 wikitext text/x-wiki kmm57568fnock6vsk09ze1njb6zf2m6 2407314 2407313 2026-04-29T10:54:56Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet 2407314 wikitext text/x-wiki q1afzmhtk4pxo2iorl7i3nb12ybk3q5 2407315 2407314 2026-04-29T10:55:02Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet 2407315 wikitext text/x-wiki 7zyjoau3ayjnz5dnzqt4a86h27bmth3 2407316 2407315 2026-04-29T11:00:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T419961)', diff saved to https://phabricator.wikimedia.org/P91891 and previous config saved to /var/cache/conftool/dbconfig/20260429-110005-fceratto.json 2407316 wikitext text/x-wiki 17f0mte7q5pc3jolwoh1sn188wx0gex 2407317 2407316 2026-04-29T11:00:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2190: After reimage 2407317 wikitext text/x-wiki 7mvr019zz97ete0bn0holag4juuog24 2407318 2407317 2026-04-29T11:10:10Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1166: after reimage to trixie 2407318 wikitext text/x-wiki azyl646hvay94ppsx8hg1lhgalz8h2t 2407319 2407318 2026-04-29T11:10:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P91895 and previous config saved to /var/cache/conftool/dbconfig/20260429-111013-fceratto.json 2407319 wikitext text/x-wiki 5rb37hfxw28uvpljk49iuxblrrdsko3 2407320 2407319 2026-04-29T11:11:18Z Stashbot 7414 moritzm: installing libpng1.6 security updates 2407320 wikitext text/x-wiki 4hxbgwkdrvk4nospmxekooly3m7x8w7 2407321 2407320 2026-04-29T11:11:50Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5005.eqsin.wmnet with OS bookworm 2407321 wikitext text/x-wiki l691rgi3w4lntxcrgppzewjc9w1dvfn 2407322 2407321 2026-04-29T11:20:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P91898 and previous config saved to /var/cache/conftool/dbconfig/20260429-112021-fceratto.json 2407322 wikitext text/x-wiki 5g0ol1on94mhrxeq4yz5jg4arnn0uj7 2407323 2407322 2026-04-29T11:30:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T419961)', diff saved to https://phabricator.wikimedia.org/P91899 and previous config saved to /var/cache/conftool/dbconfig/20260429-113029-fceratto.json 2407323 wikitext text/x-wiki jmw1wol8km0ye0mo1nilntnkkc3cxo7 2407324 2407323 2026-04-29T11:30:52Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance 2407324 wikitext text/x-wiki tg3ieji0ikrs000h6f1x1pp2mwle0zw 2407325 2407324 2026-04-29T11:31:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 (T419961)', diff saved to https://phabricator.wikimedia.org/P91901 and previous config saved to /var/cache/conftool/dbconfig/20260429-113105-fceratto.json 2407325 wikitext text/x-wiki ck72xr78uyqrgatrc99g1rwcon8vdnf 2407326 2407325 2026-04-29T11:34:53Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2175: After reimage 2407326 wikitext text/x-wiki 666baaqzsu65n8wjlbr4i3k9b3hf3jp 2407330 2407326 2026-04-29T11:38:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T419961)', diff saved to https://phabricator.wikimedia.org/P91903 and previous config saved to /var/cache/conftool/dbconfig/20260429-113813-fceratto.json 2407330 wikitext text/x-wiki mp9rw0ntk4tgeouspq9cniuomrnwqw3 2407331 2407330 2026-04-29T11:39:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2407331 wikitext text/x-wiki 8mdskni4n4qcevc6gm5lx05mxsk7dda 2407332 2407331 2026-04-29T11:39:56Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2407332 wikitext text/x-wiki dkwmxcz9fq2buq6eqkpufud8lkqiy80 2407333 2407332 2026-04-29T11:42:37Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply 2407333 wikitext text/x-wiki jml5z86o23hb6x93z21ble3eubszp57 2407334 2407333 2026-04-29T11:43:08Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply 2407334 wikitext text/x-wiki 7wbdfs5mtuw46v6nbfjzea2xqtzrnu9 2407335 2407334 2026-04-29T11:43:47Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply 2407335 wikitext text/x-wiki 15pcf0bksk3vny2ypeo6plx8q9obteg 2407336 2407335 2026-04-29T11:44:18Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply 2407336 wikitext text/x-wiki n01894799vajjeztzc9unrk0yqknk0i 2407337 2407336 2026-04-29T11:46:02Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2190: After reimage 2407337 wikitext text/x-wiki lqd5k24jxipfbp4ldin4agrtl8xp6rf 2407338 2407337 2026-04-29T11:47:16Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2407338 wikitext text/x-wiki idmrvn0cphzdqp3tk7kwwgh8h4882x9 2407339 2407338 2026-04-29T11:47:46Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2407339 wikitext text/x-wiki rwvr2fybggmf2891uf670p2dgw0o9h5 2407340 2407339 2026-04-29T11:47:52Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2407340 wikitext text/x-wiki qcb7mo46p7yi94iv68pmrair387y6z0 2407341 2407340 2026-04-29T11:48:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P91905 and previous config saved to /var/cache/conftool/dbconfig/20260429-114821-fceratto.json 2407341 wikitext text/x-wiki 5krtjq35jusp0rszdd6jl3b256v8552 2407342 2407341 2026-04-29T11:48:30Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2407342 wikitext text/x-wiki 3o5w0dx3pf0msdlbj44fv7r2rw8e7i4 2407343 2407342 2026-04-29T11:51:20Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2407343 wikitext text/x-wiki hixuz8pb7rnyajumzzjxb505k8ul5f0 2407344 2407343 2026-04-29T11:51:50Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2407344 wikitext text/x-wiki j2yoiz7j0ubywk2nhipagpg1q0qokgg 2407345 2407344 2026-04-29T11:51:58Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2407345 wikitext text/x-wiki g1nricv4vym96tkx31g5ersm01zbfup 2407346 2407345 2026-04-29T11:52:28Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2407346 wikitext text/x-wiki j3k4gtmwxucon89tq8bt4xlmkx836vg 2407347 2407346 2026-04-29T11:52:35Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2407347 wikitext text/x-wiki 5bpbl7njqer906cctdnq55j2kew9ikz 2407348 2407347 2026-04-29T11:53:12Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2407348 wikitext text/x-wiki n5wgfgusp8jg4tx5gng6db1fnuk0ab0 2407349 2407348 2026-04-29T11:53:42Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply 2407349 wikitext text/x-wiki f9xpy4053dtv2i7n4w5x98273dhic02 2407350 2407349 2026-04-29T11:54:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply 2407350 wikitext text/x-wiki flq74pojzykzcjqxozld5hnrcono0xt 2407351 2407350 2026-04-29T11:54:31Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply 2407351 wikitext text/x-wiki 515puefjk9nk3sqjqdtij9xkgvl9rdv 2407352 2407351 2026-04-29T11:54:37Z Stashbot 7414 Emperor: TLS key rollover for thanos-fe T424674 2407352 wikitext text/x-wiki envbq1376mcz67v8xm8ifopdm90xhv7 2407353 2407352 2026-04-29T11:55:01Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply 2407353 wikitext text/x-wiki psuriicy2cbtp4wcaox9pmsx1z9j2h2 2407354 2407353 2026-04-29T11:56:37Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2407354 wikitext text/x-wiki ac4yu12vfzmw50kcc7yu0offfk7uk2d 2407355 2407354 2026-04-29T11:57:09Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2407355 wikitext text/x-wiki taoi77uy4b2sognxeh52iworpfxycay 2407356 2407355 2026-04-29T11:57:15Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply 2407356 wikitext text/x-wiki d9hvtx8mx1jvemjm86ugtsl38h86uef 2407357 2407356 2026-04-29T11:57:47Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply 2407357 wikitext text/x-wiki b9ijt8muw3thpfb4klo3w8i4p31l6od 2407358 2407357 2026-04-29T11:58:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P91906 and previous config saved to /var/cache/conftool/dbconfig/20260429-115829-fceratto.json 2407358 wikitext text/x-wiki 0a5liz0hc4h80mw555u918g9168u9zr 3 16252 2406905 2269995 2026-04-28T13:26:08Z StrikerBot 8475 Welcome to Toolforge! 2406905 wikitext text/x-wiki k7yf76504dzvzbexpjb2y84qqi2tovz 0 17290 2406888 2406766 2026-04-28T13:14:59Z Stashbot 7414 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI 2406888 wikitext text/x-wiki r5a096v9hdsmym58xff5yci99zjl42i 2406914 2406888 2026-04-28T13:34:40Z Stashbot 7414 James_F: Zuul: [oojs/ui] Drop PHP 8.2 testing 2406914 wikitext text/x-wiki 8mqie3z5obo75jzcbykko2n8bay38q1 2406915 2406914 2026-04-28T13:34:45Z Stashbot 7414 James_F: Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing 2406915 wikitext text/x-wiki 2dh0k05qsvazn1jpjyoywe84whog2u2 2406995 2406915 2026-04-28T16:32:36Z Stashbot 7414 James_F: Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing 2406995 wikitext text/x-wiki jk9z1buzt9r2ahg89m0vo6jupcceicp 2406999 2406995 2026-04-28T17:07:10Z Stashbot 7414 James_F: Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing 2406999 wikitext text/x-wiki h3czec9lvbduc4r8neucywhozmzlxl2 2407000 2406999 2026-04-28T17:07:15Z Stashbot 7414 James_F: Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing 2407000 wikitext text/x-wiki e90rt0jimux63wog0z0adqb87a0gotp 2407001 2407000 2026-04-28T17:07:22Z Stashbot 7414 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing 2407001 wikitext text/x-wiki f7f9xixjq2zecdzqjmakbvjv7pwjchf 2407012 2407001 2026-04-28T17:41:15Z Stashbot 7414 bd808: Unblock 79.192.0.0/10 2407012 wikitext text/x-wiki thbv4e7zjz4yyeve5yursw44ecawtcl 2407024 2407012 2026-04-28T18:11:45Z Stashbot 7414 bd808: Unblock 86.0.0.0/16 2407024 wikitext text/x-wiki 6703yt9a73o7kc4jojsxm234knvoysy 0 20365 2406997 2406555 2026-04-28T16:51:31Z Krinkle 44 /* Debugging databases */ rm warning, works fine and is afaik preferred. I assume "sql" will be updated if/when its underlying mwscript is ever not supported. mwscript-k8s is too complicated to interface directly for all use cases, that's why sql exists. 2406997 wikitext text/x-wiki 8tolgii19zokvdgq3gkbh26ymtseaej 498 30942 2406932 2406682 2026-04-28T14:33:22Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate 2406932 wikitext text/x-wiki pznl9mb0dbjxotifhnb6evlbu5p8e1s 2406933 2406932 2026-04-28T14:33:54Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate 2406933 wikitext text/x-wiki md0fiksiibxhuq33nqeuha5dvwxrr0e 0 221439 2407179 2374675 2026-04-29T07:44:25Z BRouberol-WMF 39141 /* History */ 2407179 wikitext text/x-wiki bqmy3zivljk4blub50x0xgi0d8w1clu 0 440607 2407068 2406545 2026-04-28T21:49:01Z Neil Shah-Quinn (WMF) 36601 Move details on SQL syntax to [[Data Platform/Data Lake/SQL|SQL subpage]] 2407068 wikitext text/x-wiki 0p39pnw295j1ifj6itipx7kdvajcukk 0 441095 2407118 2377392 2026-04-29T05:46:02Z AYounsi (WMF) 10040 /* Primary outbound port utilization over 80% (TransitPeeringTransportOutSaturation) */ 2407118 wikitext text/x-wiki 85byr59m060bna6v3kx6zqkipktmlzl 0 448298 2406855 2031520 2026-04-28T12:40:35Z MLechvien-WMF 49197 2406855 wikitext text/x-wiki qcmn3mhjkx9gjlis6xtw90z2wd38ujz 0 448566 2406960 2380408 2026-04-28T15:36:21Z BryanDavis 1604 /* Contact the ML team */ link phab tag 2406960 wikitext text/x-wiki f5ax6hzgczxlahgpssjs4jiq2eexa38 0 449160 2406860 2406784 2026-04-28T12:42:43Z Dexbot 30554 Bot: Updating the report 2406860 wikitext text/x-wiki km4c9ntcvf4yol5drvx5y8sggxqeuqi 2407076 2406860 2026-04-29T00:02:22Z Dexbot 30554 Bot: Updating the report 2407076 wikitext text/x-wiki cxjksns4ggnbfnuv4nubyphud4dsg2k 2407092 2407076 2026-04-29T05:09:02Z Dexbot 30554 Bot: Updating the report 2407092 wikitext text/x-wiki r0e56w1eq2eczc1qr2xz4q62p82q73f 2407227 2407092 2026-04-29T08:57:32Z Dexbot 30554 Bot: Updating the report 2407227 wikitext text/x-wiki ji2283mlb1v4ei5zsv60d6mv7rj5bal 0 451533 2406971 2395323 2026-04-28T15:56:02Z APuthin-WMF 52425 Setting up next meeting 2406971 wikitext text/x-wiki 2b592psqlfm5or43msc1n48rwik4705 0 453597 2407067 2399816 2026-04-28T21:47:55Z BryanDavis 1604 /* 2026 */ ingress-ngix -> istio 2407067 wikitext text/x-wiki ncsuvx2hwspsjbcl8wxq91m3sn1cfe4 2407069 2407067 2026-04-28T21:49:19Z Lucas Werkmeister 8093 /* 2026 */ add listarchive, like for 2026-04-01 2407069 wikitext text/x-wiki tp2yk24cdm4e4wrzq03qam3i8xtrm8d 116 453906 2407075 2406551 2026-04-28T23:09:41Z Gitlabaccountapprovalbot 37332 @dtorsani was approved. 2407075 wikitext text/x-wiki 03u77nygz4fzf0uqlf9hlu9n4eharg1 2407083 2407075 2026-04-29T02:27:30Z Gitlabaccountapprovalbot 37332 @dw31415 was approved. 2407083 wikitext text/x-wiki 9zgkrq2qroytzrk0p2i3rwboiln04k9 0 455989 2406884 2405840 2026-04-28T13:11:43Z Phuedx (WMF) 47850 First pass at spreading out the detail about non-cache-splitting experiments into the relevant sections 2406884 wikitext text/x-wiki c2494i6kyqcy7isti1tfgw1hf7q0kyd 2406929 2406884 2026-04-28T14:13:52Z Phuedx (WMF) 47850 /* HTTP header */ 2406929 wikitext text/x-wiki rsx21y7nv6poybcvkhjgbmmnegafi4x 0 456005 2406930 2362223 2026-04-28T14:19:16Z BJensen-WMF 48942 /* Tutorial: Using the WebUI */ Add a first pass at instructions for the Requestctl web ui. 2406930 wikitext text/x-wiki kp2q7imujpkrb91ojq6avg6cpg58s9a 0 459115 2407088 2398756 2026-04-29T03:38:10Z AndrewBogott (WMF) 51047 /* Cluster-api (capi) Components */ 2407088 wikitext text/x-wiki 725udc8fk27bhadd5du2yb4d5rn5jjr 0 459242 2407079 2368516 2026-04-29T01:02:33Z AndrewBogott (WMF) 51047 2407079 wikitext text/x-wiki cie3sg7f2t8d7far5kwodqac2ff5p7m 0 459290 2407074 2405707 2026-04-28T22:59:37Z XCollazo-WMF 31846 /* NEXT TRAIN */ 2407074 wikitext text/x-wiki r2q05besafdixdv9zubtj7ovg6yvq6e 0 459308 2406945 2351840 2026-04-28T15:16:14Z BryanDavis 1604 bold in lede; use [[Template:Tracked]] 2406945 wikitext text/x-wiki eo3a6hm6wqtc0ohlmcwmz8jmx1znje7 2407013 2406945 2026-04-28T17:43:54Z AndrewBogott (WMF) 51047 Added note about project status 2407013 wikitext text/x-wiki 4k11ah6lf5zu4ohoqt3665v2uxfho6a 0 459699 2407057 2406489 2026-04-28T20:58:51Z Stashbot 7414 larssandergreen: civicrm upgraded from be3bb76b to 081d5a29 2407057 wikitext text/x-wiki s3gvp2t6ip4agos45wcu6mq4xuk0mkk 0 459719 2406996 2406365 2026-04-28T16:45:02Z BTullis (WMF) 25295 /* Service Level Indicators (SLIs) */ 2406996 wikitext text/x-wiki pzywksheapnr4oxicuk2o4dyc7ycriy 1 459740 2407244 2405014 2026-04-29T09:27:50Z Lewisiscrazy 43317 /* Request for clarification */ Reply 2407244 wikitext text/x-wiki nu24ed32zr4x0ryu3pribm2etfl59ly 3 459802 2407029 2385654 2026-04-28T18:28:57Z StrikerBot 8475 Welcome to Toolforge! 2407029 wikitext text/x-wiki pmgi3pqsdjg14sj4cecqyqa9yq688pg 116 460070 2407077 2406015 2026-04-29T00:49:31Z EtherpadBackupBot 54504 rooks-questions-to-alex archived by BDavis (WMF) 2407077 wikitext text/x-wiki 8jtj46zac5np5yhsfdfbr5txoyree05 2407078 2407077 2026-04-29T00:49:57Z EtherpadBackupBot 54504 better-security-patch-management archived by BDavis (WMF) 2407078 wikitext text/x-wiki s2l5iyo0q19ej4pe50qgrygw9mvbo6u 2 460100 2406958 2406822 2026-04-28T15:29:58Z Effie Mouzeli (WMF) 12880 2406958 wikitext text/x-wiki 6hbx9tcu5g2rvhj73munbzvybhq51sp 2406965 2406958 2026-04-28T15:50:18Z Effie Mouzeli (WMF) 12880 2406965 wikitext text/x-wiki 02143zdywb0r196cmuaex5rcelaj2wf 2406966 2406965 2026-04-28T15:51:58Z Effie Mouzeli (WMF) 12880 2406966 wikitext text/x-wiki j2xl0c687o1dic8afm98csi3qjv9juy 0 460102 2406936 2026-04-28T14:49:41Z APuthin-WMF 52425 Adding notes from March meeting 2406936 wikitext text/x-wiki 8hohlf23ajqctkc64thguyauutrj65l 0 460103 2406968 2026-04-28T15:53:03Z APuthin-WMF 52425 Add April meeting notes 2406968 wikitext text/x-wiki 7by4qcqm87k9rztdtl8pqsmmuu4ve8y 0 460104 2407021 2026-04-28T18:05:49Z Jlwoodwa 38518 Redirected page to [[Edge uniques]] 2407021 wikitext text/x-wiki a2u3bwwfex7axsw1c63ly7ndvtg5114 0 460105 2407050 2026-04-28T20:32:22Z CDobbins-WMF 37390 Created page with "==Before you begin== '''Permissions''': Some processes documented in runbooks for Traffic will need varying levels of permission to perform. Make sure you have appropriate permissions to complete your task. ==What is a runbook?== A runbook is a detailed set of instructions that explain how to perform a common task or procedure, so it can be easily and accurately repeated by others. Runbooks are particularly useful for incident response operations. Creating runbooks in..." 2407050 wikitext text/x-wiki 1cjxafdi2igewcpjkzehgm8yqk0xctz 2 460106 2407061 2026-04-28T21:18:38Z Effie Mouzeli (WMF) 12880 Created page with "<!-- ======================================================================= Wikitech service page template Use this as a starting point when creating a new service page on Wikitech. A well-written page that follows this structure will contain everything an SRE needs to assess the service during a SCROLL review — without the page feeling like a checklist. Instructions: - Replace all guidance with real prose. - Delete comment b..." 2407061 wikitext text/x-wiki 3rz07mxjqezis93gdbjqmaz1l6zb0mp 0 460107 2407065 2026-04-28T21:47:33Z Neil Shah-Quinn (WMF) 36601 Create page with content from [[Data Platform/Data Lake]]. 2407065 wikitext text/x-wiki qk8dfxhx5h53w1vp2yr5vv3907glhpz 2 460108 2407137 2026-04-29T06:25:47Z SSapaty (WMF) 28955 Initial content for updated process 2407137 wikitext text/x-wiki no9bvftnow1r5u1ep3fiyxgw2ugpyqv 2407138 2407137 2026-04-29T06:32:06Z SSapaty (WMF) 28955 2407138 wikitext text/x-wiki 0ul0g2qidh5xaxvw59tfkuzgswpa724 2407139 2407138 2026-04-29T06:32:45Z SSapaty (WMF) 28955 2407139 wikitext text/x-wiki mc29c62byyalwmnyb60k3sc9h2zi10v 2407141 2407139 2026-04-29T06:34:44Z SSapaty (WMF) 28955 2407141 wikitext text/x-wiki j1mr7hsxekmfyoaqmuj5ps9moff52u0 2407142 2407141 2026-04-29T06:36:40Z SSapaty (WMF) 28955 2407142 wikitext text/x-wiki rin0d26txgdrgj1hpmdlmfgvl7qrjd1