Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.13 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2442912 2442210 2026-07-30T09:16:54Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 2442912 wikitext text/x-wiki tchqrn0p1a39fpyi2brfk1vumhb8f37 2442913 2442912 2026-07-30T09:16:57Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 2442913 wikitext text/x-wiki 4tjhxtc4mqcvtzaqyh97bw65miy0gek 2442916 2442913 2026-07-30T09:18:57Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 2442916 wikitext text/x-wiki 6em8ahnvnwuc1v9czlo2oj8fkotqlwk 2442918 2442916 2026-07-30T09:23:48Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 2442918 wikitext text/x-wiki prsvb8jj7f587yoilafj1ckybk2qnjk 2442919 2442918 2026-07-30T09:24:33Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 2442919 wikitext text/x-wiki rxqpmpwr2l34ly8wfzx37ks7ojzhjkr 2442920 2442919 2026-07-30T09:29:05Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 2442920 wikitext text/x-wiki djd1qcw3l6lscfk9hif9vl0ay5yc20r 2442945 2442920 2026-07-30T10:02:36Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission 2442945 wikitext text/x-wiki k9cnpzjhbqx2jl3uyp5pg9tfx3vyqou 2442947 2442945 2026-07-30T10:14:35Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission 2442947 wikitext text/x-wiki 65dncpcyfuj1aizw4mia6wvn16kuin9 0 3170 2442642 2437719 2026-07-29T16:43:33Z BCornwall-WMF 31521 /* Change GeoDNS / Depool a Site */ We no longer use geodns, so remove it from the heading 2442642 wikitext text/x-wiki fgk0pd6kk3tw74p3r1gexk6ujgcvih2 2442681 2442642 2026-07-29T17:34:31Z BCornwall-WMF 31521 ce: Rework all code blocks, some formatting changes too 2442681 wikitext text/x-wiki 5nriullv7omc9w54ov9ou0336549bhf 2442734 2442681 2026-07-29T18:33:18Z BCornwall-WMF 31521 ce: clean up redundant/time-based info 2442734 wikitext text/x-wiki 5sumvge6piafywk4xd6hvbkwsjqssi6 2442737 2442734 2026-07-29T18:35:24Z BCornwall-WMF 31521 Link to [[SRE/Traffic]] for contact info 2442737 wikitext text/x-wiki eeo9ltndhnf12jkhyp0fxbx34cykmub 2442738 2442737 2026-07-29T18:37:21Z BCornwall-WMF 31521 /* Deploying DNS changes */ use irc template 2442738 wikitext text/x-wiki loesf3rlnez4htjwxkzpec730bh4noq 2442740 2442738 2026-07-29T18:43:13Z BCornwall-WMF 31521 /* DNS Host Maintenance */ ce: Linkify, fix numbering, add shell prompt 2442740 wikitext text/x-wiki n863713u7ifhzsoffi66unaor2zebh8 2442741 2442740 2026-07-29T18:46:14Z BCornwall-WMF 31521 /* Change one or more services' public IPs */ condense examples, use gerrit template 2442741 wikitext text/x-wiki j43x8gygbzy6hsmwuqpdq8sibjpj16i 2442743 2442741 2026-07-29T18:52:25Z BCornwall-WMF 31521 /* Change one or more services' public IPs */ ce: shorten more gerrit changes using the template 2442743 wikitext text/x-wiki s4qmp9ghyxtf6yojxmifeyxbukn7fjp 2442744 2442743 2026-07-29T18:53:35Z BCornwall-WMF 31521 /* Depool a site */ Remove two-year old mention of no longer using geodns 2442744 wikitext text/x-wiki 9d2r3uzs126p8ctmc1175kearvyes38 2442746 2442744 2026-07-29T19:00:08Z BCornwall-WMF 31521 /* Deploying DNS changes */ Remove mention of using ns servers for SSH 2442746 wikitext text/x-wiki mwyo0u1xtqjwje6scgwazy4w7t3af5t 2442749 2442746 2026-07-29T19:04:29Z BCornwall-WMF 31521 /* Wipe caches */ Remove redundant help output from cookbook command 2442749 wikitext text/x-wiki bu5gtdff0fi5r2a8jyovyeeetbnyfue 2442751 2442749 2026-07-29T19:10:41Z BCornwall-WMF 31521 /* Domain templates */ mediawiki.{com,net} are no longer symlinks to .org, so replace the whole thing with example.com 2442751 wikitext text/x-wiki bou99seb0fztzz19ju6413fvhfbxjxu 2442786 2442751 2026-07-29T20:16:13Z BCornwall-WMF 31521 /* Wipe caches */ Reduce heading lengths 2442786 wikitext text/x-wiki hwrqw40qb7v8y0gx2tamwmp6cq07lsc 0 3788 2442792 2439013 2026-07-29T20:24:44Z BCornwall-WMF 31521 /* TTL */ ce: Italicize keep and grace 2442792 wikitext text/x-wiki i7kgovfr1x07se0ut3t6rf2olirx4gq 0 4108 2442294 2442255 2026-07-29T12:25:40Z ScheduleDeploymentBot 37566 Add [[gerrit:1319075]] to Wednesday, July 29 UTC afternoon backport window 2442294 wikitext text/x-wiki huq9ko8ks5lpry95az9o644p5my3pbu 2442303 2442294 2026-07-29T12:34:11Z ScheduleDeploymentBot 37566 Add [[gerrit:1319077]] to Wednesday, July 29 UTC afternoon backport window 2442303 wikitext text/x-wiki qj9ge1gitg116ys6auzr5qn7uawviwm 2442304 2442303 2026-07-29T12:34:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1319078]] to Wednesday, July 29 UTC afternoon backport window 2442304 wikitext text/x-wiki 0ku69efqiec3s0emtuj7qofwllv7x0v 2442305 2442304 2026-07-29T12:34:26Z ScheduleDeploymentBot 37566 Add [[gerrit:1319079]] to Wednesday, July 29 UTC afternoon backport window 2442305 wikitext text/x-wiki gezftidi6zli0fbj5x1ag444ur90i0r 2442306 2442305 2026-07-29T12:34:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1319080]] to Wednesday, July 29 UTC afternoon backport window 2442306 wikitext text/x-wiki 6md7k1ind5tj4g00psb4u1ywzu917ek 2442310 2442306 2026-07-29T12:38:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1309675]] to Wednesday, July 29 UTC afternoon backport window 2442310 wikitext text/x-wiki h0jiuey4131okocbp33gbzdb5qc0fem 2442968 2442310 2026-07-30T11:40:59Z ScheduleDeploymentBot 37566 Add [[gerrit:1319449]] to Thursday, July 30 UTC afternoon backport window 2442968 wikitext text/x-wiki t5yryxdun9x7qzola9xzfv0d29g0hyj 498 6726 2442939 2441242 2026-07-30T09:54:18Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission 2442939 wikitext text/x-wiki ntxxnyzqe3njkbsh324e3zquazcmhfq 2442944 2442939 2026-07-30T10:02:14Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission 2442944 wikitext text/x-wiki oatiugh90vtj05wkhl59jg8k60dtgk5 0 7919 2442265 2442264 2026-07-29T12:00:09Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-coord1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" 2442265 wikitext text/x-wiki tvs2619gf4v0wadm6uyzdgj7b2gda4f 2442266 2442265 2026-07-29T12:00:13Z Stashbot 7414 marostegui: Rename tables T425074 2442266 wikitext text/x-wiki ttubhm6fo9ahngbtzdhwyixpukqqap4 2442267 2442266 2026-07-29T12:01:00Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1098 hosts 2442267 wikitext text/x-wiki qx61tx21bonggqm6sggkiq7blm4e6w9 2442268 2442267 2026-07-29T12:04:18Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance 2442268 wikitext text/x-wiki 1tkqipkprn28ixlg8mk5wbph600856h 2442269 2442268 2026-07-29T12:04:26Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1222 (T431660)', diff saved to https://phabricator.wikimedia.org/P95530 and previous config saved to /var/cache/conftool/dbconfig/20260729-120424-cwilliams.json 2442269 wikitext text/x-wiki g1p36dcf1mbk38ah1s4gwaeqpvgcich 2442270 2442269 2026-07-29T12:07:59Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Maintenance 2442270 wikitext text/x-wiki kfclvpjd74l99np2nmx1z3lgb9wi88r 2442271 2442270 2026-07-29T12:08:19Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance 2442271 wikitext text/x-wiki 97q5fff1jv9qf69rlbjt8o5b0z2uhl9 2442272 2442271 2026-07-29T12:08:41Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2442272 wikitext text/x-wiki jsozn1bca97c3mv8hwtwasftw7ud1a4 2442273 2442272 2026-07-29T12:08:49Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1196 (T431660)', diff saved to https://phabricator.wikimedia.org/P95533 and previous config saved to /var/cache/conftool/dbconfig/20260729-120847-cwilliams.json 2442273 wikitext text/x-wiki 4q75dpg2aecv63omjxgyls5glu47983 2442274 2442273 2026-07-29T12:11:40Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2442274 wikitext text/x-wiki clsoup8373or0cwxc72p932ajy3dbsd 2442275 2442274 2026-07-29T12:11:48Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T431660)', diff saved to https://phabricator.wikimedia.org/P95534 and previous config saved to /var/cache/conftool/dbconfig/20260729-121146-cwilliams.json 2442275 wikitext text/x-wiki b9valbjp496t49i7ydt3rxez8le0mj7 2442276 2442275 2026-07-29T12:11:54Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance 2442276 wikitext text/x-wiki bf2zthdwiffabzxz1rgi0g769yqrqon 2442277 2442276 2026-07-29T12:12:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2442277 wikitext text/x-wiki fzvl1shyu41r3eyguyeube9i1208ct9 2442278 2442277 2026-07-29T12:12:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1158 (T431660)', diff saved to https://phabricator.wikimedia.org/P95535 and previous config saved to /var/cache/conftool/dbconfig/20260729-121211-cwilliams.json 2442278 wikitext text/x-wiki omrq8v031709t9mz3cgqaoilch25te0 2442279 2442278 2026-07-29T12:12:38Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Maintenance 2442279 wikitext text/x-wiki r2ofa2pltwi1ng5an5aqfg4pqhrsrkd 2442281 2442279 2026-07-29T12:14:44Z Stashbot 7414 Daimona: Creating new DB tables for the CampaignEvents extension in x1.testwiki, x1.test2wiki, x1.officewiki, and x1.wikishared # T429339 2442281 wikitext text/x-wiki 9rsppm881t5dx5emjki7vsk6fd96n4u 2442282 2442281 2026-07-29T12:15:26Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1196: Maintenance 2442282 wikitext text/x-wiki l1ocsnybx0eqmabum7phnxav7f1ohg8 2442283 2442282 2026-07-29T12:15:52Z Stashbot 7414 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: sync 2442283 wikitext text/x-wiki ppttd7y5dufcu8u1b8boq6mjtyevj9v 2442284 2442283 2026-07-29T12:17:04Z Stashbot 7414 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: sync 2442284 wikitext text/x-wiki r7dt6p6cxwotawsxl4o3qaatf5l8fsr 2442285 2442284 2026-07-29T12:17:30Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1159: Maintenance 2442285 wikitext text/x-wiki bk4pq1y7ty22vn5xzw5cpfx5kkfhra6 2442286 2442285 2026-07-29T12:19:12Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1212: Maintenance 2442286 wikitext text/x-wiki r5rixok1fksholttqria9k0vabt6ilu 2442287 2442286 2026-07-29T12:19:30Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance 2442287 wikitext text/x-wiki lj27a3izaf5mmugghswj9lf12kkw36w 2442288 2442287 2026-07-29T12:19:39Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1223 (T431660)', diff saved to https://phabricator.wikimedia.org/P95540 and previous config saved to /var/cache/conftool/dbconfig/20260729-121937-cwilliams.json 2442288 wikitext text/x-wiki 9s5woxkgguvuym6pi8dbl5b3xpjdnz4 2442289 2442288 2026-07-29T12:20:23Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1158: Maintenance 2442289 wikitext text/x-wiki rc9oa86fucfe35pu4ke0ux4sti26vom 2442290 2442289 2026-07-29T12:20:40Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1051 hosts 2442290 wikitext text/x-wiki liblch3zlh71rhearygdmy2cgz000tc 2442291 2442290 2026-07-29T12:22:33Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1241: Maintenance 2442291 wikitext text/x-wiki tnpzct98sozen56okxer5t60mxg17qp 2442292 2442291 2026-07-29T12:22:47Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance 2442292 wikitext text/x-wiki iq0tihp99jhqgxjmbrwt6txemfa78nd 2442293 2442292 2026-07-29T12:22:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1242 (T431660)', diff saved to https://phabricator.wikimedia.org/P95544 and previous config saved to /var/cache/conftool/dbconfig/20260729-122254-cwilliams.json 2442293 wikitext text/x-wiki rk9cfhl6tngu7are13kuarbiqr0jr9u 2442295 2442293 2026-07-29T12:25:46Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1223: Maintenance 2442295 wikitext text/x-wiki e9ksw95huikfrzi1hzzpfyvpaig68ti 2442296 2442295 2026-07-29T12:29:28Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1242: Maintenance 2442296 wikitext text/x-wiki soexcyzotp04x1205i4j0ykph3bbgi2 2442297 2442296 2026-07-29T12:32:53Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve101[2-5].eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2442297 wikitext text/x-wiki qvgxgw490r23cfl6ozoonszhviz5gwr 2442298 2442297 2026-07-29T12:32:57Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1012.eqiad.wmnet 2442298 wikitext text/x-wiki 9acl9t7lqzrizr2k2lmsvjztv2uhkjy 2442299 2442298 2026-07-29T12:33:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-coord1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" 2442299 wikitext text/x-wiki go610yik3rlmcs4h563j01j3fxuj8jo 2442300 2442299 2026-07-29T12:33:18Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442300 wikitext text/x-wiki pffrzxrx0e3t77834tphircg47wkba5 2442301 2442300 2026-07-29T12:33:20Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-test-coord1001.eqiad.wmnet 2442301 wikitext text/x-wiki 9uhzx0phk7rrikn0qtyrlowvx71fmt2 2442302 2442301 2026-07-29T12:34:07Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.decommission for hosts an-test-master[1001-1002].eqiad.wmnet 2442302 wikitext text/x-wiki ahq0bvl31h5xgjy8eniy6fdhsaa3k48 2442307 2442302 2026-07-29T12:35:54Z Stashbot 7414 ayounsi@dns1004: START - running authdns-update 2442307 wikitext text/x-wiki b3hfkf6fahwx2ib3vvy0fyovo93wv93 2442308 2442307 2026-07-29T12:38:00Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1012.eqiad.wmnet 2442308 wikitext text/x-wiki sokoipr8meug66s10uiksek6a69svo5 2442309 2442308 2026-07-29T12:38:04Z Stashbot 7414 ayounsi@dns1004: END - running authdns-update 2442309 wikitext text/x-wiki k6isfrtr9t6fpadryy1pjuwgptj9ql9 2442311 2442309 2026-07-29T12:40:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test2001.codfw.wmnet 2442311 wikitext text/x-wiki h5gddop6bc6o7twv0w7kdf6cx90pweo 2442312 2442311 2026-07-29T12:40:53Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test2001.codfw.wmnet 2442312 wikitext text/x-wiki 3y3gpiymwyj9emq0xxgkh80q0msoy9v 2442313 2442312 2026-07-29T12:44:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2442313 wikitext text/x-wiki tqpgbqx1pn4e7e9grb24ws5noeohec3 2442314 2442313 2026-07-29T12:45:03Z Stashbot 7414 sukhe: sudo cumin 'O:url_downloader' 'disable-puppet "merging CR 1313948"': T429175 2442314 wikitext text/x-wiki s3w21bi0v7yhbcs272n9bldxnrkvpj2 2442315 2442314 2026-07-29T12:45:12Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1012.eqiad.wmnet 2442315 wikitext text/x-wiki j90d38l1f4xdrff98n7wek0mq56ojet 2442316 2442315 2026-07-29T12:45:15Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1012.eqiad.wmnet 2442316 wikitext text/x-wiki 96744hiq105cmz7drke3kbcbjpmzmy2 2442317 2442316 2026-07-29T12:45:22Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1013.eqiad.wmnet 2442317 wikitext text/x-wiki a3gtrm3fvli98pkinaiqlvhz4ms9lp2 2442318 2442317 2026-07-29T12:48:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-master[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" 2442318 wikitext text/x-wiki ma29egqh1xqzwj3da6wpvdxgkkpmb64 2442319 2442318 2026-07-29T12:50:09Z Stashbot 7414 sukhe: sudo cumin 'O:url_downloader' 'run-puppet-agent --enable "merging CR 1313948"': T429175 2442319 wikitext text/x-wiki 6atff6ly3b6asrtx0zql71urmsegc99 2442320 2442319 2026-07-29T12:50:27Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet 2442320 wikitext text/x-wiki jcprqflzuwght053e6p65pv4kid7y37 2442321 2442320 2026-07-29T12:56:07Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet 2442321 wikitext text/x-wiki eaue98gcpvyao07wse1hdmscxm6wcdc 2442322 2442321 2026-07-29T12:56:09Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet 2442322 wikitext text/x-wiki b4l92afueiyt755illdnqptbp8yhmjw 2442323 2442322 2026-07-29T12:56:14Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet 2442323 wikitext text/x-wiki f9diuoiej72dxqkjjxtiht44urevegx 2442324 2442323 2026-07-29T12:57:50Z Stashbot 7414 sukhe: sudo cumin 'A:lvs and (A:eqiad or A:codfw)' 'disable-puppet "adding new service urldownloader"': T429175 2442324 wikitext text/x-wiki 4web0yraux3aqzcjpqthevglarsa1h2 2442325 2442324 2026-07-29T12:59:21Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Maintenance 2442325 wikitext text/x-wiki r2ixy9vxxkroqm8ydyymqi6u6lrpswq 2442326 2442325 2026-07-29T12:59:44Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance 2442326 wikitext text/x-wiki o0r6qhj0cfixsed97ao1k4h2b1nyrmt 2442327 2442326 2026-07-29T12:59:52Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1229 (T431660)', diff saved to https://phabricator.wikimedia.org/P95559 and previous config saved to /var/cache/conftool/dbconfig/20260729-125950-cwilliams.json 2442327 wikitext text/x-wiki lc8y2wraz3azm4905dqm6yg2xb55ppr 2442328 2442327 2026-07-29T13:00:29Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply 2442328 wikitext text/x-wiki 6sgqyw12rlt47u29iiicdhscw9odboo 2442329 2442328 2026-07-29T13:00:34Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply 2442329 wikitext text/x-wiki 1z9hy7da748p6v2f0x5e0a1oyxel7to 2442330 2442329 2026-07-29T13:01:20Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2442330 wikitext text/x-wiki hlewhi8wqm4v7ddz4aklv900h0zq5t7 2442331 2442330 2026-07-29T13:01:23Z Stashbot 7414 sukhe: sukhe@lvs1020:~$ sudo systemctl restart pybal.service 2442331 wikitext text/x-wiki 1am467f32u12d75j792rn1u4wd8lxfz 2442332 2442331 2026-07-29T13:02:32Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1196: Maintenance 2442332 wikitext text/x-wiki lo458cw2352i2niuqcl339m3p4g0vmk 2442333 2442332 2026-07-29T13:02:52Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance 2442333 wikitext text/x-wiki i1hpq3n9zd3fa1bf42egzed2jupp1wy 2442334 2442333 2026-07-29T13:03:00Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1206 (T431660)', diff saved to https://phabricator.wikimedia.org/P95562 and previous config saved to /var/cache/conftool/dbconfig/20260729-130258-cwilliams.json 2442334 wikitext text/x-wiki kfbbik269ovnbhbaod77qlue8h2ts7p 2442335 2442334 2026-07-29T13:05:27Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1159: Maintenance 2442335 wikitext text/x-wiki 9xthddgthaq62q4g2u3j6r5qrrw6dr1 2442336 2442335 2026-07-29T13:05:32Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2073.codfw.wmnet with OS trixie 2442336 wikitext text/x-wiki sw807zd09jgbidkyjum8kaeosam4gqk 2442337 2442336 2026-07-29T13:05:42Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt IPs new switches - cmooney@cumin1003" 2442337 wikitext text/x-wiki muc7mf5ysk3800vfj1yqzyp07bomqs5 2442338 2442337 2026-07-29T13:05:47Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance 2442338 wikitext text/x-wiki sp03revc3m7yemrmvy211orsgfvw3r5 2442339 2442338 2026-07-29T13:05:55Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1229: Maintenance 2442339 wikitext text/x-wiki bkb9aav75tnljmvyisgzk2b2dv86q5k 2442340 2442339 2026-07-29T13:06:01Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1077.eqiad.wmnet with OS trixie 2442340 wikitext text/x-wiki koc95zmhshh6913jvz4m3dcxsrmmlw5 2442341 2442340 2026-07-29T13:06:09Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2442341 wikitext text/x-wiki o0ph6lvuuqg0bezh9zmnsj1sivggp9v 2442342 2442341 2026-07-29T13:06:18Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1161 (T431660)', diff saved to https://phabricator.wikimedia.org/P95564 and previous config saved to /var/cache/conftool/dbconfig/20260729-130616-cwilliams.json 2442342 wikitext text/x-wiki eyle37z489c0y1hflcaizosa5w47w0d 2442343 2442342 2026-07-29T13:06:20Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet 2442343 wikitext text/x-wiki 0wq3l9sw51lo5t6zqfcciwu20wxl952 2442344 2442343 2026-07-29T13:07:02Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1158: Maintenance 2442344 wikitext text/x-wiki 37ad060zdlxy78cos2yrkpgplas5dsj 2442345 2442344 2026-07-29T13:07:19Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt IPs new switches - cmooney@cumin1003" 2442345 wikitext text/x-wiki e97mdf64galqujh7p0d6onq8y6jdo71 2442346 2442345 2026-07-29T13:07:21Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442346 wikitext text/x-wiki o67emi0e4kicqn56me464hntoohno0a 2442347 2442346 2026-07-29T13:07:23Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance 2442347 wikitext text/x-wiki l60bos36zpf9v51tvrrx06yts76hguy 2442348 2442347 2026-07-29T13:07:32Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1170 (T431660)', diff saved to https://phabricator.wikimedia.org/P95566 and previous config saved to /var/cache/conftool/dbconfig/20260729-130730-cwilliams.json 2442348 wikitext text/x-wiki 7vcoxd9c6qjqtlfiqyp0fokv5pzu2fn 2442349 2442348 2026-07-29T13:07:39Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1319075|remove throttle exceptions for concluded events]] 2442349 wikitext text/x-wiki fktjabsbawmiv64jtupg5v0xnj22n5d 2442350 2442349 2026-07-29T13:08:54Z Stashbot 7414 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service 2442350 wikitext text/x-wiki k8gzbg6i9y1xw9mougc92822739re54 2442351 2442350 2026-07-29T13:09:08Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1206: Maintenance 2442351 wikitext text/x-wiki nviqikvi8mhmzsvte3tsxndskivkmgb 2442352 2442351 2026-07-29T13:09:46Z Stashbot 7414 samtar@deploy1003: anzx, samtar: Backport for [[gerrit:1319075|remove throttle exceptions for concluded events]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442352 wikitext text/x-wiki dnr2iz9bw6na7rgy7l57ldges2pnnqk 2442353 2442352 2026-07-29T13:10:19Z Stashbot 7414 samtar@deploy1003: anzx, samtar: Continuing with deployment 2442353 wikitext text/x-wiki 1vj8mmokb3ls0ja3ymm3p4y3okm9gug 2442354 2442353 2026-07-29T13:12:13Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Maintenance 2442354 wikitext text/x-wiki fpz1us2yanrpkxp66jxv9py1l7pv7px 2442355 2442354 2026-07-29T13:12:51Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1223: Maintenance 2442355 wikitext text/x-wiki d7y5d7h5eeejqd653z4y7yl69f05l0t 2442356 2442355 2026-07-29T13:13:23Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet 2442356 wikitext text/x-wiki rzeclu9z86zsaokbzs4zprujx1748hs 2442357 2442356 2026-07-29T13:13:24Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet 2442357 wikitext text/x-wiki a4xmm262p78539ltmbx98e2mel9tjki 2442358 2442357 2026-07-29T13:13:28Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet 2442358 wikitext text/x-wiki l6rg6ni2x9fku5epgnei2lxkjvko3mc 2442359 2442358 2026-07-29T13:13:38Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1170: Maintenance 2442359 wikitext text/x-wiki qxc6be7fvqk6u9bxyyopqkkulkmjz5w 2442360 2442359 2026-07-29T13:14:39Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319075|remove throttle exceptions for concluded events]] (duration: 07m 00s) 2442360 wikitext text/x-wiki b3fd5pay7c6qoczo9goqtlgl7k2eivr 2442361 2442360 2026-07-29T13:16:11Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1242: Maintenance 2442361 wikitext text/x-wiki imyzt104vpia07flhj6zmbz94wf3z6y 2442362 2442361 2026-07-29T13:16:30Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.reboot-single for host conf2005.codfw.wmnet 2442362 wikitext text/x-wiki 5e3dbvehpqbkjjuot9chg9j3iyqojl4 2442363 2442362 2026-07-29T13:16:31Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance 2442363 wikitext text/x-wiki igu4w55flgw6pojzzm9b34y6fc2jcbs 2442364 2442363 2026-07-29T13:16:40Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1243 (T431660)', diff saved to https://phabricator.wikimedia.org/P95574 and previous config saved to /var/cache/conftool/dbconfig/20260729-131638-cwilliams.json 2442364 wikitext text/x-wiki nkq9jaiqfn652sydgmyntro3ziqr6e3 2442365 2442364 2026-07-29T13:18:14Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1309675|Enable the abuse filter block action on Hindi Wikipedia (T431830)]] 2442365 wikitext text/x-wiki pxseqs4dk3lqxnogrcx9pmgoab3hb6c 2442366 2442365 2026-07-29T13:18:18Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: url_downloader@eqiad 2442366 wikitext text/x-wiki c4tekdbeo2bvwqrm5ztjsdwcywysbc7 2442367 2442366 2026-07-29T13:18:30Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=97) for role: url_downloader@eqiad 2442367 wikitext text/x-wiki if6vsk2ryhvdogzq0g368bl71lts23s 2442368 2442367 2026-07-29T13:18:35Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet 2442368 wikitext text/x-wiki hvrjnbmitlck5477lp5aqfbs2q03kqy 2442369 2442368 2026-07-29T13:19:25Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-master[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" 2442369 wikitext text/x-wiki mralmlnre2tjad5622vqkem6iwmup81 2442370 2442369 2026-07-29T13:19:27Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2442370 wikitext text/x-wiki 7pgr30antyoukmihvplsninwde75hi4 2442371 2442370 2026-07-29T13:19:28Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts an-test-master[1001-1002].eqiad.wmnet 2442371 wikitext text/x-wiki nkbwxp5od6zidxypgvb103p2sugs7ro 2442373 2442371 2026-07-29T13:20:23Z Stashbot 7414 samtar@deploy1003: dreamrimmer, samtar: Backport for [[gerrit:1309675|Enable the abuse filter block action on Hindi Wikipedia (T431830)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442373 wikitext text/x-wiki 5ro7kl45otzx2vwwdc8arp53y44z6aw 2442374 2442373 2026-07-29T13:22:05Z Stashbot 7414 samtar@deploy1003: dreamrimmer, samtar: Continuing with deployment 2442374 wikitext text/x-wiki dcwe97qselg1peqrf21imwa74lznmlg 2442375 2442374 2026-07-29T13:22:07Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2442375 wikitext text/x-wiki t9xns1st152tu937m0nhhpvermszp33 2442376 2442375 2026-07-29T13:22:29Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2005.codfw.wmnet 2442376 wikitext text/x-wiki kcylka1j55hw40nx481f8utvvl016t7 2442377 2442376 2026-07-29T13:22:47Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2442377 wikitext text/x-wiki 81ljlkvtj228vd72v3v2e0xm7p98md7 2442378 2442377 2026-07-29T13:23:06Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1243: Maintenance 2442378 wikitext text/x-wiki l3anob1bzq0zk29cf9fpa940zotjf2f 2442379 2442378 2026-07-29T13:23:16Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie 2442379 wikitext text/x-wiki du0fiq8hlssw9o15lupng2ge5db6ozt 2442380 2442379 2026-07-29T13:23:36Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.move-vlan for host db1265 2442380 wikitext text/x-wiki p7846lcr5omrgb0f0icr5ate6wvcl8t 2442381 2442380 2026-07-29T13:23:38Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host db1265 2442381 wikitext text/x-wiki rgfeo0jd1rt2ppw6l8887q37wsrpgpg 2442382 2442381 2026-07-29T13:24:31Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1077.eqiad.wmnet with reason: host reimage 2442382 wikitext text/x-wiki ecxjelt5whdsm71hdxcb3l0pk2vzyqy 2442383 2442382 2026-07-29T13:24:35Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet 2442383 wikitext text/x-wiki 17uvb5jgmjs2d4i6yloo8voas7rhzux 2442384 2442383 2026-07-29T13:24:38Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet 2442384 wikitext text/x-wiki ps3szwr6r61zr9mao0beufnllhhqwx5 2442385 2442384 2026-07-29T13:24:40Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve101[2-5].eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2442385 wikitext text/x-wiki 5241pa5pufw6cg0csfk2zf3smevw4zn 2442386 2442385 2026-07-29T13:24:47Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: url_downloader@eqiad 2442386 wikitext text/x-wiki 3jgw20u0mo5e84b7wcxpncnuexf1xc7 2442387 2442386 2026-07-29T13:25:10Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet 2442387 wikitext text/x-wiki 9714cph982em6ifp8bgs113s842twlv 2442388 2442387 2026-07-29T13:26:11Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1309675|Enable the abuse filter block action on Hindi Wikipedia (T431830)]] (duration: 07m 56s) 2442388 wikitext text/x-wiki mjg2rtemgke56adepgwwsdkv6auzu18 2442389 2442388 2026-07-29T13:26:14Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2073.codfw.wmnet with reason: host reimage 2442389 wikitext text/x-wiki i0piwvijrtqsl7kwybxqnha8stgdif9 2442390 2442389 2026-07-29T13:27:09Z Stashbot 7414 sukhe@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: url_downloader@eqiad 2442390 wikitext text/x-wiki 9ykl8hurjghfuicyzxy1vrd64mpb9b6 2442391 2442390 2026-07-29T13:27:34Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2442391 wikitext text/x-wiki 6n2y4zii8s6r4vy25eilm8ylr7bzyfe 2442392 2442391 2026-07-29T13:27:43Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: url_downloader@eqiad 2442392 wikitext text/x-wiki lu0wl0xcklmw1qw16mod3e13j7fdhj2 2442393 2442392 2026-07-29T13:28:10Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1077.eqiad.wmnet with reason: host reimage 2442393 wikitext text/x-wiki pzsewyrrhwoimutqo6lg06psusf728u 2442394 2442393 2026-07-29T13:29:10Z Stashbot 7414 sukhe@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: url_downloader@eqiad 2442394 wikitext text/x-wiki aktrkd9ot1e0bwcg687fnt4sglr8fcl 2442395 2442394 2026-07-29T13:30:40Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1319079|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319080|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319077|SI: Instrument abuse filter hits link (T433053)]], [[gerrit:1319078|SI: Instrument abuse filter hits link (T433053)]] 2442395 wikitext text/x-wiki 39xw1s2lprgtn7sqqh6l1jvuye2fjuc 2442396 2442395 2026-07-29T13:30:53Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet 2442396 wikitext text/x-wiki ezjqsgd2nxuyz8ylh6fhpuwkpejp9hx 2442397 2442396 2026-07-29T13:32:01Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2073.codfw.wmnet with reason: host reimage 2442397 wikitext text/x-wiki e4odyc8x3d3w18ga2ryrgi2mr3ikvmu 2442398 2442397 2026-07-29T13:32:39Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1319079|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319080|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319077|SI: Instrument abuse filter hits link (T433053)]], [[gerrit:1319078|SI: Instrument abuse filter hits link (T433053)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified t 2442398 wikitext text/x-wiki islamcpzy83oa4vyuwbv6xsw88alwme 2442399 2442398 2026-07-29T13:33:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/kartotherian: apply 2442399 wikitext text/x-wiki kbjhc7rdd4cf5h1juskwdkui8bzskdf 2442400 2442399 2026-07-29T13:35:59Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2442400 wikitext text/x-wiki 47hi4fek0n4qfwypnwqhhw1jx65hsd1 2442401 2442400 2026-07-29T13:36:32Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage 2442401 wikitext text/x-wiki 033bewwpvclixbnpf0pzfm44sxcvpca 2442402 2442401 2026-07-29T13:38:18Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2442402 wikitext text/x-wiki hndht8l609k6jg8sfvi3yi0nj7um62c 2442403 2442402 2026-07-29T13:39:45Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2442403 wikitext text/x-wiki nxol87hk4i6nsfcgech2w1gblob14ll 2442404 2442403 2026-07-29T13:40:03Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319079|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319080|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319077|SI: Instrument abuse filter hits link (T433053)]], [[gerrit:1319078|SI: Instrument abuse filter hits link (T433053)]] (duration: 09m 22s) 2442404 wikitext text/x-wiki e7we99f7pnztduluujrjt7q58wyklj2 2442405 2442404 2026-07-29T13:44:03Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage 2442405 wikitext text/x-wiki e4eefvat2vizm42buyfisxmbbyoxd7d 2442406 2442405 2026-07-29T13:44:19Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply 2442406 wikitext text/x-wiki kdb09tpvf8hdgcopmrhy7jxrupicjjr 2442408 2442406 2026-07-29T13:46:50Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host conf2005.codfw.wmnet with OS bookworm 2442408 wikitext text/x-wiki 8fd4sr64g8fonbawoz98vcny91w3mts 2442409 2442408 2026-07-29T13:47:01Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/kartotherian: apply 2442409 wikitext text/x-wiki desp6q1t7h94hlrvyrzryri43t6wmqd 2442410 2442409 2026-07-29T13:47:10Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1077.eqiad.wmnet with OS trixie 2442410 wikitext text/x-wiki hj7fyf4a7pqckpssyncb0blubtv1sf2 2442411 2442410 2026-07-29T13:48:18Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply 2442411 wikitext text/x-wiki o8h6itgelww4dv4i8cmb00phy5h386f 2442413 2442411 2026-07-29T13:49:19Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply 2442413 wikitext text/x-wiki s0awc22aotfjksfpg1pr1icdfpe4x7x 2442414 2442413 2026-07-29T13:49:55Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2442414 wikitext text/x-wiki ekp5shf3jvca1bdtk00l7fyvbv5n2hb 2442415 2442414 2026-07-29T13:50:43Z Stashbot 7414 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service 2442415 wikitext text/x-wiki rbce93wqjqgree1wo6akqa8gu8iwqnf 2442416 2442415 2026-07-29T13:50:47Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply 2442416 wikitext text/x-wiki 95uigigybc83isbh9usklr77484u3yg 2442417 2442416 2026-07-29T13:53:08Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1229: Maintenance 2442417 wikitext text/x-wiki lmsr5m26ml2t4cvbsukh1e7pwxcz7qp 2442418 2442417 2026-07-29T13:53:28Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance 2442418 wikitext text/x-wiki k6umqskf9jxjp1fw3acj9ped77lukdl 2442419 2442418 2026-07-29T13:53:37Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1233 (T431660)', diff saved to https://phabricator.wikimedia.org/P95587 and previous config saved to /var/cache/conftool/dbconfig/20260729-135335-cwilliams.json 2442419 wikitext text/x-wiki 2099ty6wr72dfx6p41dg1upswdj6g81 2442420 2442419 2026-07-29T13:54:01Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2073.codfw.wmnet with OS trixie 2442420 wikitext text/x-wiki gbla9f3yh1979mvhqt83or5whfdb7mh 2442421 2442420 2026-07-29T13:55:53Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1206: Maintenance 2442421 wikitext text/x-wiki 2jll94ugxt68rp8wr5b6p7e31c1hfa2 2442422 2442421 2026-07-29T13:56:14Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance 2442422 wikitext text/x-wiki tjhgrdiz4xt2ylbk47jk8vxbxyr2mnn 2442423 2442422 2026-07-29T13:56:22Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1218 (T431660)', diff saved to https://phabricator.wikimedia.org/P95590 and previous config saved to /var/cache/conftool/dbconfig/20260729-135621-cwilliams.json 2442423 wikitext text/x-wiki 7ksvghniwilpo6aqnq2qw0ydp57vu6f 2442424 2442423 2026-07-29T13:58:22Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie 2442424 wikitext text/x-wiki 6c8a3lfetj6zizle7iln3p014e3cid9 2442425 2442424 2026-07-29T13:58:27Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=urldownloader 2442425 wikitext text/x-wiki 0qqom31lwb6vl1qb9fbik1ji9v6ko8f 2442426 2442425 2026-07-29T13:59:01Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Maintenance 2442426 wikitext text/x-wiki ayb54mawzk2gw9ccmodmz3nfr4hk997 2442427 2442426 2026-07-29T13:59:19Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance 2442427 wikitext text/x-wiki 4rwjp99bsiq0bd0v0bn5j7jyiklk42h 2442428 2442427 2026-07-29T13:59:27Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1185 (T431660)', diff saved to https://phabricator.wikimedia.org/P95592 and previous config saved to /var/cache/conftool/dbconfig/20260729-135925-cwilliams.json 2442428 wikitext text/x-wiki okle4vua8w04nt78magy0fp54hvxux6 2442429 2442428 2026-07-29T13:59:54Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1233: Maintenance 2442429 wikitext text/x-wiki ith9hkk8ivk0v61gyzvfrocaqzyb90r 2442430 2442429 2026-07-29T14:00:56Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2442430 wikitext text/x-wiki 54f3bcojrw5uv1x2wapl39s65tkmu3x 2442431 2442430 2026-07-29T14:01:02Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2442431 wikitext text/x-wiki iiv6oviqe10if8hwhsc7ptzrjivipvd 2442432 2442431 2026-07-29T14:02:30Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1218: Maintenance 2442432 wikitext text/x-wiki hfa1n1dhvnk0fclxhz4gpwq16yj31os 2442433 2442432 2026-07-29T14:02:41Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1170: Maintenance 2442433 wikitext text/x-wiki 5thnm015gsxdwjh05n7qzogdt3lzwhn 2442434 2442433 2026-07-29T14:03:03Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance 2442434 wikitext text/x-wiki 7dykwovg4q5ig2ke0khx561h4simd1z 2442435 2442434 2026-07-29T14:03:09Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2442435 wikitext text/x-wiki pffp1xprr3tocir2jq07p6wk1uh09qo 2442436 2442435 2026-07-29T14:03:11Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1174 (T431660)', diff saved to https://phabricator.wikimedia.org/P95595 and previous config saved to /var/cache/conftool/dbconfig/20260729-140309-cwilliams.json 2442436 wikitext text/x-wiki 6ru0uzzfkhf7rpsh2xltg51drbdjxfi 2442437 2442436 2026-07-29T14:05:52Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on conf2005.codfw.wmnet with reason: host reimage 2442437 wikitext text/x-wiki tag4u285cdnpoffmb8fusq91mpmgu19 2442438 2442437 2026-07-29T14:06:01Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: url_downloader@eqiad 2442438 wikitext text/x-wiki p24svtauim3bvv9xd0f82m11a7bmtnr 2442439 2442438 2026-07-29T14:06:28Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1185: Maintenance 2442439 wikitext text/x-wiki m0clv0vnxxu4gby24zrecmmsym3gf2f 2442440 2442439 2026-07-29T14:08:23Z Stashbot 7414 sukhe@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: url_downloader@eqiad 2442440 wikitext text/x-wiki np3bdtiavn0akoe2l2kfhm5lpxow1ur 2442441 2442440 2026-07-29T14:09:21Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1174: Maintenance 2442441 wikitext text/x-wiki ni8ofb45bkszasx6ll3r6i7hl10qshj 2442442 2442441 2026-07-29T14:09:42Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2005.codfw.wmnet with reason: host reimage 2442442 wikitext text/x-wiki 1k8qt58a08lwk17wp7nljenpc6sd8dc 2442443 2442442 2026-07-29T14:09:49Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1243: Maintenance 2442443 wikitext text/x-wiki f4d506gr05xzo1jqmnth1sicl5zgde3 2442444 2442443 2026-07-29T14:10:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.update-replication 2442444 wikitext text/x-wiki 39aud72m3wibj7hf2dwr18zua67okm6 2442445 2442444 2026-07-29T14:10:08Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.dns.netbox 2442445 wikitext text/x-wiki dyp204qaq7hz5h237f7ymu72n62hrwa 2442446 2442445 2026-07-29T14:10:09Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance 2442446 wikitext text/x-wiki 2hftxo1izjarte9j07upnxqn2h7mujk 2442447 2442446 2026-07-29T14:10:11Z Stashbot 7414 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) 2442447 wikitext text/x-wiki mpx1371d3h52k1qymw98qkgntadui93 2442448 2442447 2026-07-29T14:10:16Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1244 (T431660)', diff saved to https://phabricator.wikimedia.org/P95599 and previous config saved to /var/cache/conftool/dbconfig/20260729-141014-cwilliams.json 2442448 wikitext text/x-wiki qk37h11vqe8kxhuytwijo17sx9d8nv9 2442449 2442448 2026-07-29T14:12:44Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.update-replication 2442449 wikitext text/x-wiki ogkz38g0e1pogtruh4kt6z3ls3ceyo2 2442450 2442449 2026-07-29T14:12:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2442450 wikitext text/x-wiki r6d0pmgziwcl4v6ukyx9ol8d01bbtt5 2442451 2442450 2026-07-29T14:15:25Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add asw1-604 loopback ipv4 - pt1979@cumin2002" 2442451 wikitext text/x-wiki 8s08qmaoaqpcfd12frp1sp9ohbxpbaq 2442452 2442451 2026-07-29T14:15:30Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add asw1-604 loopback ipv4 - pt1979@cumin2002" 2442452 wikitext text/x-wiki eb9y9j7ijekhlrkaq37qyb4oit46d8y 2442453 2442452 2026-07-29T14:15:31Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442453 wikitext text/x-wiki 4n8qk0rjj1t2bc118imq6d61zfz04vd 2442454 2442453 2026-07-29T14:16:46Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1244: Maintenance 2442454 wikitext text/x-wiki f2wm953u3hsoe14s2bfmsjcy8j5u0sg 2442455 2442454 2026-07-29T14:17:36Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442455 wikitext text/x-wiki bsxa8b2i1nydcpztrxjtkdbuxsrxwzt 2442456 2442455 2026-07-29T14:21:08Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2442456 wikitext text/x-wiki 270kdp1s1epor5ojk82fmjjthf4uhik 2442457 2442456 2026-07-29T14:22:00Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2442457 wikitext text/x-wiki c6ln0ryoclfl7e0tyx7tahlfq4zllgn 2442458 2442457 2026-07-29T14:22:27Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2442458 wikitext text/x-wiki cew6lt66b8jmu3ltq5d19y0e2ueske0 2442459 2442458 2026-07-29T14:23:44Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2442459 wikitext text/x-wiki c528f75gimyoy1rwihdi060ys3ey5km 2442460 2442459 2026-07-29T14:24:19Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2442460 wikitext text/x-wiki cnpblahglk6svmzv3fezv039vgw0dvi 2442461 2442460 2026-07-29T14:24:36Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442461 wikitext text/x-wiki lyb9q89c6tle7iz81hig2d8noqy32yy 2442462 2442461 2026-07-29T14:24:49Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2442462 wikitext text/x-wiki lp2546befwlkd73tyxt2djbeyfxv5sn 2442463 2442462 2026-07-29T14:24:56Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2442463 wikitext text/x-wiki ea1nkkdc2cahtqhk3xef70wiol20tzd 2442464 2442463 2026-07-29T14:27:21Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.reimage for host db1285.eqiad.wmnet with OS trixie 2442464 wikitext text/x-wiki axe22vablyib749lnyr0cv6jdcenzku 2442465 2442464 2026-07-29T14:27:43Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.move-vlan for host db1285 2442465 wikitext text/x-wiki qrkrmhvrt6xk96mwzqgek602rawon3m 2442466 2442465 2026-07-29T14:27:44Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host db1285 2442466 wikitext text/x-wiki igc0fiyf3zlq370shxv0wl9neo8q8gk 2442467 2442466 2026-07-29T14:29:33Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2442467 wikitext text/x-wiki jpoq8cquzhbxc2tgba89y82ppelpnju 2442468 2442467 2026-07-29T14:29:58Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2442468 wikitext text/x-wiki 2vyo15ju81fwxfxsbv1olkbx7f81yyv 2442469 2442468 2026-07-29T14:30:13Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2442469 wikitext text/x-wiki cn4hp3rzrndid8s2arv5briqpkv20bk 2442470 2442469 2026-07-29T14:31:00Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2442470 wikitext text/x-wiki ntq61y3a92bnllkdseudyyjeaus5qq2 2442471 2442470 2026-07-29T14:31:08Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2005.codfw.wmnet with OS bookworm 2442471 wikitext text/x-wiki ddt2bxuncoiel41ldh1dt4d04ytopi1 2442472 2442471 2026-07-29T14:32:14Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2442472 wikitext text/x-wiki jpsb4mhmfs4ixq1vh7a8diuzci5u46u 2442473 2442472 2026-07-29T14:32:55Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2442473 wikitext text/x-wiki s7nvi0130t7bl609bhajvo6sliljc93 2442474 2442473 2026-07-29T14:32:59Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442474 wikitext text/x-wiki 4x1r5t3mgj8vfpp0pfs5a9k6roo333m 2442475 2442474 2026-07-29T14:39:22Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2074.codfw.wmnet with OS trixie 2442475 wikitext text/x-wiki qvm7tucbpzha0o2c2rkv1wmj0mzybly 2442476 2442475 2026-07-29T14:39:32Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1078.eqiad.wmnet with OS trixie 2442476 wikitext text/x-wiki ondjm4wp8lsazx7hklhoopkt53j6mjp 2442477 2442476 2026-07-29T14:41:00Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1285.eqiad.wmnet with reason: host reimage 2442477 wikitext text/x-wiki atpmgbppf22n593d3b1wfz2jd4trfqd 2442478 2442477 2026-07-29T14:41:48Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442478 wikitext text/x-wiki 7srom1btkuqvm9z5evch5h1fpqvje0k 2442479 2442478 2026-07-29T14:42:32Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442479 wikitext text/x-wiki hm950o70u7ug6fdoa7t7piu40p8sg0t 2442480 2442479 2026-07-29T14:43:06Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442480 wikitext text/x-wiki r8dj5er64a9oni707jni6txs9vjdmij 2442481 2442480 2026-07-29T14:44:28Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1285.eqiad.wmnet with reason: host reimage 2442481 wikitext text/x-wiki jzqklocn6g5g7io89sv6r7q6v558mwh 2442482 2442481 2026-07-29T14:45:55Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442482 wikitext text/x-wiki tmii40aj1l64adizpzt6ftdfygqcrcs 2442483 2442482 2026-07-29T14:46:16Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442483 wikitext text/x-wiki 8p0xe77clq35ihw9esedsd7pq0zmtbw 2442484 2442483 2026-07-29T14:46:48Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2014\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main 2442484 wikitext text/x-wiki ptmjt8itz7jnxnouo3y2mzw9x3n6l7r 2442485 2442484 2026-07-29T14:46:55Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2013\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main 2442485 wikitext text/x-wiki shps1tysbb3o6nqvu9e2wrsycb545j9 2442486 2442485 2026-07-29T14:47:00Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1233: Maintenance 2442486 wikitext text/x-wiki h9oo4uutkn0noafj5dck8n9cpu0tbmy 2442487 2442486 2026-07-29T14:47:23Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance 2442487 wikitext text/x-wiki 4v1f01actygdqjag653h9kai9i5lzvs 2442488 2442487 2026-07-29T14:47:31Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1254 (T431660)', diff saved to https://phabricator.wikimedia.org/P95613 and previous config saved to /var/cache/conftool/dbconfig/20260729-144729-cwilliams.json 2442488 wikitext text/x-wiki s6fe273pawss90mz0n5flt3r2phjchz 2442489 2442488 2026-07-29T14:48:22Z Stashbot 7414 dancy@deploy1003: Started deploy [zuul/deploy@22703a6]: Deploying https://gerrit.wikimedia.org/r/c/integration/zuul/+/1311501 (T432491) 2442489 wikitext text/x-wiki 6y8jjk75r16ksydqwpoj7nv624bcyqc 2442490 2442489 2026-07-29T14:48:26Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2015.codfw.wmnet with OS bookworm 2442490 wikitext text/x-wiki fdq27ctcprp5v22vj30i0bqu7nv58fz 2442491 2442490 2026-07-29T14:48:37Z Stashbot 7414 dancy@deploy1003: Finished deploy [zuul/deploy@22703a6]: Deploying https://gerrit.wikimedia.org/r/c/integration/zuul/+/1311501 (T432491) (duration: 00m 15s) 2442491 wikitext text/x-wiki se3wmmq1njslc1furn3uk7wn7r4rm5t 2442492 2442491 2026-07-29T14:48:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS bookworm 2442492 wikitext text/x-wiki 8oatj3axp9t4r4gh68sbfvedoxnfdum 2442493 2442492 2026-07-29T14:49:12Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Maintenance 2442493 wikitext text/x-wiki ha144q97bxan9dj326jcd3tjd59amuo 2442494 2442493 2026-07-29T14:49:32Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance 2442494 wikitext text/x-wiki n0futnics9vfhfm0ds0ainq9al1xg9q 2442495 2442494 2026-07-29T14:49:48Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1219 (T431660)', diff saved to https://phabricator.wikimedia.org/P95616 and previous config saved to /var/cache/conftool/dbconfig/20260729-144946-cwilliams.json 2442495 wikitext text/x-wiki a9f270o3l8ii71yvdxwj1a7saeo5w7x 2442496 2442495 2026-07-29T14:51:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2015 2442496 wikitext text/x-wiki 521e3mpujc5qqkxpev0sdf7kp5mz0h0 2442497 2442496 2026-07-29T14:51:49Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2015 2442497 wikitext text/x-wiki 9l7ixjq86hu19oumoefqhiurkclqnac 2442498 2442497 2026-07-29T14:52:19Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2021 2442498 wikitext text/x-wiki kxk0yi29bbe72qoc0f1khb744fc40nk 2442499 2442498 2026-07-29T14:53:10Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1185: Maintenance 2442499 wikitext text/x-wiki s0kgk24ijqx79w6c0cykud9bv09s10h 2442500 2442499 2026-07-29T14:53:30Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance 2442500 wikitext text/x-wiki p4x49hwyc3q8ld28n3wh7pq1i9k3yl2 2442501 2442500 2026-07-29T14:53:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2442501 wikitext text/x-wiki 5vvabha3svr9pe9idx7pl5ajqptv2e7 2442502 2442501 2026-07-29T14:53:36Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1254: Maintenance 2442502 wikitext text/x-wiki dy5flq4owm02mnxr9eyup6dklnrh0ep 2442503 2442502 2026-07-29T14:53:38Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1200 (T431660)', diff saved to https://phabricator.wikimedia.org/P95618 and previous config saved to /var/cache/conftool/dbconfig/20260729-145336-cwilliams.json 2442503 wikitext text/x-wiki 4p6wg2u56kihq6pmi1mtmhh0l8yvqo9 2442504 2442503 2026-07-29T14:55:33Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.wipe-cache url-downloader1005.wikimedia.org on all recursors 2442504 wikitext text/x-wiki jr5c54331fav2au2ijzwxtwf49hsako 2442505 2442504 2026-07-29T14:55:36Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) url-downloader1005.wikimedia.org on all recursors 2442505 wikitext text/x-wiki c9kdxfz20usatt0npziv7yz8c3mdpyy 2442506 2442505 2026-07-29T14:55:40Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.wipe-cache url-downloader1006.wikimedia.org on all recursors 2442506 wikitext text/x-wiki 0f7m15k46fmyemnxj48m4rcql9nvvo0 2442507 2442506 2026-07-29T14:55:43Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) url-downloader1006.wikimedia.org on all recursors 2442507 wikitext text/x-wiki t75u63tozgtfu8w0wiqdiivq5dsng91 2442508 2442507 2026-07-29T14:55:57Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1219: Maintenance 2442508 wikitext text/x-wiki 1wex6fc0gi7t6z5isg8zmr01zu0li0f 2442509 2442508 2026-07-29T14:56:02Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1174: Maintenance 2442509 wikitext text/x-wiki 5pbk4iuetua0uas4eoekyqt6vihnd7t 2442510 2442509 2026-07-29T14:56:23Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance 2442510 wikitext text/x-wiki mpa5c0ab7sx2woqixmofgqfdvjnvdep 2442511 2442510 2026-07-29T14:56:31Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1181 (T431660)', diff saved to https://phabricator.wikimedia.org/P95621 and previous config saved to /var/cache/conftool/dbconfig/20260729-145629-cwilliams.json 2442511 wikitext text/x-wiki 2z0igibxdxlg004w7a5kbuubjgsxa5o 2442512 2442511 2026-07-29T14:58:05Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1078.eqiad.wmnet with reason: host reimage 2442512 wikitext text/x-wiki gzrlntu7smw0p92w85qp5se9610rjye 2442513 2442512 2026-07-29T14:58:07Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442513 wikitext text/x-wiki 14pfy1r2g13ya5sifv5c88b7m035wfm 2442514 2442513 2026-07-29T14:58:29Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2097'] 2442514 wikitext text/x-wiki npit545z3lk94hz5c3qyumxl8nxok10 2442515 2442514 2026-07-29T14:58:35Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ms-be2097'] 2442515 wikitext text/x-wiki axt22tcqvuf2c8jh6nxcqtgw8qcvdm1 2442516 2442515 2026-07-29T14:58:42Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2098'] 2442516 wikitext text/x-wiki 0zx9woybnlbdjinxypbmxf0tjrn60tf 2442517 2442516 2026-07-29T14:58:48Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ms-be2098'] 2442517 wikitext text/x-wiki aw1ac8u93x8rmvq4goyxt425f6nuur9 2442518 2442517 2026-07-29T14:58:53Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host urldownloader1005.wikimedia.org 2442518 wikitext text/x-wiki os469h7np9xd86ziqy119u92s3p8xqb 2442519 2442518 2026-07-29T14:59:13Z Stashbot 7414 Amir1: mwscript-k8s -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --wiki=commonswiki --key '360p.mpeg4.mov' --throttle --video --missing (T358266) 2442519 wikitext text/x-wiki q9drnacf5td2u11iyy7vezdl86hjeyd 2442520 2442519 2026-07-29T14:59:15Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1285.eqiad.wmnet with OS trixie 2442520 wikitext text/x-wiki 2971du7exjou2pkekrncoy79cqb5703 2442521 2442520 2026-07-29T14:59:33Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage 2442521 wikitext text/x-wiki 6335qtbta2tgf1yro38ogfcbfr1fz6v 2442522 2442521 2026-07-29T15:00:26Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1200: Maintenance 2442522 wikitext text/x-wiki 1cnw055k3aj6is7qx2bcrrzhzuns9u5 2442523 2442522 2026-07-29T15:00:58Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS bullseye 2442523 wikitext text/x-wiki 8eksad4f941somabqn0xs3umt1tkedy 2442524 2442523 2026-07-29T15:01:28Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2098.codfw.wmnet with OS bullseye 2442524 wikitext text/x-wiki nvakw4fwvjkbx9wmkucw1srtppd0glv 2442525 2442524 2026-07-29T15:02:44Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Maintenance 2442525 wikitext text/x-wiki c05fak4t1h5p72hqmxi1asavgxthlpl 2442526 2442525 2026-07-29T15:03:19Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1005.wikimedia.org 2442526 wikitext text/x-wiki 3nll8iib4h7gm9itofsgkz95ws0oxlo 2442527 2442526 2026-07-29T15:04:38Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1078.eqiad.wmnet with reason: host reimage 2442527 wikitext text/x-wiki 5c9xigvinucarexd67z3xp51enkncjy 2442528 2442527 2026-07-29T15:04:43Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1244: Maintenance 2442528 wikitext text/x-wiki jocuuoekyb1jnlsepv9x6wipyf4f0c0 2442529 2442528 2026-07-29T15:04:48Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442529 wikitext text/x-wiki 0stpmj71bncoq3fpxwmj12fe7ybunwq 2442530 2442529 2026-07-29T15:04:52Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance 2442530 wikitext text/x-wiki ksey801cs5m2goq5goqkc9xmg3tghfa 2442531 2442530 2026-07-29T15:05:00Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1247 (T431660)', diff saved to https://phabricator.wikimedia.org/P95625 and previous config saved to /var/cache/conftool/dbconfig/20260729-150459-cwilliams.json 2442531 wikitext text/x-wiki e7l8um76z14zoqeiiva8s096ikinxdv 2442532 2442531 2026-07-29T15:08:03Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage 2442532 wikitext text/x-wiki m37lprqazrp636a360ioq723vdqasdm 2442533 2442532 2026-07-29T15:10:44Z Stashbot 7414 klausman@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on ml-serve2002.codfw.wmnet with reason: T433476 2442533 wikitext text/x-wiki fke3cjb2md3nadtt7pwdut2kfbrm05j 2442534 2442533 2026-07-29T15:10:48Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2015.codfw.wmnet with reason: host reimage 2442534 wikitext text/x-wiki lrzsr0m6ue6qp4q8a77upyw2q8fok2d 2442535 2442534 2026-07-29T15:11:19Z Stashbot 7414 klausman@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on ml-serve2004.codfw.wmnet with reason: T433478 2442535 wikitext text/x-wiki jqo6qx0exuekl22qsx5bb38i41nlh9c 2442536 2442535 2026-07-29T15:11:34Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1247: Maintenance 2442536 wikitext text/x-wiki 3dn2wra7o59ccwg3r1f1ydm5dk6jh3t 2442537 2442536 2026-07-29T15:14:13Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2015.codfw.wmnet with reason: host reimage 2442537 wikitext text/x-wiki md9ug6ceo8v9i76suhe3dk4j36go3to 2442538 2442537 2026-07-29T15:14:48Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]] 2442538 wikitext text/x-wiki pp9nq1wznqchk6d2wh7e6dcql53c2so 2442539 2442538 2026-07-29T15:19:01Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2021 - bking@cumin2003" 2442539 wikitext text/x-wiki o82jswypsdxuat2u1uwbwbyidg10ebl 2442540 2442539 2026-07-29T15:19:05Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2021 - bking@cumin2003" 2442540 wikitext text/x-wiki h9rgakb6zdt7zkwvrmv9k8qx1h2l1df 2442541 2442540 2026-07-29T15:19:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442541 wikitext text/x-wiki 4g6vbnnwnihe4duiwm7vkw7vssf602y 2442543 2442541 2026-07-29T15:19:08Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2021.codfw.wmnet 210.48.192.10.in-addr.arpa 0.1.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2442543 wikitext text/x-wiki c3hchndk6vtxvf3bgxinb6h557khk4s 2442544 2442543 2026-07-29T15:19:10Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2021.codfw.wmnet 210.48.192.10.in-addr.arpa 0.1.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2442544 wikitext text/x-wiki jibk6i064409xu8gor6i0mna1bm8vcz 2442545 2442544 2026-07-29T15:19:13Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2021 2442545 wikitext text/x-wiki 63hdjb7imlhh1c02qsx1wuio43ujbei 2442546 2442545 2026-07-29T15:19:40Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2021 2442546 wikitext text/x-wiki d2dk3oacsyydbxp6xb8q6o5a0b6ewsh 2442547 2442546 2026-07-29T15:19:42Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2021 2442547 wikitext text/x-wiki onz3btc9vt8pyhc35c2x3gsk8uzntzs 2442548 2442547 2026-07-29T15:22:15Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.reboot-single for host conf2006.codfw.wmnet 2442548 wikitext text/x-wiki e69oo69c49l13squk4lg64u2od0pd07 2442549 2442548 2026-07-29T15:25:36Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2442549 wikitext text/x-wiki q4azn7jauyyme393zcwbcw217ky0w9v 2442550 2442549 2026-07-29T15:25:51Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2098.codfw.wmnet with reason: host reimage 2442550 wikitext text/x-wiki hudnr8xxv8lv2cc2k960skuzep1zuln 2442551 2442550 2026-07-29T15:26:03Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1078.eqiad.wmnet with OS trixie 2442551 wikitext text/x-wiki 1g9hej4x4hf48jwkf44l6adehrdwzki 2442553 2442551 2026-07-29T15:28:17Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2006.codfw.wmnet 2442553 wikitext text/x-wiki exi7qv0ji0x3bf8gej8sm757mauam6f 2442554 2442553 2026-07-29T15:28:37Z Stashbot 7414 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host ms-be2074.codfw.wmnet with OS trixie 2442554 wikitext text/x-wiki sjb94qe754myuw9mc2v25pn7uw7z5yx 2442555 2442554 2026-07-29T15:29:23Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2442555 wikitext text/x-wiki iwd5qkn6g25ceyuvjmw49utqrmtys96 2442556 2442555 2026-07-29T15:32:55Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442556 wikitext text/x-wiki 94bfnomrc2sy83tzms25tda4llq2h9t 2442557 2442556 2026-07-29T15:33:31Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2098.codfw.wmnet with reason: host reimage 2442557 wikitext text/x-wiki 4p5q0rxl65tx72gt93dwv5x0q6gym9i 2442558 2442557 2026-07-29T15:33:34Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2074.codfw.wmnet with OS trixie 2442558 wikitext text/x-wiki 49q6m3oifxsbwzve89tloimxwdtdd0f 2442559 2442558 2026-07-29T15:35:27Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment 2442559 wikitext text/x-wiki nqudv1v7e666byfu384hdb4g71sknz9 2442560 2442559 2026-07-29T15:36:22Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage 2442560 wikitext text/x-wiki kg3jj0p7bcnw5mfkykf94absirw2pha 2442561 2442560 2026-07-29T15:36:29Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host conf2006.codfw.wmnet with OS bookworm 2442561 wikitext text/x-wiki im52wep86jy0iot7p895hu1an625aws 2442562 2442561 2026-07-29T15:38:38Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage 2442562 wikitext text/x-wiki 0x46f503v4gags7qonm5dfw62mt12s1 2442563 2442562 2026-07-29T15:38:58Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2015.codfw.wmnet with OS bookworm 2442563 wikitext text/x-wiki k6d4itglr4vzfr22077ang7q9wiqj12 2442564 2442563 2026-07-29T15:40:41Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1254: Maintenance 2442564 wikitext text/x-wiki 3xfq4a8ak6x9a1774dqc6pwf55sms91 2442565 2442564 2026-07-29T15:41:01Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance 2442565 wikitext text/x-wiki qnsdw403az6gqi7hnmz8bsod5u4illn 2442566 2442565 2026-07-29T15:41:09Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1259 (T431660)', diff saved to https://phabricator.wikimedia.org/P95638 and previous config saved to /var/cache/conftool/dbconfig/20260729-154107-cwilliams.json 2442566 wikitext text/x-wiki kpf2cibto2xhohyoehimtqjpo6lpkq6 2442567 2442566 2026-07-29T15:41:27Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2442567 wikitext text/x-wiki eclr6cahx9fwgg7ewatdou7ui3lw3af 2442568 2442567 2026-07-29T15:41:34Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 18s) 2442568 wikitext text/x-wiki knwz2gq6mp6yt50bnexbbtqh86n5ac1 2442569 2442568 2026-07-29T15:42:25Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards 2442569 wikitext text/x-wiki pd47muicpvyn6hi6r2bqikl65yyzxoa 2442574 2442569 2026-07-29T15:43:02Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1219: Maintenance 2442574 wikitext text/x-wiki a0lrd9e1m82u6puuyxctw9lwda8pw9n 2442575 2442574 2026-07-29T15:43:23Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance 2442575 wikitext text/x-wiki s62rb2hnrshqfn4j3chjzwa59zsldnu 2442576 2442575 2026-07-29T15:43:32Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1232 (T431660)', diff saved to https://phabricator.wikimedia.org/P95640 and previous config saved to /var/cache/conftool/dbconfig/20260729-154330-cwilliams.json 2442576 wikitext text/x-wiki 1w5w30g3j5hln82t7zxig93njhg2bho 2442578 2442576 2026-07-29T15:43:59Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage 2442578 wikitext text/x-wiki 6xitdslq8nynxjwavdddv803iy4gijx 2442579 2442578 2026-07-29T15:46:48Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]] (duration: 31m 59s) 2442579 wikitext text/x-wiki qyljcvrqbb9o3pt0sbocguks6vhgl3v 2442580 2442579 2026-07-29T15:47:09Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1200: Maintenance 2442580 wikitext text/x-wiki 82bxh03708h0kfme6m301dcbzmaaodg 2442581 2442580 2026-07-29T15:47:29Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance 2442581 wikitext text/x-wiki 0fbiskiv04fjayv58f5v0j4l7nf8n70 2442582 2442581 2026-07-29T15:47:31Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1259: Maintenance 2442582 wikitext text/x-wiki gqoj6kl8v2crpehxbz6hnvph3t5w33y 2442583 2442582 2026-07-29T15:47:37Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1207 (T431660)', diff saved to https://phabricator.wikimedia.org/P95643 and previous config saved to /var/cache/conftool/dbconfig/20260729-154735-cwilliams.json 2442583 wikitext text/x-wiki r3mvr44r62vh7o4ctxinhrk3mjy4ykj 2442584 2442583 2026-07-29T15:48:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage 2442584 wikitext text/x-wiki iy1i1sx60j3l78cbruw3qzcn0v5auhn 2442586 2442584 2026-07-29T15:49:50Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1232: Maintenance 2442586 wikitext text/x-wiki rf7th6qj9nxfsp82lnk9m0fijrqzzbz 2442587 2442586 2026-07-29T15:50:39Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Maintenance 2442587 wikitext text/x-wiki sh8u1juaq1l2f9odoxpsvmnfjyhghn1 2442588 2442587 2026-07-29T15:50:59Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance 2442588 wikitext text/x-wiki nipxm4xwqzs1rio22hz8crsnj1akksz 2442589 2442588 2026-07-29T15:51:06Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1191 (T431660)', diff saved to https://phabricator.wikimedia.org/P95646 and previous config saved to /var/cache/conftool/dbconfig/20260729-155104-cwilliams.json 2442589 wikitext text/x-wiki kq3gy5crcpq4l92sa3j9sewteywjita 2442590 2442589 2026-07-29T15:53:17Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1207: Maintenance 2442590 wikitext text/x-wiki jozkboqg7jj1jursbkg5lsey13p2j68 2442591 2442590 2026-07-29T15:55:37Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on conf2006.codfw.wmnet with reason: host reimage 2442591 wikitext text/x-wiki 8066my1v2iyl0s0ruddzd2uqjrvdtot 2442592 2442591 2026-07-29T15:55:58Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1079.eqiad.wmnet with OS trixie 2442592 wikitext text/x-wiki hmynzittlpzqkjd3pt9afyzgw0rsp9j 2442593 2442592 2026-07-29T15:57:06Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2442593 wikitext text/x-wiki gk5c0kek6b5lrdjcfbg2uas0iw6465l 2442594 2442593 2026-07-29T15:57:13Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1191: Maintenance 2442594 wikitext text/x-wiki p0317yacx0fi0minr0o1zdknp177i6y 2442595 2442594 2026-07-29T15:57:37Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2074.codfw.wmnet with OS trixie 2442595 wikitext text/x-wiki rop8ic645mommcheti00yhmu6spby6w 2442596 2442595 2026-07-29T15:59:21Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2442596 wikitext text/x-wiki 15jsrebndl2zql8qlcekq8mrsy4k6f0 2442597 2442596 2026-07-29T15:59:30Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1247: Maintenance 2442597 wikitext text/x-wiki f992qbe9b6h157tgrsvv7qn89wmjkmf 2442598 2442597 2026-07-29T15:59:39Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2006.codfw.wmnet with reason: host reimage 2442598 wikitext text/x-wiki okvpdrki09bnyp7l21qx9oj3v5lbm70 2442599 2442598 2026-07-29T15:59:49Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance 2442599 wikitext text/x-wiki pbycrml1ibgh3c0trvdjbm2tgi66r45 2442600 2442599 2026-07-29T15:59:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1248 (T431660)', diff saved to https://phabricator.wikimedia.org/P95651 and previous config saved to /var/cache/conftool/dbconfig/20260729-155956-cwilliams.json 2442600 wikitext text/x-wiki 45xzf40f80onp7lrx5p8zht9ljm5wft 2442601 2442600 2026-07-29T16:00:28Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2442601 wikitext text/x-wiki m3ylmt5s6qj2evf19cng5qksbavd4fx 2442603 2442601 2026-07-29T16:00:51Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2442603 wikitext text/x-wiki 93cs8ffrcybecvxht87dx2gle74ig1c 2442604 2442603 2026-07-29T16:01:32Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]] 2442604 wikitext text/x-wiki kwt9v1zsarx0aj3j8umugt0yqp46wgj 2442607 2442604 2026-07-29T16:06:29Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1248: Maintenance 2442607 wikitext text/x-wiki 143sgafv65wldq2npf2yuhvyln70dv8 2442608 2442607 2026-07-29T16:07:28Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442608 wikitext text/x-wiki e4jzplhbvy5urcuh3tu6ypbubssg9oo 2442609 2442608 2026-07-29T16:10:36Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment 2442609 wikitext text/x-wiki bui0u4ktoctuxewlrikqp4yx8wq3y5r 2442610 2442609 2026-07-29T16:12:24Z Stashbot 7414 topranks: hot-swap line card in FPC0 on cr1-eqiad with replacement MPC10E from Juniper T426343 2442610 wikitext text/x-wiki 8azty4b9mr7x0d06o44pmwfzuzgunx6 2442611 2442610 2026-07-29T16:14:24Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1079.eqiad.wmnet with reason: host reimage 2442611 wikitext text/x-wiki ovovsjkif05kcsw9tbwhw4z3y8xvfmo 2442613 2442611 2026-07-29T16:15:54Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2021.codfw.wmnet with OS bookworm 2442613 wikitext text/x-wiki dtt3c4beu4pwu7c62n880a9li7zhkvn 2442614 2442613 2026-07-29T16:16:50Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]] (duration: 15m 36s) 2442614 wikitext text/x-wiki qsbjm23zm2v0x6vjgsk4csgrtytgym9 2442615 2442614 2026-07-29T16:18:20Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1079.eqiad.wmnet with reason: host reimage 2442615 wikitext text/x-wiki 6tfch2cznxlsnhgtd26wb2rsyw30kbm 2442616 2442615 2026-07-29T16:19:41Z Stashbot 7414 akhatun: Deploying Refinery at 56695674 as part of weekly train 2442616 wikitext text/x-wiki 5xz9tgaqxjzwd7e94ill2kjhiz6u7vv 2442617 2442616 2026-07-29T16:20:54Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2006.codfw.wmnet with OS bookworm 2442617 wikitext text/x-wiki qbm1jhrooi6l8u1aa8wvml2t5tg5ut9 2442618 2442617 2026-07-29T16:21:14Z Stashbot 7414 akhatun@deploy1003: Started deploy [analytics/refinery@5669567] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@56695674] 2442618 wikitext text/x-wiki 5cw7913ngre6ncfwsnhsemnh21tufum 2442619 2442618 2026-07-29T16:23:44Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1051 hosts 2442619 wikitext text/x-wiki tv4s88gbend0ua0ekxclqkw6i3xe3w0 2442620 2442619 2026-07-29T16:26:08Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]] 2442620 wikitext text/x-wiki jauwhxhpdjsniiwyuk0s5w1uw24pv4j 2442621 2442620 2026-07-29T16:28:08Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442621 wikitext text/x-wiki pjfqlk29uyzwal5onikqhv6zbfzdj5v 2442622 2442621 2026-07-29T16:28:12Z Stashbot 7414 akhatun@deploy1003: Finished deploy [analytics/refinery@5669567] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@56695674] (duration: 06m 57s) 2442622 wikitext text/x-wiki j727vb9wl0v2eqf05e99s9uk47yl2zk 2442623 2442622 2026-07-29T16:34:14Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1259: Maintenance 2442623 wikitext text/x-wiki 4f80l07kmd06h4g1piuniw4ndgs3evr 2442624 2442623 2026-07-29T16:36:54Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1232: Maintenance 2442624 wikitext text/x-wiki 96k9su9lkhb1r4zpyks33ipvpn4sle8 2442625 2442624 2026-07-29T16:36:57Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1079.eqiad.wmnet with OS trixie 2442625 wikitext text/x-wiki mu0ep1o6u4kam64zo7mbvor4zdmlpxj 2442626 2442625 2026-07-29T16:37:13Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance 2442626 wikitext text/x-wiki qs0r2gz0px3ufg1ayxbw1jb3ou4ogmn 2442627 2442626 2026-07-29T16:37:21Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1234 (T431660)', diff saved to https://phabricator.wikimedia.org/P95664 and previous config saved to /var/cache/conftool/dbconfig/20260729-163719-cwilliams.json 2442627 wikitext text/x-wiki ey94ogrhoktmamp18cqi1jexj966qae 2442628 2442627 2026-07-29T16:37:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-test-master1004.eqiad.wmnet 2442628 wikitext text/x-wiki eyi7zdidak8xlctwyrg3lsg36e0um5m 2442629 2442628 2026-07-29T16:38:58Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2442629 wikitext text/x-wiki dpomll9td9w4g532128ork4aw3n6aqs 2442630 2442629 2026-07-29T16:39:02Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1207: Maintenance 2442630 wikitext text/x-wiki rorh991vlbuju3agl14segypspo38kh 2442631 2442630 2026-07-29T16:39:25Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance 2442631 wikitext text/x-wiki oypb6ti48zdg4dl0s31ur2kln5269pd 2442632 2442631 2026-07-29T16:39:31Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2442632 wikitext text/x-wiki 8ulw05off9r3mjhvmirzndri6edexib 2442633 2442632 2026-07-29T16:39:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1230 (T431660)', diff saved to https://phabricator.wikimedia.org/P95667 and previous config saved to /var/cache/conftool/dbconfig/20260729-163932-cwilliams.json 2442633 wikitext text/x-wiki t664ggc4qhju1tzwpf494ld703m02eo 2442634 2442633 2026-07-29T16:39:58Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2442634 wikitext text/x-wiki pzotplxg6iodn4t857x2jzwif20m3uv 2442635 2442634 2026-07-29T16:40:06Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2442635 wikitext text/x-wiki 7dkhi8otsbbzu1lcchnf8b6fse6mk95 2442636 2442635 2026-07-29T16:40:30Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2442636 wikitext text/x-wiki 924zjhtjt1r3l5baz257amu5ozozbm1 2442637 2442636 2026-07-29T16:41:20Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2442637 wikitext text/x-wiki jzg3rwu25dxcr9r7e4dr4ly0ujsy512 2442639 2442637 2026-07-29T16:43:12Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-master1004.eqiad.wmnet 2442639 wikitext text/x-wiki i0g75852e98oob0kyn0dlzcq255oijm 2442640 2442639 2026-07-29T16:43:16Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Rolling back deployment 2442640 wikitext text/x-wiki 56za6tvtlul9ms2olraekvyzdg2j56v 2442641 2442640 2026-07-29T16:43:26Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2442641 wikitext text/x-wiki rwgdzof2ofvodgabwwrf9gkw9sfv8hd 2442643 2442641 2026-07-29T16:43:43Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1234: Maintenance 2442643 wikitext text/x-wiki 3ltggk678lwamau0f2f0ujazfduwlrb 2442644 2442643 2026-07-29T16:43:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-test-master1003.eqiad.wmnet 2442644 wikitext text/x-wiki nh557ejqtrtl8jc5nf8jolh1y7lxefn 2442646 2442644 2026-07-29T16:43:55Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1191: Maintenance 2442646 wikitext text/x-wiki 6ja2x80eh7hi5cikz6a44sw8b8bf2ue 2442648 2442646 2026-07-29T16:44:16Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance 2442648 wikitext text/x-wiki i8rtk235voj32f8illa3p0jdimcuafw 2442649 2442648 2026-07-29T16:44:18Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2442649 wikitext text/x-wiki toc8k5w6396tywg13yyktcobo7y053p 2442650 2442649 2026-07-29T16:44:24Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1194 (T431660)', diff saved to https://phabricator.wikimedia.org/P95669 and previous config saved to /var/cache/conftool/dbconfig/20260729-164422-cwilliams.json 2442650 wikitext text/x-wiki q5ew1q43smxqi4poad8lm7kbuu2nelk 2442651 2442650 2026-07-29T16:45:15Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1230: Maintenance 2442651 wikitext text/x-wiki 3afq4q4ywedvzrwyuea2sq3epjl9hun 2442652 2442651 2026-07-29T16:46:06Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2442652 wikitext text/x-wiki khlbuqen8e48u3x4d2w5vm6tl351ngz 2442653 2442652 2026-07-29T16:46:45Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS bullseye 2442653 wikitext text/x-wiki 89ia691aizhv3lzbxcucyy1g0gihcax 2442654 2442653 2026-07-29T16:46:47Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2442654 wikitext text/x-wiki th8jlmqs0zvrtftpxus0ytxq1iy1i0x 2442655 2442654 2026-07-29T16:47:50Z Stashbot 7414 swfrench-wmf: silenced EtcdReplicationDown 57b2b421-1cc9-4e38-9276-94f223fd231c - T428495 2442655 wikitext text/x-wiki s6ledevxavrmlbiqt9x1yhy6zvqlaz5 2442656 2442655 2026-07-29T16:50:29Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1194: Maintenance 2442656 wikitext text/x-wiki pgm7uml0gjq36uq34y6bhpwcm3sl6ii 2442657 2442656 2026-07-29T16:53:12Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1248: Maintenance 2442657 wikitext text/x-wiki q2dzsgzztsiaxrejflp2w7ldwqrn3lh 2442658 2442657 2026-07-29T16:53:33Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1249.eqiad.wmnet with reason: Maintenance 2442658 wikitext text/x-wiki 249x9jsq317rsw410is4xn10buaswqt 2442659 2442658 2026-07-29T16:53:41Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1249 (T431660)', diff saved to https://phabricator.wikimedia.org/P95674 and previous config saved to /var/cache/conftool/dbconfig/20260729-165339-cwilliams.json 2442659 wikitext text/x-wiki 8nq3ly9pzpg2ypw55utidxakbhvf8ql 2442660 2442659 2026-07-29T16:54:34Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards 2442660 wikitext text/x-wiki mro9j8yyeqfwzoxrz6dry5v244ntan0 2442661 2442660 2026-07-29T16:59:54Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2442661 wikitext text/x-wiki 24pk6t8afhflmjtblz4s8m2wbf3fkku 2442662 2442661 2026-07-29T17:00:05Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1249: Maintenance 2442662 wikitext text/x-wiki 5bovmtqnci79dlvd9aaugzgoesz3xe5 2442663 2442662 2026-07-29T17:02:33Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]] (duration: 36m 25s) 2442663 wikitext text/x-wiki 01jdaar5ke9s88aeca88n2l4pu6uaon 2442664 2442663 2026-07-29T17:03:51Z Stashbot 7414 mutante: CI: contint1002/contint2002 - rm /etc/apache2/jenkins_proxy - removing legacy jenkins proxy config - jenkins is on new dedicated machines and uses jenkins_proxy_ext config T418521 2442664 wikitext text/x-wiki jegwjziwsk6ktsgqs946jcl2dmiq33y 2442665 2442664 2026-07-29T17:04:16Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2442665 wikitext text/x-wiki hp09ssi2wrtapqca4be7nhldauycg2g 2442666 2442665 2026-07-29T17:05:15Z Stashbot 7414 mutante: CI: contint1002/contint2002 - restarted httpd to be extra sure all is cleaned up - https://integration.wikimedia.org/ci/ is up and running T418521 2442666 wikitext text/x-wiki 4vv7k8eqv6chnpdew82e2d2k3tgyj80 2442667 2442666 2026-07-29T17:13:54Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Testing delay_messageblobstore_purge: true 2442667 wikitext text/x-wiki lp0mhjv8gsupqgo5tr7t1wwtrr0pqa8 2442668 2442667 2026-07-29T17:16:42Z Stashbot 7414 akhatun@deploy1003: Started deploy [analytics/refinery@5669567] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@56695674] 2442668 wikitext text/x-wiki jdvyn6gwe5amuhnnnoeim96dvbq0z5b 2442669 2442668 2026-07-29T17:17:05Z Stashbot 7414 akhatun@deploy1003: Finished deploy [analytics/refinery@5669567] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@56695674] (duration: 00m 22s) 2442669 wikitext text/x-wiki 8dufmqpplncfghjtsgj12etn2sj4dcm 2442670 2442669 2026-07-29T17:17:25Z Stashbot 7414 akhatun@deploy1003: Started deploy [analytics/refinery@5669567]: Regular analytics weekly train [analytics/refinery@56695674] 2442670 wikitext text/x-wiki ematxx7yjvia7g0fsv6g9graej5cmwu 2442673 2442670 2026-07-29T17:20:23Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Testing delay_messageblobstore_purge: true (duration: 06m 29s) 2442673 wikitext text/x-wiki 0h9jto35gv0obnc7nfhjzplrrxko4ei 2442674 2442673 2026-07-29T17:23:46Z Stashbot 7414 akhatun@deploy1003: Finished deploy [analytics/refinery@5669567]: Regular analytics weekly train [analytics/refinery@56695674] (duration: 06m 20s) 2442674 wikitext text/x-wiki ho2382ekpf98zhlh9pcmp0s2wmr8yvc 2442675 2442674 2026-07-29T17:24:28Z Stashbot 7414 akhatun@deploy1003: Started deploy [analytics/refinery@5669567] (thin): Regular analytics weekly train THIN [analytics/refinery@56695674] 2442675 wikitext text/x-wiki k5yj40h2f5z7doj972gjjd4xldy6rps 2442676 2442675 2026-07-29T17:26:31Z Stashbot 7414 akhatun@deploy1003: Finished deploy [analytics/refinery@5669567] (thin): Regular analytics weekly train THIN [analytics/refinery@56695674] (duration: 02m 02s) 2442676 wikitext text/x-wiki m9aljlai3vlshtj19517ysqpkvfp9it 2442677 2442676 2026-07-29T17:30:24Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1234: Maintenance 2442677 wikitext text/x-wiki 2uqov9igkzzbroddlmldyifsbrqshi3 2442678 2442677 2026-07-29T17:30:44Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance 2442678 wikitext text/x-wiki hni4ljcc5bpcoo846r4c823bgf23gq4 2442679 2442678 2026-07-29T17:30:52Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1235 (T431660)', diff saved to https://phabricator.wikimedia.org/P95684 and previous config saved to /var/cache/conftool/dbconfig/20260729-173051-cwilliams.json 2442679 wikitext text/x-wiki hrj1bo3b1na3adv3lnb6hs1ihlx5hzb 2442680 2442679 2026-07-29T17:31:15Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1230: Maintenance 2442680 wikitext text/x-wiki sbcx9y1lcj38035m389cdze4n5nfoq4 2442682 2442680 2026-07-29T17:37:15Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1235: Maintenance 2442682 wikitext text/x-wiki d295lurp0o9bn2bybi5a2btr48l46gm 2442683 2442682 2026-07-29T17:37:33Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1194: Maintenance 2442683 wikitext text/x-wiki d8y4s3wl04znswl0t37hqjwuufxrrrs 2442684 2442683 2026-07-29T17:37:52Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance 2442684 wikitext text/x-wiki 1i7150t7z0i3vup8eyx4f7c5958z3u0 2442685 2442684 2026-07-29T17:38:01Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1202 (T431660)', diff saved to https://phabricator.wikimedia.org/P95688 and previous config saved to /var/cache/conftool/dbconfig/20260729-173759-cwilliams.json 2442685 wikitext text/x-wiki 2fpm5y5bhf5o6axnibx47b8jk2l3jpr 2442686 2442685 2026-07-29T17:41:12Z Stashbot 7414 akhatun: Deployed refinery using scap, then deployed onto hdfs 2442686 wikitext text/x-wiki bd8wyibt530alq4y6te18qyra3vrtwb 2442687 2442686 2026-07-29T17:44:09Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1202: Maintenance 2442687 wikitext text/x-wiki 0z2u2dujb0cocdpikxdihze9bv016v3 2442688 2442687 2026-07-29T17:45:12Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2442688 wikitext text/x-wiki kejyd1zbxpw24niv70v3m6noy7s5x1z 2442689 2442688 2026-07-29T17:45:19Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2015\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main 2442689 wikitext text/x-wiki sxz0qrbwz0yw4vpw3agx2ge6vled4ak 2442690 2442689 2026-07-29T17:46:46Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1249: Maintenance 2442690 wikitext text/x-wiki 2xitvyknlrxwetz305rnd65r43w3pzq 2442691 2442690 2026-07-29T17:47:07Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance 2442691 wikitext text/x-wiki lr0p9cigir61i0podnq8y2ehqqgrqjn 2442692 2442691 2026-07-29T17:47:15Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95692 and previous config saved to /var/cache/conftool/dbconfig/20260729-174713-cwilliams.json 2442692 wikitext text/x-wiki myslogkl5ktszjimkn4mkblywibil7c 2442693 2442692 2026-07-29T17:47:28Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2442693 wikitext text/x-wiki 498nvin5s3abugmpctcub1swnsipmup 2442694 2442693 2026-07-29T17:47:31Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2442694 wikitext text/x-wiki 84p9kqr2aefuhjfcy3ws31xbytfg3gw 2442695 2442694 2026-07-29T17:47:34Z Stashbot 7414 swfrench-wmf: authdns-update to direct codfw, eqsin, ulsfo etcd clients back to codfw - T428495 2442695 wikitext text/x-wiki pqvymcv4gy206q0h2hqbjdsd2x0d2ye 2442696 2442695 2026-07-29T17:47:39Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 14s) 2442696 wikitext text/x-wiki nrriquazeicmnuh4xf761xlp73lgkz2 2442697 2442696 2026-07-29T17:47:52Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2021.codfw.wmnet, repooling source-only afterwards 2442697 wikitext text/x-wiki 2024amlarqhbeqdqd9eclk4gus6payz 2442698 2442697 2026-07-29T17:50:16Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-codfw and A:lvs (T428495) 2442698 wikitext text/x-wiki 98tmhrz4748e44ga9e6y45s4lsrhqhj 2442699 2442698 2026-07-29T17:50:43Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2022.codfw.wmnet with OS bookworm 2442699 wikitext text/x-wiki 6iu1ei14qz16ws5mtuo78u9us3z22nu 2442700 2442699 2026-07-29T17:54:12Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2022 2442700 wikitext text/x-wiki 3dq6jemilz4kbplccxbh8pt5v80z5ia 2442701 2442700 2026-07-29T17:55:42Z Stashbot 7414 swfrench-wmf: begin rolling restart of confd in codfw, eqsin, ulsfo - T428495 2442701 wikitext text/x-wiki bep89821lzg3jmlkq0rbvtbmti1eliy 2442702 2442701 2026-07-29T17:56:45Z Stashbot 7414 brett@cumin2002: END (FAIL) - Cookbook sre.loadbalancer.restart-pybal (exit_code=1) rolling-restart of pybal on A:lvs-codfw and A:lvs (T428495) 2442702 wikitext text/x-wiki 9j8fodkpr673akyzsw726clovni3nyh 2442704 2442702 2026-07-29T17:57:12Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2442704 wikitext text/x-wiki ed5bvoyiq7s19mk0cy5g6gzbmqoueg6 2442705 2442704 2026-07-29T18:02:49Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2022 - bking@cumin2003" 2442705 wikitext text/x-wiki 5f5c7uydiv0onjp8wxlysvy54mftwlq 2442706 2442705 2026-07-29T18:02:54Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2022 - bking@cumin2003" 2442706 wikitext text/x-wiki lnkavjxplxktgrxdsiyf9ys01ymvf1r 2442707 2442706 2026-07-29T18:02:55Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442707 wikitext text/x-wiki f5hv5dbbmzqvuvheglspl95uojur66b 2442708 2442707 2026-07-29T18:02:57Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2022.codfw.wmnet 211.48.192.10.in-addr.arpa 1.1.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2442708 wikitext text/x-wiki 9ycujkcd79c9b8qmt9z3m7fuwqh9va5 2442709 2442708 2026-07-29T18:02:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2022.codfw.wmnet 211.48.192.10.in-addr.arpa 1.1.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2442709 wikitext text/x-wiki ameatsn5tezscjz2j6e7hm496k311qk 2442710 2442709 2026-07-29T18:03:01Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2022 2442710 wikitext text/x-wiki iuj2speelstxjyl1qpzkbdexi9hb43n 2442711 2442710 2026-07-29T18:03:05Z Stashbot 7414 swfrench-wmf: restarted navtiming on webperf2003 - T428495 2442711 wikitext text/x-wiki tc50b1d2lg9cleeuckykzx67o5miecl 2442712 2442711 2026-07-29T18:08:51Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2022 2442712 wikitext text/x-wiki 4bg7wefjj6ejr4nfbvsleg27ndmklv5 2442713 2442712 2026-07-29T18:08:53Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2022 2442713 wikitext text/x-wiki 3zum4jc5g0e3ftk9v5xhw4d4f5otx2r 2442714 2442713 2026-07-29T18:13:13Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs (T428495) 2442714 wikitext text/x-wiki mjqmgpc6k0sxv791xi8gm2l5bo68gzc 2442715 2442714 2026-07-29T18:15:04Z Stashbot 7414 mutante: CI: contint1002/contint2002: apt-get remove --purge jenkins - jenkins be gone - T418521 2442715 wikitext text/x-wiki 2b8lc0fwnaxt7toz3zo1wri12r52hb5 2442716 2442715 2026-07-29T18:15:19Z Stashbot 7414 brett@cumin2002: END (ERROR) - Cookbook sre.loadbalancer.restart-pybal (exit_code=97) rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs (T428495) 2442716 wikitext text/x-wiki qfnyj4p2v8el8mfw1b7apmwcg5pa9mg 2442717 2442716 2026-07-29T18:16:18Z Stashbot 7414 mutante: removing jenkins during the train - living on the edge - no, just kidding, jenkins has migrated to dedicated machines, nothing should happen 2442717 wikitext text/x-wiki p42bh7ka2tnrhssfn7damsv2yke2iol 2442718 2442717 2026-07-29T18:17:22Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.13 refs T430832 2442718 wikitext text/x-wiki 8f48jj09ix5msbt4gagl8unev2erbxj 2442719 2442718 2026-07-29T18:17:35Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T428495) 2442719 wikitext text/x-wiki l5g4964zk0kb2558709dpv7bscsjlfs 2442720 2442719 2026-07-29T18:19:15Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T428495) 2442720 wikitext text/x-wiki jc5vx5nqqpv5w2l98sr6s8zbz9ct73p 2442721 2442720 2026-07-29T18:19:30Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin (T428495) 2442721 wikitext text/x-wiki 8mt6dndw96eekv9lokd3dn5bhb049q5 2442722 2442721 2026-07-29T18:22:28Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin (T428495) 2442722 wikitext text/x-wiki 0g43s9jnk4kkuuaqtn40zn8ki3fjqzn 2442723 2442722 2026-07-29T18:23:57Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1235: Maintenance 2442723 wikitext text/x-wiki mw8s8tc34k3k8o4qg99iy6bbk8ivs85 2442724 2442723 2026-07-29T18:24:22Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance 2442724 wikitext text/x-wiki 6bdcyea0kmrxs4j8ouffdr0umpr3ok3 2442725 2442724 2026-07-29T18:24:27Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs2014.codfw.wmnet 2442725 wikitext text/x-wiki tgejtudhazk0hxebkyngd2gvcsnjz0h 2442726 2442725 2026-07-29T18:24:29Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2014.codfw.wmnet 2442726 wikitext text/x-wiki qgdiaxckccrdrdnpxzwxoxqe2l9ubii 2442727 2442726 2026-07-29T18:24:31Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1251 (T431660)', diff saved to https://phabricator.wikimedia.org/P95698 and previous config saved to /var/cache/conftool/dbconfig/20260729-182428-cwilliams.json 2442727 wikitext text/x-wiki bzv3soqwmqywmnrzaswfjplmux4mwy4 2442729 2442727 2026-07-29T18:27:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2022.codfw.wmnet with reason: host reimage 2442729 wikitext text/x-wiki s2eufij2kthwvffuzn700phldmg6eqt 2442730 2442729 2026-07-29T18:30:47Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1251: Maintenance 2442730 wikitext text/x-wiki t1kfopaoz1j807a2d629iqsvyngf7n1 2442731 2442730 2026-07-29T18:30:51Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1202: Maintenance 2442731 wikitext text/x-wiki nml4joajig2qoy9l0x1hvsnqcsxptae 2442732 2442731 2026-07-29T18:31:10Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance 2442732 wikitext text/x-wiki 3304clpux9f3vd5pl5rwrz5lejgb3fn 2442733 2442732 2026-07-29T18:31:19Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1227 (T431660)', diff saved to https://phabricator.wikimedia.org/P95701 and previous config saved to /var/cache/conftool/dbconfig/20260729-183117-cwilliams.json 2442733 wikitext text/x-wiki dx61xn6g0zejen3nq7cgrcvbw0n1cej 2442736 2442733 2026-07-29T18:34:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2022.codfw.wmnet with reason: host reimage 2442736 wikitext text/x-wiki hdfyztb62pyewhr1kanmdsehlzanrfu 2442739 2442736 2026-07-29T18:37:25Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1227: Maintenance 2442739 wikitext text/x-wiki qjgyxsovwgaatkcz617yydfu789wksa 2442742 2442739 2026-07-29T18:47:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95704 and previous config saved to /var/cache/conftool/dbconfig/20260729-184732-cwilliams.json 2442742 wikitext text/x-wiki 7fmtaeqmtvtbk4hh43yzfodvmllskk6 2442745 2442742 2026-07-29T18:57:42Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P95707 and previous config saved to /var/cache/conftool/dbconfig/20260729-185740-cwilliams.json 2442745 wikitext text/x-wiki eao2wfckl6f71bh60w61meajk07uudx 2442747 2442745 2026-07-29T19:01:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2021.codfw.wmnet, repooling source-only afterwards 2442747 wikitext text/x-wiki mi2s5t7uomxvpu99yrfyhqebn62w1xe 2442748 2442747 2026-07-29T19:01:18Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2022.codfw.wmnet with OS bookworm 2442748 wikitext text/x-wiki 4w1uq8bqd1ke08iu70w94xvdwgs6jyo 2442750 2442748 2026-07-29T19:07:50Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P95709 and previous config saved to /var/cache/conftool/dbconfig/20260729-190748-cwilliams.json 2442750 wikitext text/x-wiki somkoxnwsmwnaww3wrbfoayfjo7z7nz 2442752 2442750 2026-07-29T19:11:28Z Stashbot 7414 dduvall: rolling back wmf.13 to group0 due to T433457 (cc T430832) 2442752 wikitext text/x-wiki n1nstwes1wkzg6qhdlhj21c7ejr08fr 2442754 2442752 2026-07-29T19:16:53Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2442754 wikitext text/x-wiki kupexc3ozv3hrmp0i82n9hfr3qmblas 2442755 2442754 2026-07-29T19:17:58Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95711 and previous config saved to /var/cache/conftool/dbconfig/20260729-191756-cwilliams.json 2442755 wikitext text/x-wiki ivmgfyk6c9mwmur8p7dpnmsbiteo71w 2442756 2442755 2026-07-29T19:18:45Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1251: Maintenance 2442756 wikitext text/x-wiki f7tujoob2e2miujsc8w4kqh8r7aeoxd 2442758 2442756 2026-07-29T19:19:37Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.13 refs T430832 2442758 wikitext text/x-wiki cuamraxlrl6n8iye1yaos710thzrlbo 2442759 2442758 2026-07-29T19:21:16Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1048] - vriley@cumin1003" 2442759 wikitext text/x-wiki m8m3vl0n71wuo3v7nb1xmbkr67sqzu3 2442760 2442759 2026-07-29T19:21:21Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1048] - vriley@cumin1003" 2442760 wikitext text/x-wiki 7wu1wt5bayaiv8ilmztulovv857nhhj 2442761 2442760 2026-07-29T19:21:22Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442761 wikitext text/x-wiki nxxw4mepsukbdla21rthqox0z7qwhyw 2442762 2442761 2026-07-29T19:22:12Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 2442762 wikitext text/x-wiki fsoh5fqcoyyxvjuyp8hr9nkdaw129rc 2442763 2442762 2026-07-29T19:22:28Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1048 2442763 wikitext text/x-wiki 4p8zax88p5u054yvysh0h6tqa1j0ceh 2442764 2442763 2026-07-29T19:24:27Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1227: Maintenance 2442764 wikitext text/x-wiki earbpm4yl4q84xalajse2w8exu2enrk 2442765 2442764 2026-07-29T19:24:47Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance 2442765 wikitext text/x-wiki 6xc4glzyjjj0tm6j19rify0ts55hkom 2442766 2442765 2026-07-29T19:24:56Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1231 (T431660)', diff saved to https://phabricator.wikimedia.org/P95714 and previous config saved to /var/cache/conftool/dbconfig/20260729-192454-cwilliams.json 2442766 wikitext text/x-wiki q89hc5ev2u90f07emjrx9pydkeis4u1 2442767 2442766 2026-07-29T19:29:33Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1319159|Add throttle exemption for Black Cultural Archives UK (T433529)]] 2442767 wikitext text/x-wiki 37xvuf6swgmlbayawrf7x5wp9daaw52 2442768 2442767 2026-07-29T19:31:02Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1231: Maintenance 2442768 wikitext text/x-wiki i55ivw6cjlr6tv3w0576tt1jon9jdps 2442769 2442768 2026-07-29T19:31:33Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1319159|Add throttle exemption for Black Cultural Archives UK (T433529)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442769 wikitext text/x-wiki ry0mqyii00zbxi04a732znaw874sve7 2442770 2442769 2026-07-29T19:32:20Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with deployment 2442770 wikitext text/x-wiki 7kqgnkg1flm89eav9yo04lc22gq6mb2 2442772 2442770 2026-07-29T19:36:24Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319159|Add throttle exemption for Black Cultural Archives UK (T433529)]] (duration: 06m 49s) 2442772 wikitext text/x-wiki 9vpst9jc7kitxz1a1kld2vipq2fxkrb 2442773 2442772 2026-07-29T19:37:43Z Stashbot 7414 zabe: zabe@deploy1003:~$ mwscript-k8s --comment='T433529' --follow -- resetAuthenticationThrottle.php --wiki=aawiki --signup --ip=89.36.114.94 2442773 wikitext text/x-wiki 6jvc3iuada9mprmbxt80d0gkycdlwsn 2442774 2442773 2026-07-29T19:41:09Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442774 wikitext text/x-wiki 94bpxpup38wzhhdz6cmpb4e0o53etwh 2442775 2442774 2026-07-29T19:43:50Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS bullseye 2442775 wikitext text/x-wiki 1ixqf5dn3esifjjda2jtpurt6umo406 2442776 2442775 2026-07-29T19:47:25Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442776 wikitext text/x-wiki 7jl5uizum58ljbodkoaufbn6xgg19n1 2442777 2442776 2026-07-29T20:04:46Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2442777 wikitext text/x-wiki nxuise3d5ywcs22fg33arlpg77zhks0 2442778 2442777 2026-07-29T20:09:23Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2442778 wikitext text/x-wiki bla43jafweqix2zv98s8ci702fonqet 2442779 2442778 2026-07-29T20:09:39Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442779 wikitext text/x-wiki nei877a3wy4zz26qh5zo8tar3cuhs7g 2442780 2442779 2026-07-29T20:11:31Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442780 wikitext text/x-wiki 3ofroip7xt0ohp63i9aja5r8u5qofv3 2442781 2442780 2026-07-29T20:11:51Z Stashbot 7414 aaron@deploy1003: Started scap sync-world: Backport for [[gerrit:1306988|REST: use RestExternalModules config variable (T433314 T428375)]] 2442781 wikitext text/x-wiki cwif34mluv38kl0imhd9mg4jy5yc5vs 2442782 2442781 2026-07-29T20:12:25Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1048.eqiad.wmnet with OS trixie 2442782 wikitext text/x-wiki 35dj6zieknyb2w8ypyjs0cffnblxmi9 2442783 2442782 2026-07-29T20:13:50Z Stashbot 7414 aaron@deploy1003: bpirkle, aaron: Backport for [[gerrit:1306988|REST: use RestExternalModules config variable (T433314 T428375)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442783 wikitext text/x-wiki ruvri0wnl8nt2wqhyget78qe0d517xf 2442785 2442783 2026-07-29T20:15:49Z Stashbot 7414 aaron@deploy1003: bpirkle, aaron: Continuing with deployment 2442785 wikitext text/x-wiki pe6jvvgua973u77butduryf3y7fjbf2 2442787 2442785 2026-07-29T20:17:44Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1231: Maintenance 2442787 wikitext text/x-wiki oqh3qtd212ykehgz6w227gumck8izi7 2442788 2442787 2026-07-29T20:18:04Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance 2442788 wikitext text/x-wiki fbp8x68e81tarx3cca4hmyqu52k9plx 2442789 2442788 2026-07-29T20:18:12Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1253 (T431660)', diff saved to https://phabricator.wikimedia.org/P95719 and previous config saved to /var/cache/conftool/dbconfig/20260729-201810-cwilliams.json 2442789 wikitext text/x-wiki muvl7nlpk5vcbw88sgpz0bpvd2xzq8f 2442790 2442789 2026-07-29T20:19:58Z Stashbot 7414 aaron@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306988|REST: use RestExternalModules config variable (T433314 T428375)]] (duration: 08m 07s) 2442790 wikitext text/x-wiki hbqfpjd98yskr0hgapnbwtu66mrw5rc 2442791 2442790 2026-07-29T20:24:14Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1253: Maintenance 2442791 wikitext text/x-wiki 315weg8jwhbfoepy092h6rm2ldg35nl 2442793 2442791 2026-07-29T20:30:42Z Stashbot 7414 aaron@deploy1003: Started scap sync-world: Backport for [[gerrit:1312694|Use the openapi.json endpoint for the wikibase-rest/v1 REST module (T422405)]] 2442793 wikitext text/x-wiki 4xmkyv203lcs1fwomf3xx3dm7alna5u 2442794 2442793 2026-07-29T20:32:42Z Stashbot 7414 aaron@deploy1003: aaron: Backport for [[gerrit:1312694|Use the openapi.json endpoint for the wikibase-rest/v1 REST module (T422405)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442794 wikitext text/x-wiki 31n6v42qxk04nsuumgvpunwi6smgl1f 2442795 2442794 2026-07-29T20:48:15Z Stashbot 7414 aaron@deploy1003: aaron: Continuing with deployment 2442795 wikitext text/x-wiki a1h8pu9fep0facg5bj8l05q84pdm26o 2442796 2442795 2026-07-29T20:52:39Z Stashbot 7414 aaron@deploy1003: Finished scap sync-world: Backport for [[gerrit:1312694|Use the openapi.json endpoint for the wikibase-rest/v1 REST module (T422405)]] (duration: 21m 57s) 2442796 wikitext text/x-wiki h30ov6mlxz6o038sdzp2byx3lti6x2b 2442797 2442796 2026-07-29T20:59:37Z Stashbot 7414 aaron@deploy1003: Started scap sync-world: Backport for [[gerrit:1319175|Make the "wikibase-rest/v1" external module published (T422405)]] 2442797 wikitext text/x-wiki jj417hqc7vx9zm31p81ty4y0cbg8itn 2442798 2442797 2026-07-29T21:01:48Z Stashbot 7414 aaron@deploy1003: aaron: Backport for [[gerrit:1319175|Make the "wikibase-rest/v1" external module published (T422405)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442798 wikitext text/x-wiki d092ty76dsqzrno1wh2zflwhndeqwqq 2442799 2442798 2026-07-29T21:08:26Z Stashbot 7414 aaron@deploy1003: aaron: Continuing with deployment 2442799 wikitext text/x-wiki pbwvpw6ylbzfgybdhoxbfa3huj2ukcm 2442800 2442799 2026-07-29T21:10:58Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1253: Maintenance 2442800 wikitext text/x-wiki coe1aq48zjvd3vc35a15yrb5qxrcn65 2442801 2442800 2026-07-29T21:12:08Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2015\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main 2442801 wikitext text/x-wiki ho1fvidiaqjzibpggdladjdlkvh52dk 2442802 2442801 2026-07-29T21:12:30Z Stashbot 7414 aaron@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319175|Make the "wikibase-rest/v1" external module published (T422405)]] (duration: 12m 53s) 2442802 wikitext text/x-wiki pfifqdvvvjjx724jimhu2jxks3dlwi7 2442803 2442802 2026-07-29T21:12:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2022.codfw.wmnet, repooling source-only afterwards 2442803 wikitext text/x-wiki r7efzrx1o8sixvj8dj46euhs2n6uaaw 2442804 2442803 2026-07-29T21:16:11Z Stashbot 7414 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=metawiki 'Mental Health Resource Center' 'Safety Resource Center/Mental Health' Zabe --reason 'per request [[:phab:T433118|T433118]]' 2442804 wikitext text/x-wiki izh6mcffqnfw038cpy65ufgkad0uyqj 2442805 2442804 2026-07-29T21:25:02Z Stashbot 7414 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2097.codfw.wmnet with OS bullseye 2442805 wikitext text/x-wiki nu6zt5gkcv00fwr7zed4on3n6ht8fmk 2442806 2442805 2026-07-29T21:32:37Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1048.eqiad.wmnet with OS trixie 2442806 wikitext text/x-wiki eu36n3iyhyfz9t12wn151i939ykd65a 2442808 2442806 2026-07-29T21:58:05Z Stashbot 7414 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2442808 wikitext text/x-wiki 7690nr1pjmc750xuvyjizqdm8zo5dov 2442809 2442808 2026-07-29T21:58:26Z Stashbot 7414 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2442809 wikitext text/x-wiki ffegiia1j0zd1vxnaozvlst696cyiv0 2442810 2442809 2026-07-29T21:59:00Z Stashbot 7414 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2442810 wikitext text/x-wiki h5x71iv1egy0x0rrfryjo29xorqthvm 2442811 2442810 2026-07-29T21:59:39Z Stashbot 7414 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2442811 wikitext text/x-wiki bmrczuieh48lzocg4jeetta5ilt6328 2442812 2442811 2026-07-29T22:00:40Z Stashbot 7414 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2442812 wikitext text/x-wiki 5oji5vuitn4ntg70617p9vyllsdfpj9 2442813 2442812 2026-07-29T22:01:23Z Stashbot 7414 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2442813 wikitext text/x-wiki hy0hry87vhg6w2eywb66rspmc3a8cef 2442814 2442813 2026-07-29T22:20:27Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: Switch upgrade maintenance window, T433097] 2442814 wikitext text/x-wiki 5y9obnyxy4bjdsrpmyazkj2x63wo8on 2442815 2442814 2026-07-29T22:20:44Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: Switch upgrade maintenance window, T433097] 2442815 wikitext text/x-wiki jw3yfcbmeog1eet21duyn9au7sjcz6j 2442816 2442815 2026-07-29T22:25:42Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2022.codfw.wmnet, repooling source-only afterwards 2442816 wikitext text/x-wiki ibyoor8hcc516dnimyqvap9nz9uk7cb 2442817 2442816 2026-07-29T22:53:05Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2442817 wikitext text/x-wiki 72vpol1cvl8okucru37jqd9ekjw6p4x 2442818 2442817 2026-07-29T22:53:35Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2442818 wikitext text/x-wiki nwlric9l8emjr14eskrntit3kwhksgg 2442819 2442818 2026-07-29T22:53:36Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2442819 wikitext text/x-wiki ti57l2yhpimbofkxe2acxxs0392llas 2442820 2442819 2026-07-29T22:54:06Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2442820 wikitext text/x-wiki 0g2k8epcacqz6cixfwfnbm2n96oval7 2442821 2442820 2026-07-29T23:57:43Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mr1-eqsin,mr1-eqsin IPv6,mr1-eqsin.oob,mr1-eqsin.oob IPv6 with reason: connection issue 2442821 wikitext text/x-wiki 9aeoj4rsy5geagfxd03gollxwtyt36q 2442824 2442821 2026-07-30T01:38:31Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: Switch upgrade maintenance window complete, T433097] 2442824 wikitext text/x-wiki 490ttdovjhx2z1ti8ifzlmv6rbglmwy 2442825 2442824 2026-07-30T01:38:35Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: Switch upgrade maintenance window complete, T433097] 2442825 wikitext text/x-wiki feafesvmf509jp0v0lbb6wiijvy8x2a 2442826 2442825 2026-07-30T02:00:39Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2442826 wikitext text/x-wiki 8hcyfx265t43dhksdglsa00j2cjw70a 2442827 2442826 2026-07-30T02:07:17Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 37s) 2442827 wikitext text/x-wiki ebqg7514pe0soij9iiklrxfq76lq7f3 2442828 2442827 2026-07-30T04:03:31Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance 2442828 wikitext text/x-wiki h1cyfh2abr6soru40996mygla8eslns 2442829 2442828 2026-07-30T04:03:39Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95725 and previous config saved to /var/cache/conftool/dbconfig/20260730-040337-cwilliams.json 2442829 wikitext text/x-wiki k0dvv8ok1pfbx6y2mh8sax39hfdy8u3 2442831 2442829 2026-07-30T04:22:14Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.dns.netbox 2442831 wikitext text/x-wiki 27xbpnr9s8pi100iuv19lf7wfx4p7zo 2442832 2442831 2026-07-30T04:29:06Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance 2442832 wikitext text/x-wiki j1f2x1acp23b5rrm1n51vcnbe91aldh 2442833 2442832 2026-07-30T04:29:16Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2442833 wikitext text/x-wiki 89srqlo0srnuzeuyooic2gyfz85m5xv 2442834 2442833 2026-07-30T04:29:24Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1167 (T431660)', diff saved to https://phabricator.wikimedia.org/P95726 and previous config saved to /var/cache/conftool/dbconfig/20260730-042923-cwilliams.json 2442834 wikitext text/x-wiki om6eum2ttdm39is9pf46w2reuj0v2o7 2442836 2442834 2026-07-30T04:35:31Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Maintenance 2442836 wikitext text/x-wiki mal62kngxv21yzamlemzcxmpx76sts2 2442837 2442836 2026-07-30T04:47:35Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mr1 ge-0/0/3 ipv4 - pt1979@cumin2002" 2442837 wikitext text/x-wiki qnafuwjjrrjshw93bfdvy45rvyqcwmn 2442838 2442837 2026-07-30T04:47:40Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mr1 ge-0/0/3 ipv4 - pt1979@cumin2002" 2442838 wikitext text/x-wiki 6agb6wvblldchy7qisgsd49bsexjf8e 2442839 2442838 2026-07-30T04:47:42Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442839 wikitext text/x-wiki o1wvottk89yfqqbs0xc4g3gr6fid5rq 2442840 2442839 2026-07-30T05:04:00Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95729 and previous config saved to /var/cache/conftool/dbconfig/20260730-050358-cwilliams.json 2442840 wikitext text/x-wiki rguexrag5hngexnnxep9lb5xra75ldp 2442841 2442840 2026-07-30T05:14:08Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P95731 and previous config saved to /var/cache/conftool/dbconfig/20260730-051406-cwilliams.json 2442841 wikitext text/x-wiki kjtfnfcg21xqtysjxy721n6ug16o5kz 2442842 2442841 2026-07-30T05:23:26Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Maintenance 2442842 wikitext text/x-wiki era742rbimh2ophiy1dd03oucjflzj7 2442843 2442842 2026-07-30T05:23:58Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance 2442843 wikitext text/x-wiki ih4tpoosq00yy1cq3gfbzaegkq19wp0 2442844 2442843 2026-07-30T05:23:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1172 (T431660)', diff saved to https://phabricator.wikimedia.org/P95733 and previous config saved to /var/cache/conftool/dbconfig/20260730-052354-cwilliams.json 2442844 wikitext text/x-wiki s98m6rormg7cnynea8m1fmclc3eqldk 2442845 2442844 2026-07-30T05:24:16Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P95734 and previous config saved to /var/cache/conftool/dbconfig/20260730-052414-cwilliams.json 2442845 wikitext text/x-wiki hx2s99g54ojhlxk4hnkowka58nw9idu 2442846 2442845 2026-07-30T05:30:28Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Maintenance 2442846 wikitext text/x-wiki l017qekl5f6pqt489aaf6cprbn5lgu7 2442847 2442846 2026-07-30T05:34:24Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95736 and previous config saved to /var/cache/conftool/dbconfig/20260730-053422-cwilliams.json 2442847 wikitext text/x-wiki m0p2sxrzmxuie466aamywg9qxwc2zrr 2442848 2442847 2026-07-30T05:41:13Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1217 it crashed', diff saved to https://phabricator.wikimedia.org/P95737 and previous config saved to /var/cache/conftool/dbconfig/20260730-054111-marostegui.json 2442848 wikitext text/x-wiki msvmx3uznblhz3zkyc4wre5r7jit1w0 2442849 2442848 2026-07-30T05:44:02Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: crashed 2442849 wikitext text/x-wiki m08s7whldc7in5jrmuowa060rwnaoct 2442850 2442849 2026-07-30T06:17:10Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Maintenance 2442850 wikitext text/x-wiki flzfucp8xngatwneolnp40h834uhw3w 2442851 2442850 2026-07-30T06:17:30Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance 2442851 wikitext text/x-wiki o9qrhj3e1guag311x19egk5pwce1c2f 2442852 2442851 2026-07-30T06:17:38Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1177 (T431660)', diff saved to https://phabricator.wikimedia.org/P95741 and previous config saved to /var/cache/conftool/dbconfig/20260730-061736-cwilliams.json 2442852 wikitext text/x-wiki 7qvuzl2plavuf35iohhjd4zpjmjqagy 2442853 2442852 2026-07-30T06:24:04Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1177: Maintenance 2442853 wikitext text/x-wiki g7nikwchu9pe2mrbfsupb3ylftv32q0 2442854 2442853 2026-07-30T07:10:45Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1177: Maintenance 2442854 wikitext text/x-wiki i8n9ofwt5va5izadt6rt96ugfzu1729 2442855 2442854 2026-07-30T07:11:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance 2442855 wikitext text/x-wiki m62kbsnnlffzc84j49fwqwuf74lzmao 2442856 2442855 2026-07-30T07:11:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1178 (T431660)', diff saved to https://phabricator.wikimedia.org/P95746 and previous config saved to /var/cache/conftool/dbconfig/20260730-071112-cwilliams.json 2442856 wikitext text/x-wiki kidrz101bua9jv8dkxfhq4s0xalk05a 2442857 2442856 2026-07-30T07:17:27Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1178: Maintenance 2442857 wikitext text/x-wiki k38mljx9lomy0yavlyd8rilt533oq6d 2442858 2442857 2026-07-30T07:23:12Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host an-test-master1003.eqiad.wmnet 2442858 wikitext text/x-wiki 4693leq74snkwggydifqrbcncp5r9gw 2442859 2442858 2026-07-30T07:24:20Z Stashbot 7414 klausman@dns2004: START - running authdns-update 2442859 wikitext text/x-wiki qv75kewtcaeaqvag732oy7snxofdss4 2442860 2442859 2026-07-30T07:25:44Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2075.codfw.wmnet with OS trixie 2442860 wikitext text/x-wiki ovet23qmm6kc8ehr734tlkj67ryiec6 2442861 2442860 2026-07-30T07:25:59Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1080.eqiad.wmnet with OS trixie 2442861 wikitext text/x-wiki cwa06oo4mplbfijghxi4vjuiwdceeo9 2442862 2442861 2026-07-30T07:26:30Z Stashbot 7414 klausman@dns2004: END - running authdns-update 2442862 wikitext text/x-wiki kyz6qrsl2282agb76xs2sjn99txfn8v 2442864 2442862 2026-07-30T07:35:11Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1252', diff saved to https://phabricator.wikimedia.org/P95748 and previous config saved to /var/cache/conftool/dbconfig/20260730-073510-marostegui.json 2442864 wikitext text/x-wiki s8yqxlj96d916osrb41ludcz1sgkrra 2442865 2442864 2026-07-30T07:38:34Z Stashbot 7414 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply 2442865 wikitext text/x-wiki n2q4tzybvtvtxypf1eexuv7fjgwq29d 2442866 2442865 2026-07-30T07:38:42Z Stashbot 7414 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply 2442866 wikitext text/x-wiki 9otk6f1o605f6s0f7b5er9rvljgbzhl 2442867 2442866 2026-07-30T07:44:57Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance 2442867 wikitext text/x-wiki jomb6efx9yk9ujl52tdpa7zcdp89rf9 2442868 2442867 2026-07-30T07:46:35Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2075.codfw.wmnet with reason: host reimage 2442868 wikitext text/x-wiki 4ms43rr5v0fzlxoz5fewrnsazzeczgz 2442869 2442868 2026-07-30T07:50:42Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance 2442869 wikitext text/x-wiki nhxqsffpkhmu5h9x121eummucly6ogr 2442870 2442869 2026-07-30T07:50:44Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2075.codfw.wmnet with reason: host reimage 2442870 wikitext text/x-wiki 5n6e0pkz2kjxosggev58a12p8xkuxpx 2442871 2442870 2026-07-30T07:51:00Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[1260-1262].eqiad.wmnet with reason: Maintenance 2442871 wikitext text/x-wiki jh4tdnqroq6i2wlj0s1b028ux6scpfu 2442872 2442871 2026-07-30T07:51:08Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1263 (T431660)', diff saved to https://phabricator.wikimedia.org/P95751 and previous config saved to /var/cache/conftool/dbconfig/20260730-075106-cwilliams.json 2442872 wikitext text/x-wiki 86plfjmkw8pydjrnwttc3rzc1wvqgp3 2442873 2442872 2026-07-30T07:57:39Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1263: Maintenance 2442873 wikitext text/x-wiki p0ofl0xps4qz8rjrn6cigc7v2nnz7o0 2442874 2442873 2026-07-30T08:05:36Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1048.eqiad.wmnet with OS bullseye 2442874 wikitext text/x-wiki e5s564ajohoxnr3vq21kbvehfjtbcsr 2442875 2442874 2026-07-30T08:05:56Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1178: Maintenance 2442875 wikitext text/x-wiki shpu1l8jp3kcd7q1oo25d8q3x3rjn0v 2442876 2442875 2026-07-30T08:06:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance 2442876 wikitext text/x-wiki tttf04eqf79a9xt2b5wzevgvksyf6zm 2442877 2442876 2026-07-30T08:06:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1192 (T431660)', diff saved to https://phabricator.wikimedia.org/P95754 and previous config saved to /var/cache/conftool/dbconfig/20260730-080611-cwilliams.json 2442877 wikitext text/x-wiki ednsrp436fjja5aw74tddcv9dtzyg8d 2442878 2442877 2026-07-30T08:11:35Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1252: Maintenance 2442878 wikitext text/x-wiki g0xl8qikmzse3u23sbqwctkbkbwqhe1 2442879 2442878 2026-07-30T08:11:42Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1252: Maintenance 2442879 wikitext text/x-wiki i7s2kstihah5owgwn5plq5kbby0k3sz 2442880 2442879 2026-07-30T08:12:21Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1252: Maintenance 2442880 wikitext text/x-wiki 7wfxr3w6r14y4wh4pvukti3883oey0x 2442881 2442880 2026-07-30T08:13:12Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2075.codfw.wmnet with OS trixie 2442881 wikitext text/x-wiki g22nu514s9k4t8r9jvg0sz9xl64q2cf 2442882 2442881 2026-07-30T08:14:22Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1192: Maintenance 2442882 wikitext text/x-wiki g7v03i2kw2l7a2wgw3xc34sdzdlttyp 2442883 2442882 2026-07-30T08:15:58Z Stashbot 7414 mvernon@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1080.eqiad.wmnet with OS trixie 2442883 wikitext text/x-wiki quh11rem6wd7d86r9wzm9sno827yrlb 2442884 2442883 2026-07-30T08:17:22Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1080.eqiad.wmnet with OS trixie 2442884 wikitext text/x-wiki pv9g9o63rc0rjmaomu2hm244o3d9pnv 2442886 2442884 2026-07-30T08:23:40Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2442886 wikitext text/x-wiki s323yp329gi6es4k7pckv0u1fxko34m 2442887 2442886 2026-07-30T08:24:00Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2442887 wikitext text/x-wiki 4cdvr29odhmhks5vzr6d9s8h89zm790 2442888 2442887 2026-07-30T08:24:16Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2442888 wikitext text/x-wiki kut0dv65fridkz69h3l1y3ehx023txl 2442889 2442888 2026-07-30T08:24:46Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2442889 wikitext text/x-wiki lfyfq9omucfpt9zsew3113zd6kzkg1y 2442892 2442889 2026-07-30T08:44:43Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1263: Maintenance 2442892 wikitext text/x-wiki naiuhnkv3kw5a5uoma0bwlh28jhm1c0 2442893 2442892 2026-07-30T08:50:36Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2442893 wikitext text/x-wiki 70nph8nq9r212wh281i30z9js9nsk3d 2442894 2442893 2026-07-30T08:50:49Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2442894 wikitext text/x-wiki q9m6472l2ka9b1dwr9cvufm6zt5b0aw 2442895 2442894 2026-07-30T08:51:23Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442895 wikitext text/x-wiki 3p4t9e0xj6x3lyh32h6wi8pd5h384vb 2442896 2442895 2026-07-30T08:51:38Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2442896 wikitext text/x-wiki ko24r5fsikt1gp3jg7i2bqoah3hx04k 2442898 2442896 2026-07-30T08:52:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2442898 wikitext text/x-wiki 5i5c3gzuu9rl87ntnd87s14jhn36osg 2442899 2442898 2026-07-30T08:53:02Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 23 hosts 2442899 wikitext text/x-wiki skft3n9mw8y7ulm5oppygz4n6yxqkd9 2442900 2442899 2026-07-30T08:56:32Z Stashbot 7414 jayme@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2442900 wikitext text/x-wiki g44kd6qeinudcqzvpegkzd9474xuuju 2442901 2442900 2026-07-30T08:57:29Z Stashbot 7414 jayme@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2442901 wikitext text/x-wiki 2t5cu5ecxjrf48skjcrfihzp5d76sk9 2442902 2442901 2026-07-30T08:57:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1252: Maintenance 2442902 wikitext text/x-wiki 3cgih8sb5pf6u3f11v7kp6ha8yno03j 2442903 2442902 2026-07-30T09:01:07Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1192: Maintenance 2442903 wikitext text/x-wiki icl6jfog1s4j9sn7ec5271iamvn6fge 2442904 2442903 2026-07-30T09:01:27Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance 2442904 wikitext text/x-wiki qnxzqwhmcc2o0cocth12ssswnyztx1n 2442905 2442904 2026-07-30T09:01:35Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1209 (T431660)', diff saved to https://phabricator.wikimedia.org/P95767 and previous config saved to /var/cache/conftool/dbconfig/20260730-090133-cwilliams.json 2442905 wikitext text/x-wiki 2oj1ov9audtyyrqgllins1w40ol2cij 2442906 2442905 2026-07-30T09:02:19Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Remove cable label from interfaces descriptions - ayounsi@cumin1003 2442906 wikitext text/x-wiki dv111gu4pac2dls02pkb119ojuotz30 2442907 2442906 2026-07-30T09:04:28Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442907 wikitext text/x-wiki oubfdtt1trz6id1yfvvdijhaei5snsa 2442908 2442907 2026-07-30T09:04:44Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Remove cable label from interfaces descriptions - ayounsi@cumin1003 2442908 wikitext text/x-wiki 5m3e6kfftzpuz8kg9iiqxy8c5rczae0 2442909 2442908 2026-07-30T09:06:05Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 23 hosts 2442909 wikitext text/x-wiki 75c4pge68z0j0v1evujdg9l0xybqzrx 2442910 2442909 2026-07-30T09:07:55Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Maintenance 2442910 wikitext text/x-wiki orqo1p0rq4mur7q8m63aguo9cscd554 2442914 2442910 2026-07-30T09:17:56Z Stashbot 7414 mvernon@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1080.eqiad.wmnet with OS trixie 2442914 wikitext text/x-wiki 52kh2dvk5kp3lxj7tfcilfq6e87elku 2442915 2442914 2026-07-30T09:18:50Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1080.eqiad.wmnet with OS trixie 2442915 wikitext text/x-wiki 67ectw5bnfp830hxh2mugv4l967vbkj 2442917 2442915 2026-07-30T09:19:27Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2076.codfw.wmnet with OS trixie 2442917 wikitext text/x-wiki bedypjn1yyx2iegn3tljkfu6zlm3xtt 2442921 2442917 2026-07-30T09:29:07Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance 2442921 wikitext text/x-wiki ndp7nv3rpajpfhj8jpqhmzufq9hg1vh 2442922 2442921 2026-07-30T09:29:12Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es2035 (T431660)', diff saved to https://phabricator.wikimedia.org/P95771 and previous config saved to /var/cache/conftool/dbconfig/20260730-092910-cwilliams.json 2442922 wikitext text/x-wiki k5yv69pa5flpzwppexn54d6terkf9ix 2442924 2442922 2026-07-30T09:32:46Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2442924 wikitext text/x-wiki p25gtkxatg608n4vk7s4b4b1hvrdaxv 2442925 2442924 2026-07-30T09:32:55Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2442925 wikitext text/x-wiki sdinnllpfjri5ycfc6giipt1bmvaqoq 2442926 2442925 2026-07-30T09:34:40Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2035: Maintenance 2442926 wikitext text/x-wiki mw4b9xlnen002yax2wxtyn8ck6d5o7u 2442927 2442926 2026-07-30T09:35:16Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2442927 wikitext text/x-wiki bxdaw93729tkrrmlikgr7gb8hf5tl3s 2442928 2442927 2026-07-30T09:35:22Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2442928 wikitext text/x-wiki jldwmx4sofyck5kmkiu15hvbddl6b17 2442929 2442928 2026-07-30T09:35:29Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2442929 wikitext text/x-wiki g031gdknepnss2ifmdjfs9lsazze77t 2442930 2442929 2026-07-30T09:35:32Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2442930 wikitext text/x-wiki dgq7n70oud60s9xgp38pu9esiivdjua 2442931 2442930 2026-07-30T09:39:38Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2076.codfw.wmnet with reason: host reimage 2442931 wikitext text/x-wiki imov0fh1zdveei1b847dhjqtu9vgkn9 2442932 2442931 2026-07-30T09:39:40Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2442932 wikitext text/x-wiki ij38h93a5rklj6nopfid13un6c04bhi 2442933 2442932 2026-07-30T09:39:51Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary 2442933 wikitext text/x-wiki heyw6onlfz4vifrvthl121ss9d3bols 2442934 2442933 2026-07-30T09:40:15Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2035: Maintenance 2442934 wikitext text/x-wiki md593349fnbsuxjnw61l2k6c055kyso 2442935 2442934 2026-07-30T09:41:43Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox 2442935 wikitext text/x-wiki gnyaijgnnpknjeuxshnqmrfgaxieadi 2442936 2442935 2026-07-30T09:42:13Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox 2442936 wikitext text/x-wiki 131rdkwn6qcc5qfi2vt2msbs4ksl1eo 2442937 2442936 2026-07-30T09:42:15Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool es2035: Maintenance 2442937 wikitext text/x-wiki 7f22kd9syz49hdgorv48oc24hzbd9oi 2442938 2442937 2026-07-30T09:43:40Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2076.codfw.wmnet with reason: host reimage 2442938 wikitext text/x-wiki 5rla2zq7m2hg33thf2sn05b6yw6ahdh 2442940 2442938 2026-07-30T09:54:36Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Maintenance 2442940 wikitext text/x-wiki ig8z27e6yj9x931ki1tc9iw4vjdfhoo 2442941 2442940 2026-07-30T09:54:45Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance 2442941 wikitext text/x-wiki 9h4opx5e7emaxfuog9bxe5rjblz68h7 2442942 2442941 2026-07-30T09:54:53Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1214 (T431660)', diff saved to https://phabricator.wikimedia.org/P95775 and previous config saved to /var/cache/conftool/dbconfig/20260730-095451-cwilliams.json 2442942 wikitext text/x-wiki clj37uotby8vddwgidmocvtg12h0f91 2442943 2442942 2026-07-30T10:00:52Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1214: Maintenance 2442943 wikitext text/x-wiki szk8zkskgwre5xiau1bshl5sp8u01vv 2442946 2442943 2026-07-30T10:03:06Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2076.codfw.wmnet with OS trixie 2442946 wikitext text/x-wiki clei44yx170k61qz40koo0f9nvjexgo 2442948 2442946 2026-07-30T10:27:51Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2035: Maintenance 2442948 wikitext text/x-wiki 2aqdv9dni07q7v2oox9ag61eja366cq 2442949 2442948 2026-07-30T10:47:34Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1214: Maintenance 2442949 wikitext text/x-wiki l5nlnrojgft0dbpcpnbiki43xvh0774 2442950 2442949 2026-07-30T10:47:54Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance 2442950 wikitext text/x-wiki 7cks9nlxx3nr86q55wf9ppv2a7g75gj 2442951 2442950 2026-07-30T10:48:02Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1226 (T431660)', diff saved to https://phabricator.wikimedia.org/P95783 and previous config saved to /var/cache/conftool/dbconfig/20260730-104801-cwilliams.json 2442951 wikitext text/x-wiki 5it5q1mkqyxx5egmzncymm9v6azrpwo 2442952 2442951 2026-07-30T10:55:11Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1226: Maintenance 2442952 wikitext text/x-wiki psq7n3filc6ebd7irfjd9f6t0cxkiro 2442953 2442952 2026-07-30T11:01:15Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442953 wikitext text/x-wiki h79h6l1lcphfnl9gd03dhaallvm0axx 2442954 2442953 2026-07-30T11:03:28Z Stashbot 7414 mvernon@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1080.eqiad.wmnet with OS trixie 2442954 wikitext text/x-wiki 1mo24v1n4ntypfhe9eomv8gnmrupd70 2442955 2442954 2026-07-30T11:07:39Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1080.eqiad.wmnet with OS trixie 2442955 wikitext text/x-wiki 790eysk9vqyx1hg04wq4k4ebhwpwupu 2442956 2442955 2026-07-30T11:08:17Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2077.codfw.wmnet with OS trixie 2442956 wikitext text/x-wiki gjf6jyobxij7d6wb14z858fupxiod9a 2442957 2442956 2026-07-30T11:16:26Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance 2442957 wikitext text/x-wiki 4qjac16ojrw56rt9z3va4ilaiwx9p4q 2442958 2442957 2026-07-30T11:16:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es2036 (T431660)', diff saved to https://phabricator.wikimedia.org/P95786 and previous config saved to /var/cache/conftool/dbconfig/20260730-111633-cwilliams.json 2442958 wikitext text/x-wiki pg4gvwo6ncftqbxk91pvpftxgfmrjkc 2442959 2442958 2026-07-30T11:21:59Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2036: Maintenance 2442959 wikitext text/x-wiki opwhrxr337enqoaz07n2yz3tpzficrf 2442960 2442959 2026-07-30T11:24:27Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1080.eqiad.wmnet with reason: host reimage 2442960 wikitext text/x-wiki 7drvubc26ly7amzka66oo0c2vxn9888 2442961 2442960 2026-07-30T11:27:33Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2036: Maintenance 2442961 wikitext text/x-wiki 2z7yc2qkkzt0wb75fp9e6he1h5fdiqj 2442962 2442961 2026-07-30T11:27:38Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2036: Maintenance 2442962 wikitext text/x-wiki ksxsdwvznbsts6n8oidpmf4246k8dc4 2442963 2442962 2026-07-30T11:27:57Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1080.eqiad.wmnet with reason: host reimage 2442963 wikitext text/x-wiki 91ih387jluvuxqimocng3ah1i4a5gz5 2442964 2442963 2026-07-30T11:28:51Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2077.codfw.wmnet with reason: host reimage 2442964 wikitext text/x-wiki 5x7xiykd4vz7itqss28407n7r4axpie 2442966 2442964 2026-07-30T11:32:31Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2077.codfw.wmnet with reason: host reimage 2442966 wikitext text/x-wiki 5bdm5dns4eprpjzqhq8o6xag7cukbm7 2442969 2442966 2026-07-30T11:41:10Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442969 wikitext text/x-wiki drmyvzyvfqokuyvrgu6k8qqnb6tj7g4 2442970 2442969 2026-07-30T11:41:54Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1226: Maintenance 2442970 wikitext text/x-wiki g9wijdtrot8q2qlp5pektxhxodj1sgu 2442971 2442970 2026-07-30T11:46:59Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1080.eqiad.wmnet with OS trixie 2442971 wikitext text/x-wiki j21i5qwoy8oatzgz8iwg8766398r23w 2442972 2442971 2026-07-30T11:48:11Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442972 wikitext text/x-wiki h2f4c7fqh0lz1gjghifqtv8ojnkzi8a 2442973 2442972 2026-07-30T11:51:45Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2077.codfw.wmnet with OS trixie 2442973 wikitext text/x-wiki gvj5b85vd50ex3cbca07jwzte3rwjrk 0 17290 2442735 2440746 2026-07-29T18:33:28Z Stashbot 7414 dancy: Buildkit v0.32.0 deployed to gitlab-cloud-runners staging and production (T433520) 2442735 wikitext text/x-wiki hau96i5u1h81tbftw21nlfv5guba17l 2442807 2442735 2026-07-29T21:33:18Z Stashbot 7414 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/109 2442807 wikitext text/x-wiki ow36bi19um6dx6f6xvwdjxxgdzywo7z 498 25824 2442885 2083889 2026-07-30T08:17:47Z Jon (WMF) 42946 2442885 wikitext text/x-wiki b5f50z18i8nk6yhts9vgobxv48i5w31 2442890 2442885 2026-07-30T08:26:05Z Jon (WMF) 42946 2442890 wikitext text/x-wiki 4cbq3xmcbliqofbkfauc6vcj49pxvcj 10 54864 2442602 2428372 2026-07-29T16:00:44Z Taavi-WMF 41365 elastic->open 2442602 wikitext text/x-wiki 22trpsv396zyi2e1mk5kr0npzwv0ehz 0 76447 2442638 2429931 2026-07-29T16:42:36Z BCornwall-WMF 31521 /* Management */ Remove outdated docs for depooling a site 2442638 wikitext text/x-wiki c7uzj00o9whncj6f8nhz9ie4s6lalms 2442645 2442638 2026-07-29T16:43:49Z BCornwall-WMF 31521 /* Disabling a Site */ Update anchor 2442645 wikitext text/x-wiki dtylxt96llur06r8tvb80x4ebq7e9gb 2442647 2442645 2026-07-29T16:44:06Z BCornwall-WMF 31521 /* Disabling a Site */ 2442647 wikitext text/x-wiki pvb7i7lmehvygm3tor2mtvex5r0npup 12 265249 2442570 2380881 2026-07-29T15:42:58Z Taavi-WMF 41365 Taavi-WMF moved page [[Help:Toolforge/Elasticsearch]] to [[Help:Toolforge/OpenSearch]]: we've moved to the fork 2380881 wikitext text/x-wiki 6wbps44aq79qux49wm9ge2qt9fnu436 2442585 2442570 2026-07-29T15:49:46Z Taavi-WMF 41365 innovation by sed 2442585 wikitext text/x-wiki 3gf2s6meibq1hfa2f88li6ogidwpgke 2442967 2442585 2026-07-30T11:33:51Z DCaro (WMF) 18271 /* OpenSearch for Toolforge */ 2442967 wikitext text/x-wiki btu48k8nlmy504jnjxqj9dzqe0lln4h 0 441012 2442612 2442164 2026-07-29T16:14:37Z DCaro (WMF) 18271 /* Clinic duty */ 2442612 wikitext text/x-wiki bn5u4g6iedkz4q2ci7osb1hk6vcdtl7 10 442718 2442823 2430929 2026-07-30T01:28:02Z ABreault (WMF) 37749 1.47 is being deployed 2442823 wikitext text/x-wiki 9s78ptxlvj4fznqhe90j6nnc2oiwxli 498 443628 2442863 2440269 2026-07-30T07:29:57Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/30522988957 (https://github.com/cluebotng/component-configs/commits/e38d48bc554d39c0874c954226583b005318f76f) 2442863 wikitext text/x-wiki ldpd4ia6bo21w1sd305mz8b0zrueujg 0 444845 2442407 2358294 2026-07-29T13:44:43Z LMora-WMF 38545 typo 2442407 wikitext text/x-wiki 2hubepw7r7cxzlul0ziq44xk5x9pb2a 2442412 2442407 2026-07-29T13:48:22Z LMora-WMF 38545 typo fix 2442412 wikitext text/x-wiki 38t8ysl34ty1s0cna2ylezv9462evrr 0 446215 2442891 2436194 2026-07-30T08:44:41Z DCaro (WMF) 18271 /* Find, stop and lock suspicious accounts */ 2442891 wikitext text/x-wiki l3oh4z90g409kg15ud3borvzbqx496i 2442897 2442891 2026-07-30T08:51:52Z DCaro (WMF) 18271 /* Common administrative actions */ added note on getting public-paws unstuck 2442897 wikitext text/x-wiki s3qk0r65upvden0ew2wcd6mvkgsdx1t 0 446532 2442757 2312267 2026-07-29T19:18:53Z RobH 17 /* Updating Firmware */ 2442757 wikitext text/x-wiki p1mhwpuydwk06hq7guau0btj7cqm7cw 2442784 2442757 2026-07-29T20:14:44Z RobH 17 outdated info, strikethrough for history to ensure non issue in future 2442784 wikitext text/x-wiki tmu0f7wzrm0b96ugnzxyz4ymfgti1sk 0 448609 2442577 2344211 2026-07-29T15:43:58Z Taavi-WMF 41365 hatnote to toolforge docs 2442577 wikitext text/x-wiki row4yg7hi3zbb8q8k8eudmglllh9z2v 0 448962 2442372 2213532 2026-07-29T13:20:09Z JWodstrcil (WMF) 24826 2442372 wikitext text/x-wiki 6bvtzrwbjp75oaik7ylnlz9l2xhbobz 13 449065 2442572 1944983 2026-07-29T15:42:59Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Elasticsearch]] to [[Help talk:Toolforge/OpenSearch]]: we've moved to the fork 1944983 wikitext text/x-wiki sf669qv8hkj7m8kra341tl3xmroowsd 0 449160 2442280 2442130 2026-07-29T12:14:31Z Dexbot 30554 Bot: Updating the report 2442280 wikitext text/x-wiki czki1dmggse334l1y5pnm72flel7hxn 2442822 2442280 2026-07-30T00:01:21Z Dexbot 30554 Bot: Updating the report 2442822 wikitext text/x-wiki lrxgpeevgkfmsqp6jqbl7awj10ayup5 2442830 2442822 2026-07-30T04:04:54Z Dexbot 30554 Bot: Updating the report 2442830 wikitext text/x-wiki b935l823z21ssssgv0p9xzw3g3plgml 2442835 2442830 2026-07-30T04:31:58Z Dexbot 30554 Bot: Updating the report 2442835 wikitext text/x-wiki nvwpuncolrwiev2jsi2zpy1ysrhmpma 2442923 2442835 2026-07-30T09:29:41Z Dexbot 30554 Bot: Updating the report 2442923 wikitext text/x-wiki 275s43zi38en2j6pus595wm9nvk3uo2 0 450818 2442771 2435054 2026-07-29T19:33:39Z BKing (WMF) 29017 2442771 wikitext text/x-wiki 8q3rw9mogn1hswj3f31es1f90613xmu 0 451600 2442753 2432268 2026-07-29T19:11:50Z Aaron Schulz 22 /* Testing a Chart */ ruby-git 5.0 monkey patch shim is deprecated and doesn't work with deployment-charts/.rake_modules/monkeypatch.rb 2442753 wikitext text/x-wiki cbi8tljb6vidvu4o1n7onxszq9i2hrs 12 452933 2442965 2421548 2026-07-30T11:32:06Z DCaro (WMF) 18271 /* Globally set environment variables */ 2442965 wikitext text/x-wiki du3hjbmn3yrg3i4wumuj7zbghb9ivvk 0 458713 2442703 2384269 2026-07-29T17:56:47Z SNwachukwu (WMF) 39521 2442703 wikitext text/x-wiki 0mt85f2an9rs9bqhecf7ej67mp67xmn 0 459290 2442542 2441840 2026-07-29T15:19:08Z AKhatun (WMF) 50461 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2442542 wikitext text/x-wiki fepgyxolirmkye6jkjo1y9ofhgzhw04 2442552 2442542 2026-07-29T15:26:56Z AKhatun (WMF) 50461 Fix undeployed change, move to current traind 2442552 wikitext text/x-wiki 73n2zscemwhmh8e8q7dlwgrgayopady 2442605 2442552 2026-07-29T16:02:19Z APizzata-WMF 47941 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2442605 wikitext text/x-wiki pogqx3wxaccqbqx4kvu9hhe97u4rorq 2442606 2442605 2026-07-29T16:02:34Z APizzata-WMF 47941 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2442606 wikitext text/x-wiki g5pcqp1ulk90smyjvuwop5teiyfqniq 2442728 2442606 2026-07-29T18:26:08Z AKhatun (WMF) 50461 Update deployment 2442728 wikitext text/x-wiki h7u886958cavbla4151nhpat364r7m5 0 459511 2442911 2401976 2026-07-30T09:16:41Z Jdforrester (WMF) 468 Add recent happenings, all green. 2442911 wikitext text/x-wiki g3omr9u24gsx35kd66k1q69gzg5fvig 12 460498 2442571 2026-07-29T15:42:59Z Taavi-WMF 41365 Taavi-WMF moved page [[Help:Toolforge/Elasticsearch]] to [[Help:Toolforge/OpenSearch]]: we've moved to the fork 2442571 wikitext text/x-wiki lj014octcp7yuxbad5w0hx43nb167ac 13 460499 2442573 2026-07-29T15:42:59Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Elasticsearch]] to [[Help talk:Toolforge/OpenSearch]]: we've moved to the fork 2442573 wikitext text/x-wiki 7nsxd8jguc4soqebqc2zxi7niw4hl7p 0 460500 2442671 2026-07-29T17:17:51Z BCornwall-WMF 31521 Redirected page to [[Data centers]] 2442671 wikitext text/x-wiki 4jkdk7y318pxp1xyglwgzz0qqo97frr 0 460501 2442672 2026-07-29T17:18:06Z BCornwall-WMF 31521 Redirected page to [[Data centers]] 2442672 wikitext text/x-wiki 4jkdk7y318pxp1xyglwgzz0qqo97frr