Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.11 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2436320 2436259 2026-07-14T12:19:49Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-5 2436320 wikitext text/x-wiki svxbg5pobs0o47vz2a58jss6g8y8m29 2436322 2436320 2026-07-14T12:20:43Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-5 2436322 wikitext text/x-wiki 18cqoi53qwzhcyd0i7kzvsh9abz0hvb 2436546 2436322 2026-07-14T16:33:03Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2436546 wikitext text/x-wiki dggzal1kbud87bklv5m2cfbp9gv2x7g 2436549 2436546 2026-07-14T16:45:27Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2436549 wikitext text/x-wiki m748at2vowxudrub9f5hm9wuq6is1ll 2436708 2436549 2026-07-15T09:08:23Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2436708 wikitext text/x-wiki gfdzwaahba6nluqqkrtj0f2gqdfeqxs 2436713 2436708 2026-07-15T09:20:32Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2436713 wikitext text/x-wiki mrfsic9qgy6swvj35f8oavtbwm1blzr 0 4108 2436551 2435940 2026-07-14T16:48:13Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-07-14}} */ UTC-late infra window 2436551 wikitext text/x-wiki p5i3e5sb11cstx5243kqlzc1q9pp6jc 2436566 2436551 2026-07-14T17:57:06Z ScheduleDeploymentBot 37566 Add [[gerrit:1310625]] to Tuesday, July 14 UTC late backport window 2436566 wikitext text/x-wiki duyyw5r2fxs0p3wt5xt5eu7789pf6ot 2436575 2436566 2026-07-14T19:17:34Z ScheduleDeploymentBot 37566 Add [[gerrit:1309865]] to Tuesday, July 14 UTC late backport window 2436575 wikitext text/x-wiki jmrpncud9ual6s9q6nu4bjsi2oqhir0 2436593 2436575 2026-07-14T20:38:24Z ScheduleDeploymentBot 37566 Add [[gerrit:1310653]] to Wednesday, July 15 UTC afternoon backport window 2436593 wikitext text/x-wiki fl62e2ypo780mnynbo1ijkmip7ymdnw 2436596 2436593 2026-07-14T20:44:21Z Aude 39 /* {{Deployment_day|date=2026-07-15}} */ remove 2436596 wikitext text/x-wiki 955xwrgdddjq296jw2kkkwsctfc5uyn 2436622 2436596 2026-07-15T06:54:21Z ScheduleDeploymentBot 37566 Add [[gerrit:1310470]] to Wednesday, July 15 UTC morning backport window 2436622 wikitext text/x-wiki myqk06llx7wswosaatt8wpbkfzg0zoq 498 6726 2436326 2436230 2026-07-14T12:22:12Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-harbor' 2436326 wikitext text/x-wiki h7k2j3f4xcxrwm13y5bknxxxwwauetw 2436341 2436326 2026-07-14T12:28:26Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-harbor' 2436341 wikitext text/x-wiki rhhv7hxqya7vvd3xzk0rgqg5o0ulxhd 2436507 2436341 2026-07-14T15:07:27Z Stashbot 7414 godog: fail back to classic dumps nfs for toolsbeta-test-k8s-worker-nfs prefix 2436507 wikitext text/x-wiki p44nzl80phidkls994xrvml27kwb15c 2436524 2436507 2026-07-14T15:49:21Z Stashbot 7414 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api 2436524 wikitext text/x-wiki okp9s45gqqrr6brd88ry22t58wki99n 2436525 2436524 2026-07-14T15:49:23Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2436525 wikitext text/x-wiki 8vp6xod73suadtvdjtzeq0knqc3pklu 2436527 2436525 2026-07-14T15:50:04Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2436527 wikitext text/x-wiki r2as1k224xr1yksv892sbno5678jh87 2436529 2436527 2026-07-14T15:59:52Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2436529 wikitext text/x-wiki p9w6881xdfx99y7uumlo0prxysgj87p 2436535 2436529 2026-07-14T16:11:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2436535 wikitext text/x-wiki d2oc716tl18knvev7m13b1gqh4t7tup 2436702 2436535 2026-07-15T08:51:34Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2436702 wikitext text/x-wiki d4m9j5n1p0gwgstpqryeo19o78v18yi 2436706 2436702 2026-07-15T09:02:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2436706 wikitext text/x-wiki o2m05r8mfahlatcxxrrl7r81d1z92dz 0 7919 2436284 2436283 2026-07-14T11:59:19Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1067 - blake@cumin1003" 2436284 wikitext text/x-wiki mg8y3zfon9fd0hp3mbg5ujg94xcajon 2436285 2436284 2026-07-14T11:59:21Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436285 wikitext text/x-wiki caarm1deye2x919rocm1p027qxwyoi7 2436286 2436285 2026-07-14T11:59:22Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1067.eqiad.wmnet 17.48.64.10.in-addr.arpa 7.1.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436286 wikitext text/x-wiki jiapxwkaspdsu84x6efmtbzo1hk07g9 2436287 2436286 2026-07-14T11:59:23Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1067.eqiad.wmnet 17.48.64.10.in-addr.arpa 7.1.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436287 wikitext text/x-wiki re4v8se0omzodnftb66pwyerty6mw0r 2436288 2436287 2026-07-14T11:59:24Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1067 2436288 wikitext text/x-wiki hi1g3gm6zfiqoejig6bjn3h4jdpscsw 2436289 2436288 2026-07-14T11:59:42Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1067 2436289 wikitext text/x-wiki oumqh5do1cf79kk0cn0ne180j0rsdnm 2436290 2436289 2026-07-14T11:59:43Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1067 2436290 wikitext text/x-wiki nh3z5zkkzt1e4q7kaecgng9wyoc1uz8 2436291 2436290 2026-07-14T11:59:59Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2018.codfw.wmnet with OS trixie 2436291 wikitext text/x-wiki etzwsrtoe02b6378i0tm4rk6huyadty 2436292 2436291 2026-07-14T12:01:14Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2436292 wikitext text/x-wiki lqocjbwksp0oppacl9dzvnru1vnzakh 2436293 2436292 2026-07-14T12:01:29Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1002.eqiad.wmnet 2436293 wikitext text/x-wiki 2h55lxuubs5gvtxijfy601uppl0coae 2436294 2436293 2026-07-14T12:01:30Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1002.eqiad.wmnet 2436294 wikitext text/x-wiki qkx579yl4mxcgvb2n78d56lyvq44jd4 2436295 2436294 2026-07-14T12:01:36Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1003.eqiad.wmnet 2436295 wikitext text/x-wiki 1rnqeoc9rqejiadjb3k3cptzscggqcy 2436296 2436295 2026-07-14T12:02:10Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1003.eqiad.wmnet 2436296 wikitext text/x-wiki r1emrq2q81qd7o8fdnzen3d6mko8ayu 2436297 2436296 2026-07-14T12:04:13Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436297 wikitext text/x-wiki d0qnnxc7mnkx8w1djssgczov6a399s9 2436298 2436297 2026-07-14T12:04:38Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "set ml-serve1001 back to active state - cmooney@cumin1003" 2436298 wikitext text/x-wiki 6r779peuwx03rv42czg0zdpi5gy6hu7 2436299 2436298 2026-07-14T12:05:17Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "set ml-serve1001 back to active state - cmooney@cumin1003" 2436299 wikitext text/x-wiki 31x0hcupvlvb4o2feuva1w4rb9zadia 2436300 2436299 2026-07-14T12:05:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1046.eqiad.wmnet with reason: host reimage 2436300 wikitext text/x-wiki p7m4mtnpkp57bc5eaal08b3ke3xzn89 2436301 2436300 2026-07-14T12:06:16Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1003.eqiad.wmnet 2436301 wikitext text/x-wiki behs90hbd5qe46zr71s7739zakgfhcs 2436302 2436301 2026-07-14T12:06:18Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1003.eqiad.wmnet 2436302 wikitext text/x-wiki 2xnwrdfshikhojrvbeeqdabb0ay1bpa 2436303 2436302 2026-07-14T12:06:23Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1004.eqiad.wmnet 2436303 wikitext text/x-wiki sfoh8ep2bge7c3e3ons1rj4kknz6719 2436304 2436303 2026-07-14T12:06:57Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1004.eqiad.wmnet 2436304 wikitext text/x-wiki 3a4c7xl7d56afewviz8v1mwejdbpk54 2436305 2436304 2026-07-14T12:08:29Z Stashbot 7414 atsukoito: restarting pybal on lvs1019 `low-traffic` for https://gerrit.wikimedia.org/r/1310535 2436305 wikitext text/x-wiki b3l1b4rldguqrvmyt3g87y0l36irlpg 2436306 2436305 2026-07-14T12:09:52Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1046.eqiad.wmnet with reason: host reimage 2436306 wikitext text/x-wiki m7e3qt8vf7dexte8bwhfnr41wogpjm4 2436307 2436306 2026-07-14T12:10:54Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1004.eqiad.wmnet 2436307 wikitext text/x-wiki gdqkeb627f39p7d9sca4zrxq2pmpx8s 2436308 2436307 2026-07-14T12:10:55Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1004.eqiad.wmnet 2436308 wikitext text/x-wiki 5dieu92km97ls5eplzxkk8h8a26g15x 2436309 2436308 2026-07-14T12:11:01Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1005.eqiad.wmnet 2436309 wikitext text/x-wiki cms6dqu7c0jchsp51epjelha4nnqtvb 2436310 2436309 2026-07-14T12:11:35Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1005.eqiad.wmnet 2436310 wikitext text/x-wiki b3hfppy2u9zzk1yr4uucosnio0k88tr 2436312 2436310 2026-07-14T12:12:45Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1018.eqiad.wmnet with reason: host reimage 2436312 wikitext text/x-wiki tcsyxyq1ok1ibyt0dea2a9e73u50rva 2436313 2436312 2026-07-14T12:15:12Z Stashbot 7414 atsukoito: restarting pybal on lvs2014 for https://gerrit.wikimedia.org/r/1310535 2436313 wikitext text/x-wiki f9qxc8tluxftx65tvdhs5p6j23t66y9 2436314 2436313 2026-07-14T12:15:38Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1005.eqiad.wmnet 2436314 wikitext text/x-wiki 7iu95hbivu00i5btf69tinrqoo1ntzf 2436315 2436314 2026-07-14T12:15:39Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1005.eqiad.wmnet 2436315 wikitext text/x-wiki cq4rhfphruxzifpd3tqtw0vpy2byovf 2436316 2436315 2026-07-14T12:15:44Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet 2436316 wikitext text/x-wiki d3rzmz3aemotl4ztu6uag05kx51ux4a 2436317 2436316 2026-07-14T12:16:17Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet 2436317 wikitext text/x-wiki 52tv09oetp58po4k5b66w5pe5tnly8o 2436318 2436317 2026-07-14T12:18:50Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1067.eqiad.wmnet with reason: host reimage 2436318 wikitext text/x-wiki lyg5vu1gsz5fqa5l95lsbqpdhgx7p3c 2436319 2436318 2026-07-14T12:19:16Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2018.codfw.wmnet with reason: host reimage 2436319 wikitext text/x-wiki fjipbyq7vdpbx3ixwaq0czkjb04d171 2436321 2436319 2026-07-14T12:20:41Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1018.eqiad.wmnet with reason: host reimage 2436321 wikitext text/x-wiki k3j3vkjipvk93m4th5nzwjt0h9rcqc4 2436323 2436321 2026-07-14T12:21:55Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet 2436323 wikitext text/x-wiki go40sr8nzdlwmuxyfjwemb4d83z7j5a 2436324 2436323 2026-07-14T12:21:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet 2436324 wikitext text/x-wiki 13c5q2qh7rletbhdhxpuz1hsdy77wfh 2436325 2436324 2026-07-14T12:22:02Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet 2436325 wikitext text/x-wiki m30van2fl7g1y0cik37v36qtfht7z6c 2436327 2436325 2026-07-14T12:22:25Z Stashbot 7414 atsukoito: restarting pybal on lvs2013 `low-traffic` for https://gerrit.wikimedia.org/r/1310535 2436327 wikitext text/x-wiki pfktcbiofiq8k5pnk3rfuid6j7z2ypv 2436328 2436327 2026-07-14T12:22:32Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host idp-test1005.wikimedia.org 2436328 wikitext text/x-wiki ca1pa3kw489gkdscv9ho0zeahg1gcr0 2436329 2436328 2026-07-14T12:22:34Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet 2436329 wikitext text/x-wiki jz10nftixu7zion6wftxz35kxmtntei 2436330 2436329 2026-07-14T12:23:10Z Stashbot 7414 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model goodfaith --old (T431159) 2436330 wikitext text/x-wiki 1r61pgl78e8b819xak7sey5scolwei2 2436331 2436330 2026-07-14T12:23:24Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1005.wikimedia.org 2436331 wikitext text/x-wiki sydrbnzst6qj1l3sgai5p3h0mxkpuo9 2436332 2436331 2026-07-14T12:24:10Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org 2436332 wikitext text/x-wiki 7sz4httkt1izrsyrfpb4xorhd3wm3tu 2436333 2436332 2026-07-14T12:24:15Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2018.codfw.wmnet with reason: host reimage 2436333 wikitext text/x-wiki s53y1jipuy5gdlxj24ryfo7w2cy1q2a 2436334 2436333 2026-07-14T12:25:24Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1068.eqiad.wmnet with reason: vacuum overlarge container dbs 2436334 wikitext text/x-wiki la2jpat4nzge4ayyiicfkp1otdbxw1z 2436336 2436334 2026-07-14T12:27:51Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1067.eqiad.wmnet with reason: host reimage 2436336 wikitext text/x-wiki pwdgqv018j0gfl5ztqmpywknaqrgj0c 2436337 2436336 2026-07-14T12:27:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet 2436337 wikitext text/x-wiki 5w8e6gijbg2n7nga4knd0snu2ouv402 2436338 2436337 2026-07-14T12:27:59Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet 2436338 wikitext text/x-wiki 5mj3a86jpj4grjb1f2oxpy2nx831r4j 2436339 2436338 2026-07-14T12:28:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet 2436339 wikitext text/x-wiki ov8qeb7p277ddd9ij5lfjxaw88z45g2 2436340 2436339 2026-07-14T12:28:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org 2436340 wikitext text/x-wiki sft8nnfhexwzu11tnvxwwwwc43o4tln 2436342 2436340 2026-07-14T12:29:43Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host urldownloader1006.wikimedia.org 2436342 wikitext text/x-wiki fa8qhzbuc3h5zhniqo7ikn12ga0a56c 2436343 2436342 2026-07-14T12:29:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1046.eqiad.wmnet with OS trixie 2436343 wikitext text/x-wiki sjidzvh2a0gu0h3bavwbfdnv4jj05db 2436344 2436343 2026-07-14T12:30:51Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1046: es1046 after reimage 2436344 wikitext text/x-wiki a5vzqq5ltn3uvc3ib2nfxkkqfswjqvr 2436345 2436344 2026-07-14T12:33:07Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet 2436345 wikitext text/x-wiki hcsebaw09sxm8f5yefen0irnyhoxrmr 2436346 2436345 2026-07-14T12:34:11Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1006.wikimedia.org 2436346 wikitext text/x-wiki i375xgrsbnlexx8fvk81vqfkoymian7 2436347 2436346 2026-07-14T12:35:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host urldownloader2006.wikimedia.org 2436347 wikitext text/x-wiki 2ztrhfn464l1uuuzn4ty5ook6yly06v 2436348 2436347 2026-07-14T12:38:24Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet 2436348 wikitext text/x-wiki 7ql34pdd4sdzqg1pi1u6gq6zstg9v80 2436349 2436348 2026-07-14T12:38:25Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet 2436349 wikitext text/x-wiki iionx0btghf0forejsh406w1tbqujhf 2436350 2436349 2026-07-14T12:38:30Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet 2436350 wikitext text/x-wiki f6x27sv779h8rga6twlqsrcfb1k7amk 2436351 2436350 2026-07-14T12:38:34Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki-root1002.eqiad.wmnet 2436351 wikitext text/x-wiki n9t56ojn9ggxikzlbyawnzx644imppl 2436352 2436351 2026-07-14T12:39:02Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet 2436352 wikitext text/x-wiki bvmu6cicqo3mob6its74opegh0rurww 2436353 2436352 2026-07-14T12:39:22Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1018.eqiad.wmnet with OS trixie 2436353 wikitext text/x-wiki qt09bioaniwe8ciyroyonn42z1xh0qf 2436354 2436353 2026-07-14T12:39:38Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2006.wikimedia.org 2436354 wikitext text/x-wiki 07y137xhqhoqszkas3qiu5lyi6xqa3l 2436355 2436354 2026-07-14T12:42:37Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1017.eqiad.wmnet with OS trixie 2436355 wikitext text/x-wiki 5i5qqxlh5wgnfrsnx3u3ydp3qe1sgmc 2436356 2436355 2026-07-14T12:42:47Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2018.codfw.wmnet with OS trixie 2436356 wikitext text/x-wiki hwo1e395h1ruuxls09yrd8ajsqgkan7 2436357 2436356 2026-07-14T12:43:32Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough 2436357 wikitext text/x-wiki 5dtlj9qryl47jk5hvou3i3qrl1e1nse 2436358 2436357 2026-07-14T12:43:49Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir and A:ncredir 2436358 wikitext text/x-wiki eplneoi28jzxx309fzu2h7nfmkg12xr 2436359 2436358 2026-07-14T12:44:06Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy 2436359 wikitext text/x-wiki 8mcjsvpuhixskxvuokt7rw8yjfb76d3 2436360 2436359 2026-07-14T12:44:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet 2436360 wikitext text/x-wiki bxxb4mxdeowtt0zahix7k3vmco9ivtk 2436361 2436360 2026-07-14T12:44:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet 2436361 wikitext text/x-wiki p4vlxsbv3qgo079al6bdm9f24xtxy3i 2436362 2436361 2026-07-14T12:44:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{aux-k8s-worker100*} and (A:aux-master-eqiad or A:aux-worker-eqiad) 2436362 wikitext text/x-wiki gleo0gcjtqd8pyhbzost53whm5mhtva 2436363 2436362 2026-07-14T12:44:22Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki-root1002.eqiad.wmnet 2436363 wikitext text/x-wiki 0rpvbacsf7eef1v8ms8yoxbnu2a56bu 2436364 2436363 2026-07-14T12:45:00Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling reboot on A:tcpproxy and A:tcpproxy 2436364 wikitext text/x-wiki pwf6xupb6spqc8hytmxkencp4nnkyy6 2436365 2436364 2026-07-14T12:45:15Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and A:durum 2436365 wikitext text/x-wiki efzkh86x204mrsmm0wxrqfyvhrxholj 2436366 2436365 2026-07-14T12:45:48Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=ms-fe2018.codfw.wmnet 2436366 wikitext text/x-wiki ci4kv6n4cy5arwp6d7n6sd151bn3ijd 2436367 2436366 2026-07-14T12:47:06Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=ms-fe2018.codfw.wmnet 2436367 wikitext text/x-wiki pwlf0s8pglmgl3h957xyuqp97y3izgs 2436368 2436367 2026-07-14T12:47:37Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox and A:ulsfo and (A:dnsbox) 2436368 wikitext text/x-wiki 2k5wbyknxzokmpealonuj0oc3ykc1zw 2436369 2436368 2026-07-14T12:47:39Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns4003.wikimedia.org 2436369 wikitext text/x-wiki pbr42giiwi4ggvxz4sm8b6bd0gfn7e2 2436370 2436369 2026-07-14T12:47:57Z Stashbot 7414 topranks: depool codfw pki in dns discovery ahead of lsw1-b5-codfw maintenance T430918 2436370 wikitext text/x-wiki eetn5qpxeot0czn566nbbwjjnyvrex0 2436371 2436370 2026-07-14T12:48:09Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2017.codfw.wmnet with OS trixie 2436371 wikitext text/x-wiki qdn5xi64sbn064m571fcv4fwb2oh34k 2436372 2436371 2026-07-14T12:48:38Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps-test2001.codfw.wmnet 2436372 wikitext text/x-wiki idqejtkdcmq1w3yduhl02re5dglpolh 2436373 2436372 2026-07-14T12:49:24Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1067.eqiad.wmnet with OS trixie 2436373 wikitext text/x-wiki bxbquwueeqprbl42n47g1bt1daf3zj4 2436374 2436373 2026-07-14T12:49:58Z Stashbot 7414 cmooney@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=pki,name=codfw 2436374 wikitext text/x-wiki 13q4fsxd1zc6by4r34we4wq6bxo2wod 2436375 2436374 2026-07-14T12:54:58Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps-test2001.codfw.wmnet 2436375 wikitext text/x-wiki q9ivpwtjczl2tbq2syvyt17ju8kfjec 2436376 2436375 2026-07-14T12:55:18Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-b5-codfw,lsw1-b5-codfw IPv6,lsw1-b5-codfw.mgmt,ssw1-a[1,8]-codfw.mgmt with reason: switch upgade lsw1-b5-codfw 2436376 wikitext text/x-wiki tgljb96qhzf9hozly1i2avzg116fopi 2436377 2436376 2026-07-14T12:57:08Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 30 hosts with reason: lsw1-b5-codfw JunOS upgrade 2436377 wikitext text/x-wiki q1tsxc8bcwal2mbrkcc0m05ybgzz3mp 2436378 2436377 2026-07-14T12:57:39Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.depool depool db2159: codfw rack B5 depool for maintenance 2436378 wikitext text/x-wiki fl74rhev4s57eaehq516408yowqvmc6 2436379 2436378 2026-07-14T12:57:54Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw 2436379 wikitext text/x-wiki 4c21hv6uzshwid993gohyr6lgilmguk 2436380 2436379 2026-07-14T12:58:15Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=tegola,name=codfw 2436380 wikitext text/x-wiki 29ky7b2m743daq5i69blyiznzsarxtc 2436381 2436380 2026-07-14T12:58:18Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1017.eqiad.wmnet with reason: host reimage 2436381 wikitext text/x-wiki kziwkcqd2d29tmni70rzz2v6zaup1kq 2436382 2436381 2026-07-14T12:58:20Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2159: codfw rack B5 depool for maintenance 2436382 wikitext text/x-wiki ad899o7pwmhmnzjrd9c64ctgf6oegdj 2436383 2436382 2026-07-14T12:58:43Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=tegola-vector-tiles,name=codfw 2436383 wikitext text/x-wiki tc9m2cv4peoldgbjw36fdsugf7y7c64 2436384 2436383 2026-07-14T12:59:08Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.hosts.remove-downtime for ms-be1068.eqiad.wmnet 2436384 wikitext text/x-wiki 450jjhix3cn8tsrgzpw6zozat5tb46w 2436385 2436384 2026-07-14T12:59:09Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1068.eqiad.wmnet 2436385 wikitext text/x-wiki cc12ga4jg6ngl6p0sikk020okx1f2fn 2436386 2436385 2026-07-14T13:00:58Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.depool depool db2177: codfw rack B5 depool for maintenance 2436386 wikitext text/x-wiki g3q1xfsp46tihksce7ux3z2gyks4yyw 2436387 2436386 2026-07-14T13:01:19Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2177: codfw rack B5 depool for maintenance 2436387 wikitext text/x-wiki pdfci0afy1sd0m6ihqoys9fziks4e7b 2436388 2436387 2026-07-14T13:01:22Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.depool depool db2178: codfw rack B5 depool for maintenance 2436388 wikitext text/x-wiki ggn9qvpda3kisg7ojxcm37j97ifx3o0 2436389 2436388 2026-07-14T13:01:40Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host idm-test1001.wikimedia.org 2436389 wikitext text/x-wiki tf3oxin4crwjh1kbgh7u5sxygegp5xl 2436390 2436389 2026-07-14T13:01:43Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2178: codfw rack B5 depool for maintenance 2436390 wikitext text/x-wiki 4vy4tdso8pk9byo78sy5yjj7y9w4p6g 2436391 2436390 2026-07-14T13:01:46Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_magru 2436391 wikitext text/x-wiki b5aswywo0n48pmef7wdhr8qp52juxv9 2436392 2436391 2026-07-14T13:01:47Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.depool depool db2188: codfw rack B5 depool for maintenance 2436392 wikitext text/x-wiki hbat1cy0k3gdfub0fchgb1qift4537t 2436393 2436392 2026-07-14T13:02:01Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_magru 2436393 wikitext text/x-wiki 7hwyjkpator2vpn32ahy8ft4rurjmqy 2436394 2436393 2026-07-14T13:02:08Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2188: codfw rack B5 depool for maintenance 2436394 wikitext text/x-wiki jfws5kn9oe3lqqriurc8kh4qv6hvvy7 2436395 2436394 2026-07-14T13:03:16Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7003*} and A:liberica 2436395 wikitext text/x-wiki 01b4sqng250kyixbtahvplx5fueft84 2436398 2436395 2026-07-14T13:04:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet 2436398 wikitext text/x-wiki et4fm96yfhgr8etdsbi9vqoxxr8lobn 2436399 2436398 2026-07-14T13:05:04Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1017.eqiad.wmnet with reason: host reimage 2436399 wikitext text/x-wiki 5j6gal8t4nbmo1pvhdjg8z41e5k0avd 2436400 2436399 2026-07-14T13:05:33Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2017.codfw.wmnet with reason: host reimage 2436400 wikitext text/x-wiki qxyzx84cde5kosxaaybb9xdxzjd0awc 2436401 2436400 2026-07-14T13:05:36Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm-test1001.wikimedia.org 2436401 wikitext text/x-wiki 1swk4zvgerdl3mwprmg7g4q6cssakm2 2436402 2436401 2026-07-14T13:06:29Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet 2436402 wikitext text/x-wiki i2vv5vpyflo6dz65wzu9yyrq0aedl4j 2436403 2436402 2026-07-14T13:06:30Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet 2436403 wikitext text/x-wiki 6yj2bm7gttop88wtxwiilwo5pm51gsm 2436404 2436403 2026-07-14T13:06:31Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1067.eqiad.wmnet 2436404 wikitext text/x-wiki rejr2cf78vg90umqmt81t3aqhiopylf 2436405 2436404 2026-07-14T13:06:37Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd2003.codfw.wmnet 2436405 wikitext text/x-wiki h1ljklxurwnm0sj0w9ds7niqmqvq82j 2436406 2436405 2026-07-14T13:07:04Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns4003.wikimedia.org 2436406 wikitext text/x-wiki soeqtjgi8pr23r896tik9jtl33rjfw1 2436407 2436406 2026-07-14T13:07:22Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7003*} and A:liberica 2436407 wikitext text/x-wiki agpein6svk9r5i2zlbdjp2rnpxe37h7 2436408 2436407 2026-07-14T13:09:42Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2017.codfw.wmnet with reason: host reimage 2436408 wikitext text/x-wiki khd2c3ten7ysopkq935ansg7xfv1dar 2436409 2436408 2026-07-14T13:10:36Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd2003.codfw.wmnet 2436409 wikitext text/x-wiki j3o47017efn3ts02xlorz0ne6hbxgyr 2436410 2436409 2026-07-14T13:11:22Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet 2436410 wikitext text/x-wiki 7x61zzmodrmo7aqeoyr1t8yhue0oyin 2436411 2436410 2026-07-14T13:11:45Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2001-2002,2095,2272-2278].codfw.wmnet 2436411 wikitext text/x-wiki ke9wj2szeav1onsai2nvhsefdqhboli 2436412 2436411 2026-07-14T13:12:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet 2436412 wikitext text/x-wiki bicfe383kz6qohtmtlwyytnu7rp6cgu 2436413 2436412 2026-07-14T13:13:28Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd2004.codfw.wmnet 2436413 wikitext text/x-wiki lprkfza76654zjb65jhb67qswnbhq9c 2436414 2436413 2026-07-14T13:13:44Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1029.eqiad.wmnet,service=s8 2436414 wikitext text/x-wiki 1aiefdjkz848s52uvbr7ifskb8jjvot 2436415 2436414 2026-07-14T13:13:51Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1029.eqiad.wmnet,service=s5 2436415 wikitext text/x-wiki h5rqsnac9at0gvnf8c0pi1d61gt06gc 2436416 2436415 2026-07-14T13:14:16Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1029.eqiad.wmnet,service=s5 2436416 wikitext text/x-wiki s9f7yokxxfumb794aanen6vtmfftavh 2436417 2436416 2026-07-14T13:14:21Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1029.eqiad.wmnet,service=s8 2436417 wikitext text/x-wiki q32ygacbrqtv1upcc9gfwxwb2xgm6vt 2436418 2436417 2026-07-14T13:16:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1046: es1046 after reimage 2436418 wikitext text/x-wiki oi13bc6lz952bgac223k98buqffyfdx 2436419 2436418 2026-07-14T13:16:50Z Stashbot 7414 elukey@dns1004: START - running authdns-update 2436419 wikitext text/x-wiki htl4x76bu6aqs4vlvh79wzepn687b1o 2436420 2436419 2026-07-14T13:17:28Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd2004.codfw.wmnet 2436420 wikitext text/x-wiki 5rqbvv5p73u5qcejcica0fsf9qk5g32 2436421 2436420 2026-07-14T13:17:38Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd2005.codfw.wmnet 2436421 wikitext text/x-wiki 47tvgg2cm4iyhfgfanrx2dfqi4v2rfw 2436422 2436421 2026-07-14T13:17:58Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2001-2002,2095,2272-2278].codfw.wmnet 2436422 wikitext text/x-wiki 6q3o08zl2om3ddpl1eg9aebwym5zxhd 2436423 2436422 2026-07-14T13:18:48Z Stashbot 7414 elukey@dns1004: END - running authdns-update 2436423 wikitext text/x-wiki 27vslm5i66eo352hl1n5jnn12w6mj16 2436424 2436423 2026-07-14T13:19:51Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet 2436424 wikitext text/x-wiki 0ck0dlwoby8qsv99msvdf4lawg1s9sr 2436425 2436424 2026-07-14T13:21:38Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd2005.codfw.wmnet 2436425 wikitext text/x-wiki 9v38nnfwezq4xuqhl87c4hqyk2b1wnn 2436426 2436425 2026-07-14T13:22:05Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns4004.wikimedia.org 2436426 wikitext text/x-wiki dsmzoye6ddyb21ywkzgqonxabh0v70s 2436427 2436426 2026-07-14T13:22:06Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436427 wikitext text/x-wiki ss8m5ar4qq9h96r44jslfwezixutag2 2436428 2436427 2026-07-14T13:22:18Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436428 wikitext text/x-wiki 77t4ynsvx71el4ih1aihvc2l6y7dfbu 2436429 2436428 2026-07-14T13:22:38Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2002.codfw.wmnet 2436429 wikitext text/x-wiki 1xg65eexm7po6fh9nmjlw7z3zuqg1ok 2436430 2436429 2026-07-14T13:24:29Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet 2436430 wikitext text/x-wiki 0vjoh3or4xjq1w2lncskn42ngeq7izw 2436431 2436430 2026-07-14T13:25:46Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1017.eqiad.wmnet with OS trixie 2436431 wikitext text/x-wiki csec3tmx8c91kyngdw9wib8iwubfno1 2436432 2436431 2026-07-14T13:29:00Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2017.codfw.wmnet with OS trixie 2436432 wikitext text/x-wiki jc9x1h5aj5cpkyk7zuay5ag0f5282wa 2436433 2436432 2026-07-14T13:31:08Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet 2436433 wikitext text/x-wiki nlef7j0h555zax3mq9rahzzw67txxvu 2436434 2436433 2026-07-14T13:31:51Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet 2436434 wikitext text/x-wiki 1lpju5sd0lvgk4gokvznejpfhjljlgr 2436435 2436434 2026-07-14T13:32:45Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2002.codfw.wmnet 2436435 wikitext text/x-wiki bzfyntpni7978s4er0w8f0rhcnonmhx 2436436 2436435 2026-07-14T13:34:18Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2016.codfw.wmnet with OS trixie 2436436 wikitext text/x-wiki 9bjq2wyypegb28gefeyeuuovlz5ffm6 2436437 2436436 2026-07-14T13:34:42Z Stashbot 7414 topranks: reboot lsw1-b5-codfw to upgrade JunOS T430918 2436437 wikitext text/x-wiki i3fdzq7wevfb4x8pahf89ozvxepscco 2436438 2436437 2026-07-14T13:34:44Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1016.eqiad.wmnet with OS trixie 2436438 wikitext text/x-wiki d8lkidbtts2ee2pvd2btbi3xl0g3ka0 2436440 2436438 2026-07-14T13:36:15Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns4004.wikimedia.org 2436440 wikitext text/x-wiki cpynydd0vr3cblscj0yveb1pd4hek5n 2436441 2436440 2026-07-14T13:36:16Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:dnsbox and A:ulsfo and (A:dnsbox) 2436441 wikitext text/x-wiki 6cygmyi2a81nxba195sdwj25fr70q6b 2436443 2436441 2026-07-14T13:38:10Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet 2436443 wikitext text/x-wiki qw2jgv8ss1jgbm9cpb09m31fx9bbyj4 2436444 2436443 2026-07-14T13:42:29Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=97) rolling reboot on A:durum and A:durum 2436444 wikitext text/x-wiki 1ris3fodsau6ts8xkg6olzarhfe1p8o 2436445 2436444 2026-07-14T13:43:36Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and not (A:durum-eqiad or A:durum-codfw or A:durum-esams) and A:durum 2436445 wikitext text/x-wiki l4bbvs9vxrmcg5oq6be42m9nl8tk73p 2436446 2436445 2026-07-14T13:46:46Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling reboot on A:tcpproxy and A:tcpproxy 2436446 wikitext text/x-wiki pye1rcjb6e438zvchwl17c956kuhs20 2436447 2436446 2026-07-14T13:49:07Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough 2436447 wikitext text/x-wiki etijuynabl5ca13nhfr1glcd1xmh92z 2436448 2436447 2026-07-14T13:49:38Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy 2436448 wikitext text/x-wiki r229eck2w2nsfvqlrrr8a8l17opfi7g 2436449 2436448 2026-07-14T13:50:45Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1016.eqiad.wmnet with reason: host reimage 2436449 wikitext text/x-wiki n09q15owcbnufrzt1c0tidmcsb4raab 2436450 2436449 2026-07-14T13:51:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2016.codfw.wmnet with reason: host reimage 2436450 wikitext text/x-wiki ddvlkozp6e7tot8tbmdpkhat5odes6a 2436451 2436450 2026-07-14T13:52:19Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2001-2002,2095,2272-2278].codfw.wmnet 2436451 wikitext text/x-wiki hez65ft4xr2mgphwsiu2c3dutfzgeal 2436452 2436451 2026-07-14T13:52:25Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2001-2002,2095,2272-2278].codfw.wmnet 2436452 wikitext text/x-wiki sjcackxxjjf2stwqeh79z53ynm7bni9 2436453 2436452 2026-07-14T13:54:58Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1016.eqiad.wmnet with reason: host reimage 2436453 wikitext text/x-wiki 43m1w43lg0i6oail2lqr0vafd517ti5 2436454 2436453 2026-07-14T13:56:42Z Stashbot 7414 sbisson@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: sync 2436454 wikitext text/x-wiki s6c9nwj64j5voyksbax5w8al577upcf 2436455 2436454 2026-07-14T13:57:03Z Stashbot 7414 sbisson@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: sync 2436455 wikitext text/x-wiki 88lhdzm14ytgg4lqphij7bztc6hzccn 2436456 2436455 2026-07-14T13:57:05Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir and A:ncredir 2436456 wikitext text/x-wiki 62r958z005n6g25idk90yb7s6ex7dto 2436457 2436456 2026-07-14T13:58:21Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2016.codfw.wmnet with reason: host reimage 2436457 wikitext text/x-wiki jz86edoa6izy3sjdqfug3k5jeorus1w 2436458 2436457 2026-07-14T14:00:09Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host urldownloader2005.wikimedia.org 2436458 wikitext text/x-wiki cd5tpvahs3pjw7wz1mtuoqyj6hrs36m 2436459 2436458 2026-07-14T14:00:58Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw 2436459 wikitext text/x-wiki s3g966omx1lor7s9yun2t1vzb6lm6lq 2436460 2436459 2026-07-14T14:01:06Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=tegola-vector-tiles,name=codfw 2436460 wikitext text/x-wiki d519xza1a9pdgcvtk6u309dt5itd5kd 2436461 2436460 2026-07-14T14:02:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet 2436461 wikitext text/x-wiki g4ghduvfvrhen3o7wfkhgl9md14uj3t 2436462 2436461 2026-07-14T14:02:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet 2436462 wikitext text/x-wiki ofmexf7e29rl1rzfwzssnoybmx9qx4x 2436463 2436462 2026-07-14T14:04:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2005.wikimedia.org 2436463 wikitext text/x-wiki pks6id3zy1ov4p1eedjv8ixgpsvn177 2436464 2436463 2026-07-14T14:05:26Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host urldownloader1005.wikimedia.org 2436464 wikitext text/x-wiki ghwlc705010jc4or1v5xiwf1oz8n85e 2436465 2436464 2026-07-14T14:07:21Z Stashbot 7414 sbisson@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: sync 2436465 wikitext text/x-wiki iu2irlcc5tlzgkngm1uis19j22t31zz 2436466 2436465 2026-07-14T14:07:53Z Stashbot 7414 sbisson@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: sync 2436466 wikitext text/x-wiki 6p53q9ik77ih25vkz31lw9bwhx3dtbr 2436467 2436466 2026-07-14T14:09:52Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1005.wikimedia.org 2436467 wikitext text/x-wiki leoawuhu7trcecwvym496oo2hwavke9 2436468 2436467 2026-07-14T14:11:39Z Stashbot 7414 sbisson@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: sync 2436468 wikitext text/x-wiki ff071loql2j064xp0rkr26zyphcynxp 2436469 2436468 2026-07-14T14:11:51Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2002.codfw.wmnet 2436469 wikitext text/x-wiki tmvnkpasa3fdk9f29sbxna7zbtdgfjg 2436470 2436469 2026-07-14T14:11:53Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2002.codfw.wmnet 2436470 wikitext text/x-wiki otvivmchjgwlwrm8fspo1fsea2rcww5 2436471 2436470 2026-07-14T14:12:08Z Stashbot 7414 sbisson@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: sync 2436471 wikitext text/x-wiki al1zmvg8mgd8u2tn6uw2rfhbb7zhf6l 2436472 2436471 2026-07-14T14:12:28Z Stashbot 7414 cmooney@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw 2436472 wikitext text/x-wiki lre29s70n8m1lo6tivo2tdddx515fe7 2436473 2436472 2026-07-14T14:12:41Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1016.eqiad.wmnet with OS trixie 2436473 wikitext text/x-wiki i9ubmpz5s9z1fjzlp2e09wfybwznl8n 2436474 2436473 2026-07-14T14:14:35Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1015.eqiad.wmnet with OS trixie 2436474 wikitext text/x-wiki b8sjl0ke0ekydd48fu1y6t8kp5dna54 2436475 2436474 2026-07-14T14:14:53Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.pool pool db2159: repooling after rack b5 maintenance 2436475 wikitext text/x-wiki cfzqj8yhfr7oaclagp7ee8waxwk6kvt 2436476 2436475 2026-07-14T14:15:34Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2016.codfw.wmnet with OS trixie 2436476 wikitext text/x-wiki 61l7289m2b06mcc3x4zkhbbpmy5dbbb 2436477 2436476 2026-07-14T14:16:19Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2015.codfw.wmnet with OS trixie 2436477 wikitext text/x-wiki chj7hn3iu69x239weq03hi1ediwpptq 2436478 2436477 2026-07-14T14:24:57Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum and not (A:durum-eqiad or A:durum-codfw or A:durum-esams) and A:durum 2436478 wikitext text/x-wiki iyiq08kz78c111xaicwbvxz9zdyka80 2436479 2436478 2026-07-14T14:30:50Z Stashbot 7414 seanleong-wmde@deploy2003: mwscript-k8s job started: foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https # T429939 2436479 wikitext text/x-wiki g1j1xxc9b54bmw0lt8x4rwffvkzqo9w 2436480 2436479 2026-07-14T14:30:55Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1015.eqiad.wmnet with reason: host reimage 2436480 wikitext text/x-wiki qa4cvidiwvbac2ngobdyi52lf583u1y 2436482 2436480 2026-07-14T14:33:20Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2015.codfw.wmnet with reason: host reimage 2436482 wikitext text/x-wiki 1gyw1qu71fhoe19mkvpb5l7b6v1j1ke 2436483 2436482 2026-07-14T14:34:36Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1015.eqiad.wmnet with reason: host reimage 2436483 wikitext text/x-wiki iia44qkmv9b6k8n92ic61srvyh6vcv5 2436485 2436483 2026-07-14T14:38:44Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2015.codfw.wmnet with reason: host reimage 2436485 wikitext text/x-wiki 7qi70rchdni1rbwd0j6lavs5cwu1lp3 2436486 2436485 2026-07-14T14:39:40Z Stashbot 7414 otto@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2436486 wikitext text/x-wiki 57tpcat7rt0gzi65ugxo89zhh6webmb 2436487 2436486 2026-07-14T14:40:14Z Stashbot 7414 otto@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2436487 wikitext text/x-wiki e9w505lfku3g6obdxsxz1z1sr0dfc14 2436488 2436487 2026-07-14T14:40:21Z Stashbot 7414 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2436488 wikitext text/x-wiki 20k6fdp464a74vmhiq3l74cgo0v1vlm 2436489 2436488 2026-07-14T14:41:04Z Stashbot 7414 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2436489 wikitext text/x-wiki c1ozzaauyd5k5oe8bs9iyiamdljymkh 2436490 2436489 2026-07-14T14:41:15Z Stashbot 7414 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2436490 wikitext text/x-wiki 0r43l1mwy8hmizhmlvtk78c0ws09stk 2436491 2436490 2026-07-14T14:42:05Z Stashbot 7414 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2436491 wikitext text/x-wiki iff1v57s69m03emhlrxikkh3ddk00dd 2436492 2436491 2026-07-14T14:46:10Z Stashbot 7414 javiermonton@deploy2003: Started scap sync-world: Backport for [[gerrit:1310587|stream: pageview.v1 (T425624)]] 2436492 wikitext text/x-wiki 3yapd9qg4aqw1fgld330ek0e4jdv7dd 2436493 2436492 2026-07-14T14:48:10Z Stashbot 7414 javiermonton@deploy2003: javiermonton: Backport for [[gerrit:1310587|stream: pageview.v1 (T425624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436493 wikitext text/x-wiki b1qa5oarsr8x7d2ftl3pnbogxtl1w5s 2436495 2436493 2026-07-14T14:49:25Z Stashbot 7414 javiermonton@deploy2003: javiermonton: Continuing with deployment 2436495 wikitext text/x-wiki 0sxpa1e78tz5evsrjuo795sulxsifig 2436496 2436495 2026-07-14T14:53:31Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1015.eqiad.wmnet with OS trixie 2436496 wikitext text/x-wiki plklxirbbg1i5ess0k27hpx44v73fk6 2436497 2436496 2026-07-14T14:53:45Z Stashbot 7414 javiermonton@deploy2003: Finished scap sync-world: Backport for [[gerrit:1310587|stream: pageview.v1 (T425624)]] (duration: 07m 35s) 2436497 wikitext text/x-wiki 453s80zk5eh7f07nbuy6stf4ekbabda 2436498 2436497 2026-07-14T14:54:34Z Stashbot 7414 seanleong-wmde: Finished populateSitesTable for isvwiki (T429939) 2436498 wikitext text/x-wiki abc940er7gs2h5n3rhhfvqyz85wfz8r 2436499 2436498 2026-07-14T14:57:30Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2015.codfw.wmnet with OS trixie 2436499 wikitext text/x-wiki kknw9tx5j66yy5emyljtfh3s36d4e6x 2436502 2436499 2026-07-14T14:59:56Z Stashbot 7414 dancy@deploy2003: Installing scap version "4.274.1" for 3 host(s) 2436502 wikitext text/x-wiki qvo79b9pta3u9qj6hsagyl4zem45bsy 2436503 2436502 2026-07-14T15:00:21Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2159: repooling after rack b5 maintenance 2436503 wikitext text/x-wiki a16q3etksljiqfo64wvt3rvx7u6zdlb 2436504 2436503 2026-07-14T15:00:25Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.pool pool db2177: repooling after rack b5 maintenance 2436504 wikitext text/x-wiki cp128qav3f6z2l7qszcwltov48ot1if 2436505 2436504 2026-07-14T15:01:56Z Stashbot 7414 dancy@deploy2003: Installation of scap version "4.274.1" completed for 3 hosts 2436505 wikitext text/x-wiki cruo6m35tbc31jjh0v72ui83jbrddk7 2436508 2436505 2026-07-14T15:19:30Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1014.eqiad.wmnet with OS trixie 2436508 wikitext text/x-wiki 72px376m5l39ol1ff0svjexgc7115js 2436509 2436508 2026-07-14T15:20:03Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2014.codfw.wmnet with OS trixie 2436509 wikitext text/x-wiki 0y9gfalune0ov7cywvn72onm20i215t 2436510 2436509 2026-07-14T15:20:22Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.move-vlan for host ms-fe2014 2436510 wikitext text/x-wiki 5e4nh6hvfbvq7kvbquboqv49to5hrdr 2436511 2436510 2026-07-14T15:20:30Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.netbox 2436511 wikitext text/x-wiki 5ypjhgzqy0uhiplwp2mtefmi2k584tg 2436512 2436511 2026-07-14T15:22:45Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Apply title-related policies when selecting the name of the entity - kamila@cumin1003" 2436512 wikitext text/x-wiki h2s534o7cwllgxzgb5b0gnicmd0wri2 2436513 2436512 2026-07-14T15:22:48Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Apply title-related policies when selecting the name of the entity - kamila@cumin1003 2436513 wikitext text/x-wiki 75gez4a3hdcm6t0x3epp2oxwyg26fjx 2436514 2436513 2026-07-14T15:23:41Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Apply title-related policies when selecting the name of the entity - kamila@cumin1003 2436514 wikitext text/x-wiki n39woj1j2rf973riz6rzusqof6xkqhk 2436515 2436514 2026-07-14T15:23:42Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Apply title-related policies when selecting the name of the entity - kamila@cumin1003" 2436515 wikitext text/x-wiki sud9fw4ekivz8r7ufb4lzis4t94btyx 2436516 2436515 2026-07-14T15:28:24Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2014 - mvernon@cumin2003" 2436516 wikitext text/x-wiki hnuj0gvtoplmyawzdpv64xep2dwno4d 2436517 2436516 2026-07-14T15:28:29Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2014 - mvernon@cumin2003" 2436517 wikitext text/x-wiki 6nmlrceogrf2ojhurfdov6zzfdchr41 2436518 2436517 2026-07-14T15:28:30Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436518 wikitext text/x-wiki 4etevt3ktelle5fmhurc913vt3s20fw 2436519 2436518 2026-07-14T15:28:31Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.wipe-cache ms-fe2014.codfw.wmnet 194.16.192.10.in-addr.arpa 4.9.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436519 wikitext text/x-wiki d3kjkof6ku4xr92j4vvvnc57ss9vlx3 2436520 2436519 2026-07-14T15:28:33Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2014.codfw.wmnet 194.16.192.10.in-addr.arpa 4.9.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436520 wikitext text/x-wiki 3kx9nmnwsac9s1ucphsgussnhpads0v 2436521 2436520 2026-07-14T15:28:34Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2014 2436521 wikitext text/x-wiki bumgm7wtfnbphzc0qyutbvehdvuczfu 2436522 2436521 2026-07-14T15:28:48Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2014 2436522 wikitext text/x-wiki q16emjnchbsympvvtjhbjmttcaw7d4u 2436523 2436522 2026-07-14T15:28:49Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2014 2436523 wikitext text/x-wiki g2pmok3viy6diadynr3tg52fxdm1rze 2436526 2436523 2026-07-14T15:49:37Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2014.codfw.wmnet with reason: host reimage 2436526 wikitext text/x-wiki sfuc3x3o5vw2hpm6l9rg7mknqhkc6fy 2436528 2436526 2026-07-14T15:56:53Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1014.eqiad.wmnet with OS trixie 2436528 wikitext text/x-wiki c79fposkuq65j36r07ghvrjic4cs2z8 2436530 2436528 2026-07-14T16:03:57Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2014.codfw.wmnet with OS trixie 2436530 wikitext text/x-wiki asxpnxyqvhsglrv3p3orhugo1mg3wm2 2436531 2436530 2026-07-14T16:07:01Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe 2436531 wikitext text/x-wiki qg7vfdif9123f29fnxls88l91w16rby 2436532 2436531 2026-07-14T16:07:32Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4039.ulsfo.wmnet 2436532 wikitext text/x-wiki d91q6in7w0jpimk3awtrxjkdzcrk49g 2436533 2436532 2026-07-14T16:09:51Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw 2436533 wikitext text/x-wiki p6cmo9q06r8hknesej83j1rx3rlc5k2 2436534 2436533 2026-07-14T16:10:10Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe 2436534 wikitext text/x-wiki ef92ilk7cmfs07h1rpz968oo8qdkujb 2436536 2436534 2026-07-14T16:17:11Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=ms-fe2014.codfw.wmnet 2436536 wikitext text/x-wiki d0uzgdsj99q7p457in40qozpfyxwj89 2436537 2436536 2026-07-14T16:18:05Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw 2436537 wikitext text/x-wiki 33af82srzbf698ia6psj4o6zo5usmfe 2436538 2436537 2026-07-14T16:18:20Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=ms-fe2014.codfw.wmnet 2436538 wikitext text/x-wiki lem9bby42g8m6ue84us5kd08sc61jnc 2436539 2436538 2026-07-14T16:28:18Z Stashbot 7414 blake@deploy2003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2436539 wikitext text/x-wiki 9pvoekeyomy0voqjp2dzklepldosdvs 2436540 2436539 2026-07-14T16:28:42Z Stashbot 7414 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2436540 wikitext text/x-wiki j45ikc506gkmkg06hwzjs0ksz3qfo7o 2436541 2436540 2026-07-14T16:28:49Z Stashbot 7414 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2436541 wikitext text/x-wiki eho3e1gd22rv5srrwxj8119c41yfgdw 2436542 2436541 2026-07-14T16:29:06Z Stashbot 7414 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2436542 wikitext text/x-wiki 35q3dducg95qdiug1riqtkys7ctpna8 2436543 2436542 2026-07-14T16:31:24Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2178: repooling after rack b5 maintenance 2436543 wikitext text/x-wiki sk9d3xglbp84hxmk9xav5dqdsxa901o 2436544 2436543 2026-07-14T16:31:28Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.pool pool db2188: repooling after rack b5 maintenance 2436544 wikitext text/x-wiki rxmh34kausaor6frmjqn0dftls9k9go 2436545 2436544 2026-07-14T16:32:22Z Stashbot 7414 mutante: contint1003 - main CI server - rebooting 2436545 wikitext text/x-wiki nyc26aq1d6lmz6detnjvss3y2eb7tql 2436547 2436545 2026-07-14T16:33:21Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on contint1003.wikimedia.org with reason: reboot 2436547 wikitext text/x-wiki lapmzigxwx64p5u5uk290nnzmfdx7y5 2436548 2436547 2026-07-14T16:44:21Z Stashbot 7414 sukhe: sudo cumin -b31 "A:cp" "run-puppet-agent" 2436548 wikitext text/x-wiki 69lqmqmiif1p4v0o9sgt7jia8sukwgo 2436550 2436548 2026-07-14T16:47:24Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4039.ulsfo.wmnet 2436550 wikitext text/x-wiki emj6ovjec30xvw1y6wb8fra73qwjp5d 2436552 2436550 2026-07-14T16:50:04Z Stashbot 7414 sukhe: pool cp2046 2436552 wikitext text/x-wiki of09me4vjp4ke2x2hh653wpl3na72by 2436554 2436552 2026-07-14T16:57:19Z Stashbot 7414 swfrench-wmf: reprepro include php8.3_8.3.32-1+wmf12u2 into component/php83 for bookworm-wikimedia 2436554 wikitext text/x-wiki nvydfs4o1hi5rkjhtubq9quy4qmoqgc 2436555 2436554 2026-07-14T17:01:36Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7001.magru.wmnet 2436555 wikitext text/x-wiki 6mmkresmvbc183e7p8dept9mndz15ut 2436556 2436555 2026-07-14T17:12:56Z Stashbot 7414 swfrench@deploy2003: Started scap sync-world: Deployment to pick up new production image 2436556 wikitext text/x-wiki g0jeap7z3mgi4lpinhyyy91w2io9wya 2436557 2436556 2026-07-14T17:16:56Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: repooling after rack b5 maintenance 2436557 wikitext text/x-wiki jp3s7eylppd61zkz5mmkgrodrub6rkb 2436558 2436557 2026-07-14T17:16:59Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.pool pool es2035: repooling after rack b5 maintenance 2436558 wikitext text/x-wiki 75ic77mhswxn7j79oclhr5ip8j9ho0o 2436559 2436558 2026-07-14T17:17:02Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2035: repooling after rack b5 maintenance 2436559 wikitext text/x-wiki cipn9jxeu8ye00h9qg381613fwcc5ah 2436561 2436559 2026-07-14T17:29:52Z Stashbot 7414 swfrench@deploy2003: swfrench: Deployment to pick up new production image synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436561 wikitext text/x-wiki cy24ga8mcuu0it6xhmhy96hin8inzvl 2436562 2436561 2026-07-14T17:32:18Z Stashbot 7414 swfrench@deploy2003: swfrench: Continuing with deployment 2436562 wikitext text/x-wiki j7alscwearoch5xlvjoky92y78wcrnh 2436563 2436562 2026-07-14T17:33:24Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7009.magru.wmnet 2436563 wikitext text/x-wiki 422zem2dpt16qfn5qsvlemyax4yom9j 2436564 2436563 2026-07-14T17:42:59Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7002.magru.wmnet 2436564 wikitext text/x-wiki jpp5f42ea48942k4i3560enyy4qmdpa 2436565 2436564 2026-07-14T17:44:15Z Stashbot 7414 swfrench@deploy2003: Finished scap sync-world: Deployment to pick up new production image (duration: 31m 44s) 2436565 wikitext text/x-wiki 5g4tpo72wsquo32ullqajjhnav8s38z 2436567 2436565 2026-07-14T18:18:32Z Stashbot 7414 jhuneidi@deploy2003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.11 refs T430830 2436567 wikitext text/x-wiki hcpxgueiod0go9t55alvpbq1qxfl5ve 2436568 2436567 2026-07-14T18:38:09Z Stashbot 7414 mutante: rotating phabricator-gerrit bot token (its-phabricator) 2436568 wikitext text/x-wiki ckwb9sq7og4bppczyfuzs6zo4hqpdg8 2436569 2436568 2026-07-14T18:43:11Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7010.magru.wmnet 2436569 wikitext text/x-wiki pi6772awdp10l8nes9jep7o5wkkzt9r 2436570 2436569 2026-07-14T19:02:46Z Stashbot 7414 jforrester@deploy2003: Started scap sync-world: Backport for [[gerrit:1310634|wikifunctions: Define the virtual-wikifunctions-usage table (T390557)]] 2436570 wikitext text/x-wiki 65a1q9q2pa8hvloei444ns099jzq29j 2436571 2436570 2026-07-14T19:04:56Z Stashbot 7414 jforrester@deploy2003: jforrester: Backport for [[gerrit:1310634|wikifunctions: Define the virtual-wikifunctions-usage table (T390557)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436571 wikitext text/x-wiki jnubxmkfhx6yv1925pjqcqj545s1o0b 2436572 2436571 2026-07-14T19:07:01Z Stashbot 7414 jforrester@deploy2003: jforrester: Continuing with deployment 2436572 wikitext text/x-wiki rjd3oicbj610l01ooi86ca0hkhekmbp 2436573 2436572 2026-07-14T19:11:19Z Stashbot 7414 jforrester@deploy2003: Finished scap sync-world: Backport for [[gerrit:1310634|wikifunctions: Define the virtual-wikifunctions-usage table (T390557)]] (duration: 08m 33s) 2436573 wikitext text/x-wiki kdqs767atwouyxslzg3cew1wriovf2x 2436576 2436573 2026-07-14T19:19:36Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7003.magru.wmnet 2436576 wikitext text/x-wiki oqxt27q3x32vqplsb18vd5xoo5r8dfw 2436577 2436576 2026-07-14T19:24:47Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7011.magru.wmnet 2436577 wikitext text/x-wiki 7d7nvj9acquqa199evr2hbpoq5vmagf 2436578 2436577 2026-07-14T19:57:02Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1004.eqiad.wmnet 2436578 wikitext text/x-wiki 79aelwmj91ka8x3npvoq2c6svaz631s 2436579 2436578 2026-07-14T20:00:45Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1004.eqiad.wmnet 2436579 wikitext text/x-wiki raqhtn5szvj8882ez5gsaep7cr5kxqi 2436580 2436579 2026-07-14T20:01:00Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7004.magru.wmnet 2436580 wikitext text/x-wiki b47pqahyp2vwjtsppuvpbijng3ov2ro 2436581 2436580 2026-07-14T20:02:49Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.reimage for host phab1005.eqiad.wmnet with OS trixie 2436581 wikitext text/x-wiki 3hc27cn79xdxe3ec26a66tm7gzk2no0 2436582 2436581 2026-07-14T20:04:19Z Stashbot 7414 jhuneidi@deploy2003: Started scap sync-world: Backport for [[gerrit:1309865|thwiki: Change to Wikipedia 25 logo (T431094)]] 2436582 wikitext text/x-wiki d5xlmq6j9itd38j3r0nu4hl8qiavbw0 2436583 2436582 2026-07-14T20:06:16Z Stashbot 7414 jhuneidi@deploy2003: jhuneidi, priyankar22: Backport for [[gerrit:1309865|thwiki: Change to Wikipedia 25 logo (T431094)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436583 wikitext text/x-wiki 5d3rpwpx1ylnwkmnctjfz673v3sb83z 2436584 2436583 2026-07-14T20:06:30Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7012.magru.wmnet 2436584 wikitext text/x-wiki 1y8m4fbdv0jl1hbkewpn2px8r0gg6lq 2436585 2436584 2026-07-14T20:07:43Z Stashbot 7414 jhuneidi@deploy2003: jhuneidi, priyankar22: Continuing with deployment 2436585 wikitext text/x-wiki kbdc0mj1rqs84ziln6tcjgqih7dnz7w 2436586 2436585 2026-07-14T20:12:01Z Stashbot 7414 jhuneidi@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309865|thwiki: Change to Wikipedia 25 logo (T431094)]] (duration: 07m 42s) 2436586 wikitext text/x-wiki oxkc1f2mospzyu8fpnskwg3r3c3j5bb 2436587 2436586 2026-07-14T20:20:56Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on phab1005.eqiad.wmnet with reason: host reimage 2436587 wikitext text/x-wiki lga5jbab653f9s6pb9msc2xs1ppjtff 2436588 2436587 2026-07-14T20:21:47Z Stashbot 7414 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1310625|Add logging for various re-authentication methods (T432042)]] 2436588 wikitext text/x-wiki 133255nyeijecbuwdfo928mnsioez2s 2436589 2436588 2026-07-14T20:23:38Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1005.eqiad.wmnet with reason: host reimage 2436589 wikitext text/x-wiki fj9uh03aq2uzwmrcqjqmdvhnsh5puu5 2436590 2436589 2026-07-14T20:23:41Z Stashbot 7414 sbassett@deploy2003: sbassett: Backport for [[gerrit:1310625|Add logging for various re-authentication methods (T432042)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436590 wikitext text/x-wiki kugq1luq59ha026rckyrgqvhrh4tl2k 2436591 2436590 2026-07-14T20:24:16Z Stashbot 7414 sbassett@deploy2003: sbassett: Continuing with deployment 2436591 wikitext text/x-wiki sf6e1tae0hrlwx943hmzvj2ma3z8l67 2436592 2436591 2026-07-14T20:28:34Z Stashbot 7414 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1310625|Add logging for various re-authentication methods (T432042)]] (duration: 06m 47s) 2436592 wikitext text/x-wiki epesucatdiuuz2jmsmrk9d34nbufpgs 2436594 2436592 2026-07-14T20:41:46Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host phab1005.eqiad.wmnet with OS trixie 2436594 wikitext text/x-wiki 1h7udhnz05en4yfjynj1hcr8qdx8xlj 2436595 2436594 2026-07-14T20:42:23Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7005.magru.wmnet 2436595 wikitext text/x-wiki azq1gbyjlrl1ecyvar5wl1ne80gedq2 2436597 2436595 2026-07-14T20:48:13Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7013.magru.wmnet 2436597 wikitext text/x-wiki 10qj3qkl8p18kszd1nhx5fhazratmin 2436598 2436597 2026-07-14T20:55:45Z Stashbot 7414 arlolra@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2436598 wikitext text/x-wiki gxzj88f0z26nv3aecypzti8snd6gm9s 2436599 2436598 2026-07-14T20:56:18Z Stashbot 7414 arlolra@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2436599 wikitext text/x-wiki 0miyrfjoyfqab57blpaxgipfd7yjmi8 2436600 2436599 2026-07-14T20:56:19Z Stashbot 7414 arlolra@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2436600 wikitext text/x-wiki 7f5fsfhtw7ee0y5fm458ucsisde54c3 2436601 2436600 2026-07-14T20:56:45Z Stashbot 7414 arlolra@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2436601 wikitext text/x-wiki 7u3ygbngf3s9947j4kt5gpatksefoeu 2436602 2436601 2026-07-14T21:11:17Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on gerrit2003.wikimedia.org with reason: reboot 2436602 wikitext text/x-wiki 2uutfejofaqa9vpmu958mqkql8a2rsw 2436603 2436602 2026-07-14T21:11:39Z Stashbot 7414 mutante: gerrit2003 (gerrit.wikimedia.org) - reboot for maintenance 2436603 wikitext text/x-wiki dvradko5s5pv46blcxncgv9ywe97l8q 2436604 2436603 2026-07-14T21:13:12Z Stashbot 7414 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:15:00 on gerrit.wikimedia.org with reason: reboot 2436604 wikitext text/x-wiki jf22fn6796j97ulwa3gt74juc6m3xd4 2436605 2436604 2026-07-14T21:23:35Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7006.magru.wmnet 2436605 wikitext text/x-wiki s5gwf7xweg7eu7vvnqcv2h2axa01hmq 2436606 2436605 2026-07-14T21:29:49Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7014.magru.wmnet 2436606 wikitext text/x-wiki 76rwkxekvg4sx7slqhyfsgu524gxh02 2436607 2436606 2026-07-14T22:04:46Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7007.magru.wmnet 2436607 wikitext text/x-wiki i78wvqmo4thbe7gziln3bydz6wm1odo 2436608 2436607 2026-07-14T22:09:32Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7015.magru.wmnet 2436608 wikitext text/x-wiki 3r6y91e6u494rqku7msqp12o4v0ayo5 2436609 2436608 2026-07-14T22:46:01Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7008.magru.wmnet 2436609 wikitext text/x-wiki 0yqpcvxu6yy148t75zoic3cs1jrqhbg 2436610 2436609 2026-07-14T22:46:03Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_magru 2436610 wikitext text/x-wiki mttbqhf6uumbtl30rb183tnyi1c1yu7 2436611 2436610 2026-07-14T22:51:22Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet 2436611 wikitext text/x-wiki 4z0r353hjeeibve89lw5qgd43r23nkh 2436612 2436611 2026-07-14T22:51:24Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_magru 2436612 wikitext text/x-wiki 75218uljgafgej5wc95ox8xpveyfmly 2436614 2436612 2026-07-15T01:14:02Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2436614 wikitext text/x-wiki 3oeqzp4ml8ut8je0jeszq3ozy2xm9kn 2436615 2436614 2026-07-15T01:15:27Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2436615 wikitext text/x-wiki hzs3z7ot7t93a9vbbd4a371ml7d7xw6 2436616 2436615 2026-07-15T02:00:23Z Stashbot 7414 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image 2436616 wikitext text/x-wiki 8wic4o7ju0z7tnajvdi5r40pxd35hax 2436617 2436616 2026-07-15T02:07:00Z Stashbot 7414 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 37s) 2436617 wikitext text/x-wiki gl4v2983vuhn796bkhi866o94tml8bq 2436623 2436617 2026-07-15T07:06:09Z Stashbot 7414 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1310470|SiteStats: Add temporary account columns to site_stats (T339291)]] 2436623 wikitext text/x-wiki cg7o8mugypdi9v4bfrta2bpcz5zwmzf 2436624 2436623 2026-07-15T07:08:14Z Stashbot 7414 kharlan@deploy2003: kharlan: Backport for [[gerrit:1310470|SiteStats: Add temporary account columns to site_stats (T339291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436624 wikitext text/x-wiki kt7d2bhogppawv0911rd7762iq79y46 2436625 2436624 2026-07-15T07:09:34Z Stashbot 7414 kharlan@deploy2003: kharlan: Continuing with deployment 2436625 wikitext text/x-wiki ks6v22i082jnj2do6nrth03xpo2n2a2 2436626 2436625 2026-07-15T07:13:56Z Stashbot 7414 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1310470|SiteStats: Add temporary account columns to site_stats (T339291)]] (duration: 07m 48s) 2436626 wikitext text/x-wiki ejm5nl7sifs09nrs306uxzd6sna0b8z 2436627 2436626 2026-07-15T07:19:10Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2013.codfw.wmnet with OS trixie 2436627 wikitext text/x-wiki ck7lumbdw0jdksuu68tu2021npylcix 2436628 2436627 2026-07-15T07:19:28Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.move-vlan for host ms-fe2013 2436628 wikitext text/x-wiki poxecz6yurt2i1vdatj0pdxw1nhxqyu 2436629 2436628 2026-07-15T07:19:46Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.netbox 2436629 wikitext text/x-wiki o5iito1pd337w9m2961oagplvaqij91 2436630 2436629 2026-07-15T07:20:21Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1013.eqiad.wmnet with OS trixie 2436630 wikitext text/x-wiki n8awtbg4fv73elkhm7jijzrooqy35ik 2436631 2436630 2026-07-15T07:24:23Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2013 - mvernon@cumin2003" 2436631 wikitext text/x-wiki mdlanma4pkgj88l980mwpoxo79vku42 2436632 2436631 2026-07-15T07:24:27Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2013 - mvernon@cumin2003" 2436632 wikitext text/x-wiki q385yowe9rzhsmbbhyd5o65ywmiz051 2436633 2436632 2026-07-15T07:24:29Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436633 wikitext text/x-wiki 0np7wvws40ixhsbz906u9grfv0a2am1 2436634 2436633 2026-07-15T07:24:30Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.wipe-cache ms-fe2013.codfw.wmnet 87.0.192.10.in-addr.arpa 7.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436634 wikitext text/x-wiki jdhv1a4812wovda3gln5oc3zazaik3h 2436635 2436634 2026-07-15T07:24:32Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2013.codfw.wmnet 87.0.192.10.in-addr.arpa 7.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436635 wikitext text/x-wiki 3g0rvkp252qxzyeqm1qcx8mxhve2bp2 2436636 2436635 2026-07-15T07:24:33Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2013 2436636 wikitext text/x-wiki sfv0f42h01z6vskq4yzog1m85dmfdy6 2436637 2436636 2026-07-15T07:30:46Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Security updates 2436637 wikitext text/x-wiki ox0zagd3awnjgg5z4z6nwt78ktgeetg 2436638 2436637 2026-07-15T07:30:47Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436638 wikitext text/x-wiki aoua4xh8846wa90w3bj8y7sv82thkv2 2436639 2436638 2026-07-15T07:30:54Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436639 wikitext text/x-wiki n7qs066pq3j652p9mmpkvbtc2imu3ie 2436640 2436639 2026-07-15T07:30:56Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Security updates 2436640 wikitext text/x-wiki mhdk4j55o3xcpe2vxen06qoqtmafhuc 2436641 2436640 2026-07-15T07:31:29Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2013 2436641 wikitext text/x-wiki 5nd519zl9mnax3c0ej2wzrjnnp0si2s 2436642 2436641 2026-07-15T07:31:31Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2013 2436642 wikitext text/x-wiki bp6oxbggl6pyaeenn0xrlhixy8v9jbp 2436643 2436642 2026-07-15T07:36:45Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1013.eqiad.wmnet with reason: host reimage 2436643 wikitext text/x-wiki glsvsoyuoqw9s0nfljb3igrkxvi29x8 2436644 2436643 2026-07-15T07:40:14Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1013.eqiad.wmnet with reason: host reimage 2436644 wikitext text/x-wiki k0oaw1x2iu9nxknziglmcxo3j99b3cf 2436645 2436644 2026-07-15T07:46:43Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org 2436645 wikitext text/x-wiki a0gsxtj5hg5rnqdjv9vpzzoz9rwzo96 2436646 2436645 2026-07-15T07:47:14Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2013.codfw.wmnet with reason: host reimage 2436646 wikitext text/x-wiki 8al7auwkku1cnq4k47iyfvsd3jhqbmx 2436647 2436646 2026-07-15T07:52:53Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1004.wikimedia.org 2436647 wikitext text/x-wiki ia1zv4p818tmasvwruy4mng25f7b7x7 2436648 2436647 2026-07-15T07:53:00Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Security updates 2436648 wikitext text/x-wiki rev4sfoau1rek60aie33vf00jwsn017 2436649 2436648 2026-07-15T07:53:01Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436649 wikitext text/x-wiki pw8wbuubg4su4cbwhk5khrebpt5xa13 2436650 2436649 2026-07-15T07:53:14Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436650 wikitext text/x-wiki k9w50gulc7m6aa29xpy3829xr58ib9j 2436651 2436650 2026-07-15T07:53:15Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Security updates 2436651 wikitext text/x-wiki aan8se2xajbhdzowd5j74nwgr0b5qwd 2436652 2436651 2026-07-15T07:53:30Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2013.codfw.wmnet with reason: host reimage 2436652 wikitext text/x-wiki 4m93ees7wmipphluc9lxxv4pcbyh5cr 2436653 2436652 2026-07-15T07:58:45Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1013.eqiad.wmnet with OS trixie 2436653 wikitext text/x-wiki 3kp5snuifzyfr4z9tfi98797qj7wdiv 2436654 2436653 2026-07-15T07:58:55Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host bast5005.wikimedia.org 2436654 wikitext text/x-wiki 5sm9dd14727asf2zc7i63ltji1f8dfg 2436656 2436654 2026-07-15T08:05:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast5005.wikimedia.org 2436656 wikitext text/x-wiki bj7k8aq5j3a9i5u2arvpc6g71odpv1g 2436657 2436656 2026-07-15T08:06:13Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org 2436657 wikitext text/x-wiki h5ema0pypru4rs3ltqku65jxlz68udy 2436658 2436657 2026-07-15T08:07:56Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1012.eqiad.wmnet with OS trixie 2436658 wikitext text/x-wiki 4mijpnhztkvzxg5xsajd0dblk0h4qby 2436659 2436658 2026-07-15T08:09:19Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1022: Security updates 2436659 wikitext text/x-wiki afog9ob3kpugory0f52ptaehoki95ux 2436660 2436659 2026-07-15T08:09:21Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436660 wikitext text/x-wiki 9y5mp0fmendykg026wgdy5x344qje1y 2436661 2436660 2026-07-15T08:09:28Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436661 wikitext text/x-wiki sj1hdupszd3u7ebno1f2lyblb01s8ju 2436662 2436661 2026-07-15T08:09:29Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1022: Security updates 2436662 wikitext text/x-wiki ais39k4iwtehr8u5pvij5b7wglxxpa6 2436663 2436662 2026-07-15T08:12:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet 2436663 wikitext text/x-wiki kpqosppwxtbj540gx6y97vsk7k7nm2o 2436664 2436663 2026-07-15T08:12:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org 2436664 wikitext text/x-wiki twhhcxxk106hc5qi6k75i8p5p9i90kn 2436665 2436664 2026-07-15T08:13:24Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org 2436665 wikitext text/x-wiki a3gzure8we60k1a8nnohtya0qveajcj 2436666 2436665 2026-07-15T08:14:45Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2013.codfw.wmnet with OS trixie 2436666 wikitext text/x-wiki sb5y6bumyeph41vpft4qdgqenlaxtap 2436667 2436666 2026-07-15T08:19:30Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org 2436667 wikitext text/x-wiki toi4mq52zhimrvguevm8onwiwwbf3a7 2436668 2436667 2026-07-15T08:19:50Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet 2436668 wikitext text/x-wiki cwz64mdil44w4usazas5mfn31gxlljv 2436669 2436668 2026-07-15T08:22:00Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1012.eqiad.wmnet with reason: host reimage 2436669 wikitext text/x-wiki b4bhi67ff3h8rsj8c96znq5ie5w10h0 2436670 2436669 2026-07-15T08:26:01Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet 2436670 wikitext text/x-wiki 7d54wb8ejtsbzla5sfv3nmuxa6jjqfb 2436671 2436670 2026-07-15T08:26:50Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw 2436671 wikitext text/x-wiki eghu1qnlfd3n1eipmhco9hxs8a5ai45 2436672 2436671 2026-07-15T08:27:37Z Stashbot 7414 elukey@dns1004: START - running authdns-update 2436672 wikitext text/x-wiki lp16pe316wdr6p9vt7muouozudmdlu1 2436673 2436672 2026-07-15T08:28:26Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1022: Security updates 2436673 wikitext text/x-wiki 7grui1ctbket6exu7dew850vo66nkzi 2436674 2436673 2026-07-15T08:28:27Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436674 wikitext text/x-wiki kjbf0o61vhwygp7e1i8kd78t2hiabqj 2436675 2436674 2026-07-15T08:28:31Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436675 wikitext text/x-wiki btwl8pgrmkhe6l0s86x8n4isahh2dse 2436676 2436675 2026-07-15T08:28:33Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1022: Security updates 2436676 wikitext text/x-wiki i48j6koyu1sx92ucfbpj8quztogpg2q 2436677 2436676 2026-07-15T08:29:26Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1012.eqiad.wmnet with reason: host reimage 2436677 wikitext text/x-wiki dhn1m0e5go2ijtf6cpds96cfpab6dqn 2436678 2436677 2026-07-15T08:30:04Z Stashbot 7414 elukey@dns1004: END - running authdns-update 2436678 wikitext text/x-wiki medzupn6vv1wv5pz7vof99xz40woupm 2436682 2436678 2026-07-15T08:31:19Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet 2436682 wikitext text/x-wiki nqysf5iue7p63vzgz74w3wmtym9d6ej 2436683 2436682 2026-07-15T08:35:02Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw 2436683 wikitext text/x-wiki ryvn2zzp1e91fr83ecj40sd7ilmffpo 2436684 2436683 2026-07-15T08:41:03Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=ms-fe2013.codfw.wmnet 2436684 wikitext text/x-wiki qjnju1yidebhswhlyk10nnw0ajg753y 2436685 2436684 2026-07-15T08:42:12Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=ms-fe2013.codfw.wmnet 2436685 wikitext text/x-wiki b5cf5kyk5oezi67t0czcx5xc8cr6rgi 2436686 2436685 2026-07-15T08:43:20Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2012.codfw.wmnet with OS trixie 2436686 wikitext text/x-wiki g0ciyt81a1hdk0dsdnajlmy2n4xa6qv 2436687 2436686 2026-07-15T08:44:00Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.move-vlan for host ms-fe2012 2436687 wikitext text/x-wiki o557qslct5e771bmcsfifc3c1xzv4r5 2436688 2436687 2026-07-15T08:44:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.netbox 2436688 wikitext text/x-wiki ef685jcmbr2yqzhcu5ov9pxdzc44sfq 2436689 2436688 2026-07-15T08:47:37Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1012.eqiad.wmnet with OS trixie 2436689 wikitext text/x-wiki 3raigslgcg4akls6gk9xtf4y48aa6qc 2436690 2436689 2026-07-15T08:50:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2012 - mvernon@cumin2003" 2436690 wikitext text/x-wiki 0cerl31qtjdt3021a2d4y4rx34osrp8 2436691 2436690 2026-07-15T08:50:11Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2012 - mvernon@cumin2003" 2436691 wikitext text/x-wiki 1js3hcs67mhwxys26ahyb2allaujn21 2436692 2436691 2026-07-15T08:50:13Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436692 wikitext text/x-wiki iu36c7e5uyi56weubp1589birqezey9 2436693 2436692 2026-07-15T08:50:15Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.wipe-cache ms-fe2012.codfw.wmnet 44.48.192.10.in-addr.arpa 4.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436693 wikitext text/x-wiki dqpgtm7tuj1jow6f8qgpgdcwh8o4lks 2436694 2436693 2026-07-15T08:50:16Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2012.codfw.wmnet 44.48.192.10.in-addr.arpa 4.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436694 wikitext text/x-wiki 7y36d5tfkfht33co2xymd6sfdv33r59 2436695 2436694 2026-07-15T08:50:17Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2012 2436695 wikitext text/x-wiki 2leyj9jhpe9bu6t5ibiycut5g72g6t6 2436696 2436695 2026-07-15T08:50:56Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1022: Security updates 2436696 wikitext text/x-wiki jv16ns5txo2xwwzrf3s9es3rozafyhm 2436697 2436696 2026-07-15T08:50:57Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436697 wikitext text/x-wiki ip931vwmdzojec9kbnen8shptjkw434 2436698 2436697 2026-07-15T08:50:58Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2012 2436698 wikitext text/x-wiki egvyr8v0bw006ubtdz7kolhvx6iu3rh 2436699 2436698 2026-07-15T08:50:59Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2012 2436699 wikitext text/x-wiki f320ujrgktqpethbyuq0mrji4efa5at 2436700 2436699 2026-07-15T08:51:10Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436700 wikitext text/x-wiki l4ls33zdf6owlc4ahpxg7z9if8boz9a 2436701 2436700 2026-07-15T08:51:11Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1022: Security updates 2436701 wikitext text/x-wiki cossv3p6818x886eo209zclymetp23y 2436703 2436701 2026-07-15T08:52:03Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1011.eqiad.wmnet with OS trixie 2436703 wikitext text/x-wiki ptu5wj9qt8ln7pl4pop2l9m95zduqbi 2436704 2436703 2026-07-15T08:54:59Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436704 wikitext text/x-wiki nn85fnjy7mlmqxiovbh8g1okntwxclw 2436705 2436704 2026-07-15T08:55:50Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436705 wikitext text/x-wiki qtoca83mweqdmk5qpzzxux7yt8eqwy6 2436707 2436705 2026-07-15T09:05:54Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage 2436707 wikitext text/x-wiki 15oh9toan40vq1tryfwx1toied8n91s 2436709 2436707 2026-07-15T09:08:24Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage 2436709 wikitext text/x-wiki jgjtmovpj1gjeiobxvi84ey02pho4w1 2436710 2436709 2026-07-15T09:08:46Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage 2436710 wikitext text/x-wiki 8wmj63uzbiqtdiqs7nd5d9mxqyd0z0j 2436711 2436710 2026-07-15T09:12:44Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage 2436711 wikitext text/x-wiki n5yhkuaqxa4kgp8jxtjhqwcrjz1ke8d 2436714 2436711 2026-07-15T09:21:21Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Security updates 2436714 wikitext text/x-wiki r20a3iazbj21ijea11vej7z9bq2m9k1 2436715 2436714 2026-07-15T09:21:23Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436715 wikitext text/x-wiki 49wx4fzs4esfaybr5dooesfdyjor0kc 2436716 2436715 2026-07-15T09:21:29Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436716 wikitext text/x-wiki djxhgg3tdg50p676xdfn8f1pwhn1qvi 2436717 2436716 2026-07-15T09:21:31Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Security updates 2436717 wikitext text/x-wiki 73dxaof8y640mt82j2g48utv37qyrjv 2436718 2436717 2026-07-15T09:27:41Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1011.eqiad.wmnet with OS trixie 2436718 wikitext text/x-wiki 318fgsefdqooj95mf33wucgmn9sqyam 2436719 2436718 2026-07-15T09:30:17Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1010.eqiad.wmnet with OS trixie 2436719 wikitext text/x-wiki bnborqe8sp7ez2sku7z8cjw5pg4nn1j 2436720 2436719 2026-07-15T09:31:00Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2012.codfw.wmnet with OS trixie 2436720 wikitext text/x-wiki bs6n455y412i9xt73zi935yb4jnv4vx 2436721 2436720 2026-07-15T09:31:17Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw 2436721 wikitext text/x-wiki 9gqwxoq6pp9d0uq7q5biavg9tqwmpr3 2436722 2436721 2026-07-15T09:31:22Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=tegola-vector-tiles,name=eqiad 2436722 wikitext text/x-wiki fi10u82egzvvqrv7kt64s3joitxbezb 2436723 2436722 2026-07-15T09:31:30Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad 2436723 wikitext text/x-wiki 7ul4tqp7xdnm9jnjaop0yieulq9fz25 2436724 2436723 2026-07-15T09:32:19Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-codfw 2436724 wikitext text/x-wiki cfogo2amvt6igasdsq92woul2kfxque 2436725 2436724 2026-07-15T09:32:38Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2001-2002,2005-2006,2011-2016].codfw.wmnet 2436725 wikitext text/x-wiki n2ckoax2s5ab55uhrevlmj7txkm2n4m 2436726 2436725 2026-07-15T09:33:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet 2436726 wikitext text/x-wiki 7xb1honag2klibdkjmuq11izp3ik2xj 2436727 2436726 2026-07-15T09:36:38Z Stashbot 7414 blake@deploy2003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2436727 wikitext text/x-wiki mjmppzkkt3qccpco8kgzbsdtyyudn9s 2436728 2436727 2026-07-15T09:36:55Z Stashbot 7414 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2436728 wikitext text/x-wiki jw3h0pkbzt8m1dpg364z7bulfcgm4w2 2436730 2436728 2026-07-15T09:37:17Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet 2436730 wikitext text/x-wiki 10wo7b184nm46n9pbufd23vm9liomij 2436731 2436730 2026-07-15T09:40:45Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw 2436731 wikitext text/x-wiki 7hlishqol5plvrt6jsczt2wnjec5vet 2436732 2436731 2026-07-15T09:40:50Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow1002.eqiad.wmnet 2436732 wikitext text/x-wiki kal10l22tw11nre6tcudbx19p2dsn26 2436733 2436732 2026-07-15T09:42:15Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2001-2002,2005-2006,2011-2016].codfw.wmnet 2436733 wikitext text/x-wiki ra3p3iwipgxvz3c4vaghgdn5mxankvp 2436734 2436733 2026-07-15T09:43:22Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Security updates 2436734 wikitext text/x-wiki 2d5btmn3ii8pq37xh2sagi9g9skjpyj 2436735 2436734 2026-07-15T09:43:23Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436735 wikitext text/x-wiki 4kirmin6hq1n7z82lk12s8zxxvtu3ml 2436736 2436735 2026-07-15T09:43:35Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436736 wikitext text/x-wiki 9z4ykg33tensn9mkwr9icwfdesjt954 2436737 2436736 2026-07-15T09:43:37Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Security updates 2436737 wikitext text/x-wiki ezqryfx35m0iahnwd0oa4ii3uxjgh65 2436738 2436737 2026-07-15T09:43:43Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=ms-fe2012.codfw.wmnet 2436738 wikitext text/x-wiki mc05aj4d4sgnwuag86fazkrbxon9vm7 2436739 2436738 2026-07-15T09:44:10Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage 2436739 wikitext text/x-wiki cjexn3dwp0eufibdgktkd10g7ikl9or 2436740 2436739 2026-07-15T09:44:36Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1002.eqiad.wmnet 2436740 wikitext text/x-wiki rdufyyk5n6f1mzbc7g97kjcb40pgmdy 2436741 2436740 2026-07-15T09:44:51Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=ms-fe2012.codfw.wmnet 2436741 wikitext text/x-wiki e3giumcwun7mts0ylt1gfnrzfzrwg7q 2436742 2436741 2026-07-15T09:44:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow1003.eqiad.wmnet 2436742 wikitext text/x-wiki eji33e80xe4snw0ullhddfvf086irv4 2436743 2436742 2026-07-15T09:46:50Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2011.codfw.wmnet with OS trixie 2436743 wikitext text/x-wiki 5lv3mwh7uvvc668dznnepags1oq4wvm 2436744 2436743 2026-07-15T09:47:21Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.move-vlan for host ms-fe2011 2436744 wikitext text/x-wiki iw9qx8nn36sjerwy5ogea1rgze3hbfm 2436745 2436744 2026-07-15T09:47:27Z Stashbot 7414 blake@deploy2003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2436745 wikitext text/x-wiki m3v3flyfzdydc8lx9sou0cr2p61225a 2436746 2436745 2026-07-15T09:47:29Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.netbox 2436746 wikitext text/x-wiki i89mx2sp3vsqkj1q2vof6xb1uwd4niv 2436747 2436746 2026-07-15T09:47:37Z Stashbot 7414 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2436747 wikitext text/x-wiki hy1niewif4q491ca4wvz127hky6zd28 2436748 2436747 2026-07-15T09:49:10Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage 2436748 wikitext text/x-wiki 67dku6e9i27f2fgct1hv9c55vdv874u 2436749 2436748 2026-07-15T09:49:41Z Stashbot 7414 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2436749 wikitext text/x-wiki h26dww9ujsld399mrelegy6w5cncvpb 2436750 2436749 2026-07-15T09:49:57Z Stashbot 7414 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2436750 wikitext text/x-wiki d25tobe3tkf0y2zbcu4m1d2gci5mrww 2436751 2436750 2026-07-15T09:50:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1003.eqiad.wmnet 2436751 wikitext text/x-wiki 79vw1voi3psta3w7bn658chznfyze63 2436752 2436751 2026-07-15T09:51:08Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow2003.codfw.wmnet 2436752 wikitext text/x-wiki 6g8pt2pnz4b9vgo2b7ay3rb3hnr6i8g 2436753 2436752 2026-07-15T09:51:54Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2011 - mvernon@cumin2003" 2436753 wikitext text/x-wiki 6s2yh268br8zvrnc0upm1zq5kp11e3o 2436754 2436753 2026-07-15T09:51:58Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2011 - mvernon@cumin2003" 2436754 wikitext text/x-wiki gye2ved6oncjargl8krr3wjg3wh0ul9 2436755 2436754 2026-07-15T09:51:59Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436755 wikitext text/x-wiki sxhhmazm67g40zwts7lespn6btxcfno 2436756 2436755 2026-07-15T09:52:00Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.wipe-cache ms-fe2011.codfw.wmnet 36.32.192.10.in-addr.arpa 6.3.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436756 wikitext text/x-wiki bqe3aubyae8bjnl3rdi8c7zvowsxr9c 2436757 2436756 2026-07-15T09:52:03Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2011.codfw.wmnet 36.32.192.10.in-addr.arpa 6.3.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436757 wikitext text/x-wiki 84w1rwd6s22lx258i657ipgkx5hprsh 2436758 2436757 2026-07-15T09:52:04Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2011 2436758 wikitext text/x-wiki aebute24yweguyc3n4t3fqwgemjwll6 2436759 2436758 2026-07-15T09:52:38Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2001-2002,2005-2006,2011-2016].codfw.wmnet 2436759 wikitext text/x-wiki e5s3l11jil0enogr7fg9rk2b00p6949 2436760 2436759 2026-07-15T09:52:45Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2001-2002,2005-2006,2011-2016].codfw.wmnet 2436760 wikitext text/x-wiki o8o20gbvc1biujlnar6b6iocv83swlz 2436761 2436760 2026-07-15T09:53:04Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2017-2018,2033-2039,2041].codfw.wmnet 2436761 wikitext text/x-wiki bkpsjkxr6e3crgwuif8buuz7p3jzq1a 2436763 2436761 2026-07-15T09:54:24Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2011 2436763 wikitext text/x-wiki otk1cewfy4incompcraghs25arklk5w 2436764 2436763 2026-07-15T09:54:26Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2011 2436764 wikitext text/x-wiki fcic827zhv2us3d0cvdrlpfgtvefisp 2436765 2436764 2026-07-15T09:54:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2003.codfw.wmnet 2436765 wikitext text/x-wiki rjwc4phr5tiz2iespa3l76hqatfepbs 2436766 2436765 2026-07-15T09:59:21Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2017-2018,2033-2039,2041].codfw.wmnet 2436766 wikitext text/x-wiki bhasi2y5u0ynvo7utcsjcolumebv97n 2436767 2436766 2026-07-15T10:02:50Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow2004.codfw.wmnet 2436767 wikitext text/x-wiki 9smc3drv0zdee83z12ij9rg8xgngarf 2436768 2436767 2026-07-15T10:04:22Z Stashbot 7414 topranks: push out config change to BGP_outfilter on core routers T431849 2436768 wikitext text/x-wiki 585qq3947twmuquwd4i1j2pij2kiqx3 2436769 2436768 2026-07-15T10:06:01Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet 2436769 wikitext text/x-wiki nan8lua4pivyjty29by7kaazp1ssm92 2436770 2436769 2026-07-15T10:07:26Z Stashbot 7414 cgoubert@deploy2003: Started deploy [restbase/deploy@06301bd]: Deploying 1306088 1308347 - T429944 T428279 2436770 wikitext text/x-wiki 9zad0el4msvt9sbbmm3ks2jlkpdtadz 2436771 2436770 2026-07-15T10:08:09Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1010.eqiad.wmnet with OS trixie 2436771 wikitext text/x-wiki hquu1bo0w8bu7ae99kkop2n0iqapmuk 2436772 2436771 2026-07-15T10:08:10Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2004.codfw.wmnet 2436772 wikitext text/x-wiki 617nhsgk3e2leqo5uk8k977f8sa3yh4 2436773 2436772 2026-07-15T10:09:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow3004.esams.wmnet 2436773 wikitext text/x-wiki r3zvj0rqeqj0qi4mbcetcgzdzh5ix0r 2436774 2436773 2026-07-15T10:09:39Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2017-2018,2033-2039,2041].codfw.wmnet 2436774 wikitext text/x-wiki cs5fa27c72d87663r2gzbh3iciwjo5q 2436775 2436774 2026-07-15T10:09:45Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2017-2018,2033-2039,2041].codfw.wmnet 2436775 wikitext text/x-wiki 10d7dub6zojhevs6tbkmd9p6h74kf8g 2436776 2436775 2026-07-15T10:09:49Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet 2436776 wikitext text/x-wiki l2jwxiwb85sqjd7jcx8n0xz5ytkgb3e 2436777 2436776 2026-07-15T10:10:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet 2436777 wikitext text/x-wiki i80v979gf57mhjh9hoo9rg0xkntjjkx 2436778 2436777 2026-07-15T10:10:06Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042,2044,2046,2049-2051,2055-2058].codfw.wmnet 2436778 wikitext text/x-wiki 4rfehem6jvdu3vcv1eqoc1b60rl3z6q 2436779 2436778 2026-07-15T10:11:27Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage 2436779 wikitext text/x-wiki 49xd4leu3qmrq99nk7br7r6luf5drmz 2436780 2436779 2026-07-15T10:13:23Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3004.esams.wmnet 2436780 wikitext text/x-wiki 37ouplff428kzct1s4fna4lsax7v9r7 2436781 2436780 2026-07-15T10:14:13Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1017: Security updates 2436781 wikitext text/x-wiki 33aw6wlilgpk3ybpbpuq1sercuhfgmh 2436782 2436781 2026-07-15T10:14:14Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436782 wikitext text/x-wiki pz39mvmaimn6sy10sfsd2zg1vof56pz 2436783 2436782 2026-07-15T10:14:21Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436783 wikitext text/x-wiki rmfzyyhhzmkoibr0wgfw3nk1d2s2066 2436784 2436783 2026-07-15T10:14:23Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1017: Security updates 2436784 wikitext text/x-wiki ja6mpj653p7iab76ti4ybdx2bii574y 2436785 2436784 2026-07-15T10:15:20Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage 2436785 wikitext text/x-wiki fmc6ekhx3l0szn49jz7b1y4wufc12gi 2436786 2436785 2026-07-15T10:16:56Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042,2044,2046,2049-2051,2055-2058].codfw.wmnet 2436786 wikitext text/x-wiki jrlhu8i4h58cu05fgvpizyrqyvgm66b 2436787 2436786 2026-07-15T10:17:07Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet 2436787 wikitext text/x-wiki k9kvh35cqptmyj1mcmk9241higktmax 2436788 2436787 2026-07-15T10:21:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet 2436788 wikitext text/x-wiki o7pvqe3jmumpdj6xzry5c4ynrw9jvgm 2436789 2436788 2026-07-15T10:21:55Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet 2436789 wikitext text/x-wiki lqxbtvdtnd3y64guvztk8o9lvo8cxn1 2436790 2436789 2026-07-15T10:25:13Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2436790 wikitext text/x-wiki a5cnvksrd6o1lsyn3n0bpu5zww7lrbe 2436791 2436790 2026-07-15T10:25:23Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2436791 wikitext text/x-wiki 2kkjyucb61grzckfagurvnj53p7ulep 2436792 2436791 2026-07-15T10:25:55Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet 2436792 wikitext text/x-wiki fnj41y8x3r9w9bq5nbqh014yrrfcks6 2436793 2436792 2026-07-15T10:26:51Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2042,2044,2046,2049-2051,2055-2058].codfw.wmnet 2436793 wikitext text/x-wiki op5brv4wcxw9fpsrtgh29lydj163qt8 2436794 2436793 2026-07-15T10:26:57Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2042,2044,2046,2049-2051,2055-2058].codfw.wmnet 2436794 wikitext text/x-wiki iuhkg2p3sm6vrsidtci4fqljussbx1a 2436795 2436794 2026-07-15T10:27:09Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow5003.eqsin.wmnet 2436795 wikitext text/x-wiki 845qi26umwbq0hjiumoszktf7cvynem 2436796 2436795 2026-07-15T10:27:17Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2059-2062,2064-2065,2067-2070].codfw.wmnet 2436796 wikitext text/x-wiki j8iea45xw04v4oss48tz3sdnlk228oc 2436797 2436796 2026-07-15T10:28:11Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet 2436797 wikitext text/x-wiki 90zu51txgv2oqzsv079pmvkrktga40s 2436798 2436797 2026-07-15T10:28:25Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet 2436798 wikitext text/x-wiki 23792ce9tf54wqei8uazxy37d6a183c 2436799 2436798 2026-07-15T10:30:01Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1009.eqiad.wmnet with OS trixie 2436799 wikitext text/x-wiki 85xsr7dnm7xyxck6o4ui56oedzh2ozw 2436800 2436799 2026-07-15T10:33:30Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2059-2062,2064-2065,2067-2070].codfw.wmnet 2436800 wikitext text/x-wiki cwy14iym6wrpm1swph52e64ljibzs6g 2436801 2436800 2026-07-15T10:34:26Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2011.codfw.wmnet with OS trixie 2436801 wikitext text/x-wiki ktaimlk566v2ufgvsdqbi2ulnehrznu 2436802 2436801 2026-07-15T10:34:49Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5003.eqsin.wmnet 2436802 wikitext text/x-wiki 3vweu2x8f5xg0giw3h3tqi7zvd2pn90 2436803 2436802 2026-07-15T10:35:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw 2436803 wikitext text/x-wiki rt4wzsxitwdfmj7s9lmq6gcl0eq5uy1 2436804 2436803 2026-07-15T10:35:39Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet 2436804 wikitext text/x-wiki 4xdjubq0jw5gqgmx6f9p9l5v7uf1lw2 2436805 2436804 2026-07-15T10:36:00Z Stashbot 7414 cgoubert@deploy2003: Finished deploy [restbase/deploy@06301bd]: Deploying 1306088 1308347 - T429944 T428279 (duration: 28m 34s) 2436805 wikitext text/x-wiki 4mzpabsqv8da5v8h4dvc6hadlqindip 2436806 2436805 2026-07-15T10:38:27Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet 2436806 wikitext text/x-wiki prk7iiqmjka6ffkgs1ql7lh8399ypg6 2436807 2436806 2026-07-15T10:39:07Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet 2436807 wikitext text/x-wiki 3jqotxjy0bzx694s1k8dstl64q6hh76 2436808 2436807 2026-07-15T10:39:46Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1017: Security updates 2436808 wikitext text/x-wiki qp8xi35z5yeygkeelkb384qu7q7hoit 2436809 2436808 2026-07-15T10:39:47Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436809 wikitext text/x-wiki m3g237kdoq5zxflbw5x66mul2g5e7zi 2436810 2436809 2026-07-15T10:40:00Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436810 wikitext text/x-wiki sg4yn10mmvxxs0jy6z4t7rbv4zysk7m 2436811 2436810 2026-07-15T10:40:01Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1017: Security updates 2436811 wikitext text/x-wiki pjc3xc0fc01mxayutdr4o8ul0dsfpy9 2436812 2436811 2026-07-15T10:43:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet 2436812 wikitext text/x-wiki 5fqtteze6szlg3g014jzg2y15ryaoc1 2436813 2436812 2026-07-15T10:43:56Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2059-2062,2064-2065,2067-2070].codfw.wmnet 2436813 wikitext text/x-wiki mahl8vbyoz6f75wzgbpdmis4vdh8sat 2436814 2436813 2026-07-15T10:44:02Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2059-2062,2064-2065,2067-2070].codfw.wmnet 2436814 wikitext text/x-wiki iwexzq08txaqyg6wsyv9shrlizgdugx 2436815 2436814 2026-07-15T10:44:13Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1009.eqiad.wmnet with reason: host reimage 2436815 wikitext text/x-wiki iq0282ftbzmote45szhs455nb799c8u 2436816 2436815 2026-07-15T10:44:17Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw 2436816 wikitext text/x-wiki 6hthp8fd9f6nrvp9d0i854rmqgrmlw9 2436817 2436816 2026-07-15T10:44:35Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2071-2078,2087-2088].codfw.wmnet 2436817 wikitext text/x-wiki m9wyzq4cnl6o8kvh0gw7xap2ntvo6d0 2436818 2436817 2026-07-15T10:44:57Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=ms-fe2011.codfw.wmnet 2436818 wikitext text/x-wiki fzdio8oz5k388ua08ivm327v8820scs 2436819 2436818 2026-07-15T10:45:14Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet 2436819 wikitext text/x-wiki jxy7jss47wtmc0hskdqm2e62r9alirq 2436820 2436819 2026-07-15T10:46:05Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=ms-fe2011.codfw.wmnet 2436820 wikitext text/x-wiki dug7ee8fjrptj4h4b0hpj5g9rw18bhv 2436821 2436820 2026-07-15T10:49:00Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2010.codfw.wmnet with OS trixie 2436821 wikitext text/x-wiki 5xcdfbfs1uli6t2sfrymohmso4xbulz 2436822 2436821 2026-07-15T10:49:28Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1009.eqiad.wmnet with reason: host reimage 2436822 wikitext text/x-wiki d3zp2yhr283qio2voqk4q4esxv7337a 2436823 2436822 2026-07-15T10:49:30Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.move-vlan for host ms-fe2010 2436823 wikitext text/x-wiki 0mac4um3c3bdxgwaoaagbrbl0uvgyu3 2436824 2436823 2026-07-15T10:49:38Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.netbox 2436824 wikitext text/x-wiki cbmu2mdqv5u8g50gk5noxcf0af2ar15 2436825 2436824 2026-07-15T10:54:32Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2010 - mvernon@cumin2003" 2436825 wikitext text/x-wiki 0n6u3bdgjwq3ita0d9xli4r3tqmunby 2436826 2436825 2026-07-15T10:54:37Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2010 - mvernon@cumin2003" 2436826 wikitext text/x-wiki c8wta14n0xo6ijnn5ecv38t8cmbeoqe 2436827 2436826 2026-07-15T10:54:38Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436827 wikitext text/x-wiki rnl1ptx5ljb9lzs1dfaryu65ku2gfl7 2436828 2436827 2026-07-15T10:54:39Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.wipe-cache ms-fe2010.codfw.wmnet 76.16.192.10.in-addr.arpa 6.7.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436828 wikitext text/x-wiki ftvf77jwl7yuyxa44id62ully0xyz3t 2436829 2436828 2026-07-15T10:54:41Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2010.codfw.wmnet 76.16.192.10.in-addr.arpa 6.7.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436829 wikitext text/x-wiki 7teqs77sr92tzbdhznfp86mj2pamqzj 2436830 2436829 2026-07-15T10:54:42Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2010 2436830 wikitext text/x-wiki 7ykg8ncfgpuz7bxuqenl46qkjeurcmj 2436831 2436830 2026-07-15T10:54:44Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2071-2078,2087-2088].codfw.wmnet 2436831 wikitext text/x-wiki 8p49vj3u4lya8yrff7qixnrhdzn81ag 2436832 2436831 2026-07-15T10:54:46Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436832 wikitext text/x-wiki 8s19bztkq6dbm2d1nclc22j939gdspv 2436833 2436832 2026-07-15T10:54:52Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2010 2436833 wikitext text/x-wiki fkyk838z25z7xa56q5trk9enxp146c2 2436834 2436833 2026-07-15T10:54:53Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2010 2436834 wikitext text/x-wiki 7tm0sucef25k17tax8jxrnxcfbptrap 2436835 2436834 2026-07-15T10:54:57Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436835 wikitext text/x-wiki 3f3sfefeq2rctqdvm4652tycnoy1aa1 2436837 2436835 2026-07-15T11:00:03Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet 2436837 wikitext text/x-wiki 6kx586psq2wlthbv926cq2t6ltcnvpu 2436838 2436837 2026-07-15T11:01:32Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad 2436838 wikitext text/x-wiki t034aa1zclbzuikt1vf160tanxrqisl 2436839 2436838 2026-07-15T11:01:42Z Stashbot 7414 elukey@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=tegola-vector-tiles,name=eqiad 2436839 wikitext text/x-wiki lfonlb4qm9grq7gp033x9qbudba0fge 2436840 2436839 2026-07-15T11:02:16Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2071-2078,2087-2088].codfw.wmnet 2436840 wikitext text/x-wiki 0t2mxkneze787qz9j06qitw53yrrz2l 2436841 2436840 2026-07-15T11:02:23Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2071-2078,2087-2088].codfw.wmnet 2436841 wikitext text/x-wiki qsf7j6g8v4d51wc2ijqx3pwsr5vih74 2436842 2436841 2026-07-15T11:02:47Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2089-2095,2102-2104].codfw.wmnet 2436842 wikitext text/x-wiki oout9zysi3jdwskoj0ycidg0pvo6t6h 2436843 2436842 2026-07-15T11:05:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet 2436843 wikitext text/x-wiki kg3dbcxgjzfutvio83cpofggex1824o 2436844 2436843 2026-07-15T11:09:38Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1009.eqiad.wmnet with OS trixie 2436844 wikitext text/x-wiki 0bntmt0d4zi3cocbomi6xpxwyxn79gq 2436845 2436844 2026-07-15T11:10:13Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1018: Security updates 2436845 wikitext text/x-wiki t8jtpxlfc2wrg33tuuy2wh34ks6ejyc 2436846 2436845 2026-07-15T11:10:14Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436846 wikitext text/x-wiki 7md1srmq4oi06gabyy6s0ig3rhatylb 2436847 2436846 2026-07-15T11:10:21Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436847 wikitext text/x-wiki 99ky0sd11x6doylskwv4k94mm5iu61r 2436848 2436847 2026-07-15T11:10:22Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1018: Security updates 2436848 wikitext text/x-wiki cwcihp6e869kurzyi66gl9dcrssq67k 2436849 2436848 2026-07-15T11:12:20Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage 2436849 wikitext text/x-wiki j2ykjl6n1tozyl4r6177mb0ifleryje 2436850 2436849 2026-07-15T11:12:55Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2089-2095,2102-2104].codfw.wmnet 2436850 wikitext text/x-wiki l5jcq7b7l8rvh2e885qpx1lzk1wylmn 2436851 2436850 2026-07-15T11:18:50Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage 2436851 wikitext text/x-wiki ixxk6kqk0828a68q89l53rjwtvs6eaa 2436852 2436851 2026-07-15T11:20:17Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2089-2095,2102-2104].codfw.wmnet 2436852 wikitext text/x-wiki 03g3wj6l11osg5zyriyf7v194frvm7x 2436853 2436852 2026-07-15T11:20:23Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2089-2095,2102-2104].codfw.wmnet 2436853 wikitext text/x-wiki r4ip0wwix7mioo2z3rnhx8rxor4pa1y 2436854 2436853 2026-07-15T11:20:42Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2105-2114].codfw.wmnet 2436854 wikitext text/x-wiki jsd6nznqd62trkd7ujcioujjtgpds24 2436855 2436854 2026-07-15T11:26:22Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2105-2114].codfw.wmnet 2436855 wikitext text/x-wiki k0fiqjeyksocsy4vaewfa934abatjy8 2436856 2436855 2026-07-15T11:29:14Z Stashbot 7414 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1224074|Switch math rendering for group0 from native to mathjax (T413973)]] 2436856 wikitext text/x-wiki k5hb46j0l0uyfrmrjpzblmgrmk0k7rj 2436857 2436856 2026-07-15T11:31:08Z Stashbot 7414 krinkle@deploy2003: physikerwelt, krinkle: Backport for [[gerrit:1224074|Switch math rendering for group0 from native to mathjax (T413973)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436857 wikitext text/x-wiki 8fdz7sgfl2jdfkyjwzzy8yodqn6uo9f 2436858 2436857 2026-07-15T11:36:09Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1018: Security updates 2436858 wikitext text/x-wiki 8wcg92e1iv5vy5drrqbsmvbf3ori4jf 2436859 2436858 2026-07-15T11:36:10Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2436859 wikitext text/x-wiki ovq2vl8zlu98wngfltgf2i5cv33idjk 2436860 2436859 2026-07-15T11:36:23Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2436860 wikitext text/x-wiki e6kc3s84xgfwy3sjnupaid6xtjvsrxs 2436861 2436860 2026-07-15T11:36:25Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1018: Security updates 2436861 wikitext text/x-wiki g4e0ynoe9lg04wj04awt0o8u6eumq06 2436862 2436861 2026-07-15T11:36:35Z Stashbot 7414 krinkle@deploy2003: physikerwelt, krinkle: Continuing with deployment 2436862 wikitext text/x-wiki 3z74uqu79mnqyqtpcixfoyg22g7dh16 2436863 2436862 2026-07-15T11:36:40Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2105-2114].codfw.wmnet 2436863 wikitext text/x-wiki b4qcymq9x3q42wngd31w47fzf7fpc2o 2436864 2436863 2026-07-15T11:36:46Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2105-2114].codfw.wmnet 2436864 wikitext text/x-wiki 7eb6l2yw4t7xy8o6qodlm8x03r4ehxz 2436865 2436864 2026-07-15T11:37:05Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2115,2124-2132].codfw.wmnet 2436865 wikitext text/x-wiki iam461ol3dj9hzgh9empier9na90o0p 2436866 2436865 2026-07-15T11:37:56Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2010.codfw.wmnet with OS trixie 2436866 wikitext text/x-wiki o32n10bz0mb1np12iucxf19eeht4ib1 2436867 2436866 2026-07-15T11:40:53Z Stashbot 7414 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1224074|Switch math rendering for group0 from native to mathjax (T413973)]] (duration: 11m 38s) 2436867 wikitext text/x-wiki qjlwamwlnxir11crlt852o3x7v9t49d 2436868 2436867 2026-07-15T11:43:17Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2115,2124-2132].codfw.wmnet 2436868 wikitext text/x-wiki eubbq7fqex7hsd12c1lxddv18u8gmw8 2436869 2436868 2026-07-15T11:44:57Z Stashbot 7414 mvolz@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply 2436869 wikitext text/x-wiki hkgtaoh5u9ogb8288w7gr306bytrshf 2436870 2436869 2026-07-15T11:45:22Z Stashbot 7414 mvolz@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2436870 wikitext text/x-wiki h0si4nsp38ewntdi8yvhj0fml84w9kp 2436871 2436870 2026-07-15T11:46:28Z Stashbot 7414 mvolz@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply 2436871 wikitext text/x-wiki d97y83z561c9xn7bm1pexqyq6zz8pjq 2436872 2436871 2026-07-15T11:46:53Z Stashbot 7414 mvolz@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply 2436872 wikitext text/x-wiki 848cfx6hsm81fae41gj50xw6gixbbsj 2436873 2436872 2026-07-15T11:47:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes 2436873 wikitext text/x-wiki fd9uoniv6n6puhzdn72z2gxy8kt7f9q 2436874 2436873 2026-07-15T11:48:50Z Stashbot 7414 mvolz@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply 2436874 wikitext text/x-wiki jgaqaynnkdsoyz1zfnbgpz7zd52520x 2436875 2436874 2026-07-15T11:49:06Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw 2436875 wikitext text/x-wiki 5ize7fgqlvvqw9k74574oz0u5js7vx6 2436876 2436875 2026-07-15T11:49:23Z Stashbot 7414 mvolz@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply 2436876 wikitext text/x-wiki ohrq961ljx5ji51pupyyhiet0x2v7pg 2436877 2436876 2026-07-15T11:51:17Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2115,2124-2132].codfw.wmnet 2436877 wikitext text/x-wiki r0fk6lm2k718ix9tl9gy1p1vy0xj1vs 2436878 2436877 2026-07-15T11:51:23Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2115,2124-2132].codfw.wmnet 2436878 wikitext text/x-wiki du0rmwy3tkktj42ugkqj5f33spsa9el 2436879 2436878 2026-07-15T11:51:43Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2133-2142].codfw.wmnet 2436879 wikitext text/x-wiki pi0sjaob53v8wh4h2czdxmtw9z16nfe 2436880 2436879 2026-07-15T11:57:25Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2133-2142].codfw.wmnet 2436880 wikitext text/x-wiki 31um2tidxs9k2yae7qqc003ty6m4nzl 0 17290 2436712 2436270 2026-07-15T09:13:13Z Stashbot 7414 James_F: Docker: [php83] Update PHP to 8.3.32 for T431099 2436712 wikitext text/x-wiki qj6c4leszlu207qx61o3xm4tcbwldqi 498 30942 2436311 2436282 2026-07-14T12:12:32Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048' (T431682) 2436311 wikitext text/x-wiki 5g3v4b1pg8y3s51q5ydnzyaoze6ccp9 2436481 2436311 2026-07-14T14:31:02Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::cloudweb}' 2436481 wikitext text/x-wiki 4ibouxdmf1w12rzpigp88wa8yl325g9 2436484 2436481 2026-07-14T14:36:10Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::cloudweb}' 2436484 wikitext text/x-wiki nr29vywjvdh9smblyvtsaqu2lvi1kec 498 443628 2436655 2436112 2026-07-15T08:00:31Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29399091032 (https://github.com/cluebotng/component-configs/commits/5880b5b641021f572e9dd6a3d76443db4a607114) 2436655 wikitext text/x-wiki sjq9x5b3vjxygynoshagitbqiwhs32f 0 444677 2436500 2434700 2026-07-14T14:57:57Z BJensen-WMF 48942 /* Service ports */ Reserve port 4457 for mw-pretrain. 2436500 wikitext text/x-wiki 3pmdc3t7dnng50plqzi5wq0s16u0lqv 0 448732 2436553 2435973 2026-07-14T16:51:03Z LSandergreen-WMF 46345 /* Fail Mail Storm - lots of different jobs failing at once */ long running upgrader reads 2436553 wikitext text/x-wiki eu578emvgced6nxbbr7trk59headsg5 0 449160 2436613 2436118 2026-07-15T00:01:39Z Dexbot 30554 Bot: Updating the report 2436613 wikitext text/x-wiki 7v2fb6cg8b1vjz5496lnvmhn654yh1h 0 451159 2436439 2005999 2026-07-14T13:34:49Z Jynus 38174 /* Check disk rebuilding status into the RAID */ 2436439 wikitext text/x-wiki kzrjnmysoshgj5wh44flnlvkry5uhah 2436442 2436439 2026-07-14T13:36:21Z Jynus 38174 /* Check disk rebuilding status into the RAID */ style and fix mistake 2436442 wikitext text/x-wiki gud0s7i4ayyzxfzqca3p73fxxex5mh1 0 452383 2436501 2403397 2026-07-14T14:59:23Z HNowlan (WMF) 15760 Add emergency compensation details. 2436501 wikitext text/x-wiki tse3w1977r9tng7vldzio5a60em971d 0 454000 2436574 2375797 2026-07-14T19:11:58Z BCornwall-WMF 31521 Rename configuration section to "Management", add section for rebooting 2436574 wikitext text/x-wiki ruao0lz3zp60e7q2t0bz6v7z442y3dm 2 455839 2436679 2433548 2026-07-15T08:30:22Z MABot 10804 Bot: Fixing broken redirect from User:MX to moved target page [[User:DreamRimmer]] 2436679 wikitext text/x-wiki b2wtjxrx4uuxpckj8upiautic83sdd2 3 455840 2436680 2433549 2026-07-15T08:30:33Z MABot 10804 Bot: Fixing broken redirect from User talk:MX to moved target page [[User talk:DreamRimmer]] 2436680 wikitext text/x-wiki 456sjnpuja4gpn42wwh8vf7knf7l4g9 0 458854 2436494 2313017 2026-07-14T14:48:14Z DCaro (WMF) 18271 added a note about the source of emails in toolsbeta 2436494 wikitext text/x-wiki r7295tvjt0pyb1e0v3npbnp0oqqz0ld 2436762 2436494 2026-07-15T09:53:44Z DCaro (WMF) 18271 added note of what first to do in toolsbeta 2436762 wikitext text/x-wiki hm53s1jsgpaiiwlx4dpm116irw1uqne 0 459560 2436560 2432051 2026-07-14T17:17:20Z DAdedoyin (WMF) 39150 Test 2436560 wikitext text/x-wiki ckaumasdqv56cwm83ibkir26tv6osfa 0 459699 2436618 2435992 2026-07-15T03:44:34Z Stashbot 7414 cstone: payments-wiki upgraded from 776f7d37 to 5482106f 2436618 wikitext text/x-wiki a21rdib0sotac4pjpd31uqaf0r1jrs4 2436619 2436618 2026-07-15T04:30:23Z Stashbot 7414 eileen: SmashPig upgraded from 87dfbd6b to 34082cca 2436619 wikitext text/x-wiki enguv0txfzrtapllnuilkotkyjuhoxq 2436620 2436619 2026-07-15T04:58:01Z Stashbot 7414 eileen: civicrm upgraded from 24d6e9a0 to d38ae4d6 2436620 wikitext text/x-wiki 2q32u8z1vttixkq14kqmdfk9uefghf2 2 460039 2436335 2436274 2026-07-14T12:25:40Z Atsuko-WMF 54082 /* Service-related changes */ better diffing 2436335 wikitext text/x-wiki 06jwxwrhtyoy933k93zqpviuzduw7bz 116 460070 2436836 2435507 2026-07-15T10:59:05Z EtherpadBackupBot 54504 mathml-mediawiki-2025 archived by Krinkle 2436836 wikitext text/x-wiki 0xkcyd9voctm2rf2zgkqbceb35lfd0i 1 460409 2436506 2435981 2026-07-14T15:05:08Z AWesterinen-WMF 54189 /* named subqueries */ Reply 2436506 wikitext text/x-wiki 5onkyb2k2ibqouq7n7p4cdl4tx5xa7d 2436729 2436506 2026-07-15T09:37:00Z Udehb-WMF 40856 /* non-replacement of label service ? */ Reply 2436729 wikitext text/x-wiki stibdpxn1hnzzf0pt6ou3s6kf0vu8u5 3 460411 2436681 2433171 2026-07-15T08:30:42Z MABot 10804 Bot: Fixing broken redirect from User talk:Velorak to moved target page [[User talk:Renamed user 86b7651314b97c07bfb294c41f05b19d]] 2436681 wikitext text/x-wiki jada89p9ty2om3r65udc5x79lrst0m5 0 460424 2436396 2435729 2026-07-14T13:03:26Z DMburugu (WMF) 25135 Add basic details on service and component identity 2436396 wikitext text/x-wiki 16103x311tq87ec3fahl5k9fphvg3ko 2436397 2436396 2026-07-14T13:04:40Z DMburugu (WMF) 25135 2436397 wikitext text/x-wiki lgonfj7f2kytw60l69v60o7x2v4sxkd 3 460446 2436621 2026-07-15T05:40:37Z StrikerBot 8475 Welcome to Toolforge! 2436621 wikitext text/x-wiki 1p7yx4ckytmb9bgp4nzxuuawj0pkyus