Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.19 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 739 2455781 2454950 2026-09-08T18:33:23Z Stashbot 7414 andrewbogott: shutting down dumps-10 as there is no response on T435767 2455781 wikitext text/x-wiki 7bdtct5yi7c9iwb3ylnlpse9a21n2iy 498 3086 2455705 2455338 2026-09-08T14:13:49Z Stashbot 7414 komla@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T436614) 2455705 wikitext text/x-wiki mbard07divph4bj0e996qwegdhtuxpb 2455706 2455705 2026-09-08T14:13:53Z Stashbot 7414 komla@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers (T436614) 2455706 wikitext text/x-wiki 1orp39kl33ldzhsmwabf0c4nka9b5uf 2455708 2455706 2026-09-08T14:19:03Z Stashbot 7414 komla@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T436614) 2455708 wikitext text/x-wiki jt2087cobqgke05kkz0x85ktgpw93a4 2455717 2455708 2026-09-08T14:40:21Z Stashbot 7414 komla@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T436614) 2455717 wikitext text/x-wiki j2spcggrvvgh4tbf0qfxnu9dizaavtu 0 4039 2455824 2450686 2026-09-08T19:51:07Z WBrown (WMF) 37696 update per discussion with SBasset on a security task 2455824 wikitext text/x-wiki ntv72jzectd633nttfed120nmq4j2b4 0 4108 2455763 2455447 2026-09-08T17:47:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1334945]] to Wednesday, September 09 UTC afternoon backport window 2455763 wikitext text/x-wiki rj8vya0lk54185eug69ydzqxv0qu8un 2455764 2455763 2026-09-08T17:48:13Z ScheduleDeploymentBot 37566 Add [[gerrit:1337983]] to Wednesday, September 09 UTC afternoon backport window 2455764 wikitext text/x-wiki 9mmilva8e4ffsgeq9fnat175yd9edcm 2455765 2455764 2026-09-08T18:03:31Z ScheduleDeploymentBot 37566 Add [[gerrit:1326425]] to Tuesday, September 08 UTC late backport window 2455765 wikitext text/x-wiki 86kxfa7kdq5ri76uy1vsmcshd3lpjr9 2455787 2455765 2026-09-08T18:52:45Z ScheduleDeploymentBot 37566 Add [[gerrit:1337997]] to Tuesday, September 08 UTC late backport window 2455787 wikitext text/x-wiki mmb6ggsd3srp9egi1gx5vc7lzenqeym 2455789 2455787 2026-09-08T18:53:17Z ScheduleDeploymentBot 37566 Add [[gerrit:1337998]] to Tuesday, September 08 UTC late backport window 2455789 wikitext text/x-wiki cgs7mp3b8x39v8049qv9eg7o7x4ph9s 2455790 2455789 2026-09-08T18:53:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1337996]] to Tuesday, September 08 UTC late backport window 2455790 wikitext text/x-wiki ms1haurstgcfx6xpkwmm3vsyvr08cte 2455794 2455790 2026-09-08T19:10:06Z ScheduleDeploymentBot 37566 Add [[gerrit:1335706]] to Tuesday, September 08 UTC late backport window 2455794 wikitext text/x-wiki ju4jg9n9snkzey99aqner6l6zu4rmyb 2455938 2455794 2026-09-09T04:03:30Z ScheduleDeploymentBot 37566 Add [[gerrit:1335706]] to Wednesday, September 09 UTC morning backport window 2455938 wikitext text/x-wiki 9jdxzh2ixzcli6aysb1q9w8zt0kpmgs 2455994 2455938 2026-09-09T11:08:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1338151]] to Wednesday, September 09 UTC afternoon backport window 2455994 wikitext text/x-wiki nhz4prpqvlu83fc7x7o29leeo2abezj 0 4673 2455997 2452536 2026-09-09T11:13:50Z Taavi 13997 /* 2025-present */ update archive mapping 2455997 wikitext text/x-wiki iag4pga6ipva89qrz9f51nl8fi1jfg6 0 7919 2455545 2455544 2026-09-08T12:00:46Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2010.codfw.wmnet with OS bookworm 2455545 wikitext text/x-wiki kxqsew7k3un87s1umfpq75fvt2ehbmk 2455546 2455545 2026-09-08T12:01:13Z Stashbot 7414 zabe@deploy1003: zabe, somerandomdeveloper: Backport for [[gerrit:1337898|Skip DB read-only check for CentralAuthSessionManager (T437273)]], [[gerrit:1337899|Skip DB read-only check for CentralAuthSessionManager (T437273)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455546 wikitext text/x-wiki gicubm973zeiiaxuszmv0zbqhz6s3x0 2455547 2455546 2026-09-08T12:01:40Z Stashbot 7414 zabe@deploy1003: zabe, somerandomdeveloper: Continuing with deployment 2455547 wikitext text/x-wiki t2u9k3cebq8g0wbc6ap5ag2gqi1cz1u 2455548 2455547 2026-09-08T12:01:49Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reboot-single for host aqs2011.codfw.wmnet 2455548 wikitext text/x-wiki rol68xpo5z1n7ff19a847mbsshk3x58 2455549 2455548 2026-09-08T12:06:30Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337898|Skip DB read-only check for CentralAuthSessionManager (T437273)]], [[gerrit:1337899|Skip DB read-only check for CentralAuthSessionManager (T437273)]] (duration: 09m 54s) 2455549 wikitext text/x-wiki 9dvuzaaj6vx0zea67impi0xpm1ks0d2 2455550 2455549 2026-09-08T12:09:18Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host aqs2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2455550 wikitext text/x-wiki i5axszdh37ryjne9jwo2i8zfxsiv31t 2455551 2455550 2026-09-08T12:10:49Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2011.codfw.wmnet 2455551 wikitext text/x-wiki icrur53kqriy2apdgfys0jb1hho0xk9 2455552 2455551 2026-09-08T12:10:51Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2011.codfw.wmnet 2455552 wikitext text/x-wiki czj1ulkwivnxs07s034op6yjagjk3zv 2455553 2455552 2026-09-08T12:11:04Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2455553 wikitext text/x-wiki h5hwuxr0mcpprfpsmsypzlrrj071vdt 2455554 2455553 2026-09-08T12:11:48Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host aqs2011.codfw.wmnet with OS bookworm 2455554 wikitext text/x-wiki ny4ie26sfm54m6nmc0kkq61ar1uejv1 2455555 2455554 2026-09-08T12:11:50Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2012.codfw.wmnet 2455555 wikitext text/x-wiki 9x0aynybc4kb4ca5sbc7btrpncd1jc9 2455556 2455555 2026-09-08T12:22:13Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad 2455556 wikitext text/x-wiki mhaama5bmweulikzr52mi3wzqlv58jh 2455557 2455556 2026-09-08T12:22:17Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet 2455557 wikitext text/x-wiki lc3cfmrofuf5jptoezv4ar00j0rjism 2455558 2455557 2026-09-08T12:31:50Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2011.codfw.wmnet with reason: host reimage 2455558 wikitext text/x-wiki 5s0d5fp3thv17ot2v02mj0fe1x8dk3d 2455559 2455558 2026-09-08T12:32:19Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet 2455559 wikitext text/x-wiki e4qbalomvga1un2xfdaqrxrfgtjfmik 2455560 2455559 2026-09-08T12:36:59Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2011.codfw.wmnet with reason: host reimage 2455560 wikitext text/x-wiki swt09hfyu0ujalhme5pp92reeztmywe 2455561 2455560 2026-09-08T12:38:50Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet 2455561 wikitext text/x-wiki pd7x0e4ipleu0bvncg80jxxkoaq67dx 2455562 2455561 2026-09-08T12:38:51Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet 2455562 wikitext text/x-wiki qobatg2bwm8xna9ulibjvtm8r6lnlb0 2455563 2455562 2026-09-08T12:38:57Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet 2455563 wikitext text/x-wiki ard83r0gewu4msyl1y7w99qigt2fsov 2455564 2455563 2026-09-08T12:40:15Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cloudsw1-d5-eqiad 2455564 wikitext text/x-wiki hgssy2zgjzmtb6qv0fonng8esv4hzax 2455565 2455564 2026-09-08T12:40:28Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-d5-eqiad 2455565 wikitext text/x-wiki 8rxwy1vp56s8pjsu29wpby3njp303yp 2455566 2455565 2026-09-08T12:41:23Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f1-eqiad 2455566 wikitext text/x-wiki 90w9j4orjk1aftg56272re4jd1k1fko 2455567 2455566 2026-09-08T12:41:30Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-eqiad 2455567 wikitext text/x-wiki rgm06bhoxs1odzg54oq5mt1jrhu70ly 2455568 2455567 2026-09-08T12:42:02Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr2-eqiad 2455568 wikitext text/x-wiki q7qq41u8bp1ujaibw9itkzp9zd52vag 2455569 2455568 2026-09-08T12:42:09Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqiad 2455569 wikitext text/x-wiki rwcza6d9k0jz6zteqan905y84m0gztj 2455570 2455569 2026-09-08T12:42:32Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr1-eqiad 2455570 wikitext text/x-wiki qsp61auor4948ksw9rgy9bj9c2mstw8 2455571 2455570 2026-09-08T12:42:44Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-eqiad 2455571 wikitext text/x-wiki 8kdzkia2p7pzmvf6ofnpqrsjsucsxk4 2455572 2455571 2026-09-08T12:42:55Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cloudsw1-b1-codfw 2455572 wikitext text/x-wiki idk86z13vcfbrl3q9blfvlat3mmmdu4 2455573 2455572 2026-09-08T12:43:03Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-b1-codfw 2455573 wikitext text/x-wiki h78f5qdp73uaouqooj76ucpzqfudjf1 2455574 2455573 2026-09-08T12:43:59Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet 2455574 wikitext text/x-wiki fkqyn0s9875f1fempi6xpd8atfxpn9e 2455575 2455574 2026-09-08T12:44:54Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-e1-eqiad 2455575 wikitext text/x-wiki 7o0n3xbfaqn6zqijm6ogh4wgnyszkzf 2455576 2455575 2026-09-08T12:45:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e1-eqiad 2455576 wikitext text/x-wiki 55srraz83vtpddans2vxazartz92n8s 2455577 2455576 2026-09-08T12:45:20Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cloudsw1-e4-eqiad 2455577 wikitext text/x-wiki ic4nqwar78l8bglcs72zmc0r6vmdm9n 2455578 2455577 2026-09-08T12:45:27Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-e4-eqiad 2455578 wikitext text/x-wiki eh1rf3ggqq95bci6xd28369r47pk13o 2455579 2455578 2026-09-08T12:45:51Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-e2-eqiad 2455579 wikitext text/x-wiki ogtzo0e3pmd1m1uk7rqctmgwswcrh0j 2455580 2455579 2026-09-08T12:45:58Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e2-eqiad 2455580 wikitext text/x-wiki nnd7tgxky23ywrm1kgk3qcz9a01kafd 2455581 2455580 2026-09-08T12:49:21Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cloudsw1-c8-eqiad 2455581 wikitext text/x-wiki nltf95832us8o5ph82khsqh7m3682fc 2455582 2455581 2026-09-08T12:49:33Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-c8-eqiad 2455582 wikitext text/x-wiki j485tw36to5uyh4juv170ur6mggakuy 2455583 2455582 2026-09-08T12:49:37Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-e1-eqiad 2455583 wikitext text/x-wiki iyxn8h9a66veo015djpqwn6larorqjl 2455584 2455583 2026-09-08T12:49:38Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e1-eqiad 2455584 wikitext text/x-wiki ei1x4jujyytafvqn49hqvyagosp1fyx 2455585 2455584 2026-09-08T12:49:40Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cloudsw1-e4-eqiad 2455585 wikitext text/x-wiki frkfelcqu26e916elsmrju0jqii8tw4 2455586 2455585 2026-09-08T12:49:41Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-e4-eqiad 2455586 wikitext text/x-wiki hzjlb86usj0ubrasbo0s6jwutbdnd4d 2455587 2455586 2026-09-08T12:49:43Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-e2-eqiad 2455587 wikitext text/x-wiki cjitew4um39u0occdd0uti65ljime9b 2455588 2455587 2026-09-08T12:49:43Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e2-eqiad 2455588 wikitext text/x-wiki q3bhcqc58ifgu61740wqxs0wpq44aue 2455589 2455588 2026-09-08T12:49:46Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cloudsw1-f4-eqiad 2455589 wikitext text/x-wiki 0obnugg7nw3cgcfof0ytw857o0u94ks 2455590 2455589 2026-09-08T12:49:52Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-f4-eqiad 2455590 wikitext text/x-wiki 3wtf2j3ysq7qzri7udmzrqfwsffpo8h 2455591 2455590 2026-09-08T12:49:55Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-e3-eqiad 2455591 wikitext text/x-wiki r5k9uc6s3f7ewy1rjz2y190j35yx5bb 2455592 2455591 2026-09-08T12:50:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e3-eqiad 2455592 wikitext text/x-wiki f8it2bh8nwgng5d3s9djmk8ibds1vvj 2455593 2455592 2026-09-08T12:50:05Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f2-eqiad 2455593 wikitext text/x-wiki 8ubvfvvzcqfcm7wclx0fgnbcppss4tr 2455594 2455593 2026-09-08T12:50:13Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f2-eqiad 2455594 wikitext text/x-wiki 6nf8npstknyymjhno5duen5001c8gee 2455595 2455594 2026-09-08T12:50:15Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f3-eqiad 2455595 wikitext text/x-wiki roua7byztm9mjbl2v8dlh5g3yv30chz 2455596 2455595 2026-09-08T12:50:18Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet 2455596 wikitext text/x-wiki ggtmfua9mqiina1mkv2b4i1ttjzro7w 2455597 2455596 2026-09-08T12:50:20Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet 2455597 wikitext text/x-wiki bqj1od5o1vgr5yvu3breeuw9mnk85b0 2455598 2455597 2026-09-08T12:50:21Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f3-eqiad 2455598 wikitext text/x-wiki fr15qlm1zxd0qqn4n7xrdizvahubtub 2455599 2455598 2026-09-08T12:50:25Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr3-ulsfo 2455599 wikitext text/x-wiki drcx8mln0to01f2xiesy8s7ap04ctla 2455600 2455599 2026-09-08T12:50:26Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet 2455600 wikitext text/x-wiki 5htldoztqtsscvi04gzrjbfxqyp4xom 2455601 2455600 2026-09-08T12:50:31Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw 2455601 wikitext text/x-wiki f60p07hhjbjlbodlqq0or5a4lezz4jn 2455602 2455601 2026-09-08T12:50:42Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr3-ulsfo 2455602 wikitext text/x-wiki no58t3iu6r9vocr2oaa8fpjf4mfj069 2455603 2455602 2026-09-08T12:50:45Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr4-ulsfo 2455603 wikitext text/x-wiki enzi1q1tdotzm6qr2yal8syzn68hp49 2455604 2455603 2026-09-08T12:50:57Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr4-ulsfo 2455604 wikitext text/x-wiki r2jx1ll2x17kw7r2ufscp1tyu9e9slr 2455605 2455604 2026-09-08T12:51:00Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr3-eqsin 2455605 wikitext text/x-wiki 1v12ezozbtdtkjd37axf7jby1oy6l53 2455606 2455605 2026-09-08T12:51:35Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr3-eqsin 2455606 wikitext text/x-wiki 7ajvld7137h2to5moumdtbpb3lvzjhi 2455607 2455606 2026-09-08T12:51:38Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr1-drmrs 2455607 wikitext text/x-wiki gukyah7rreofnm0vp33j4ptmacubmbn 2455608 2455607 2026-09-08T12:51:52Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-drmrs 2455608 wikitext text/x-wiki jy9lglx2nbxjnxon38nqlk8whrylbc7 2455609 2455608 2026-09-08T12:51:54Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr1-esams 2455609 wikitext text/x-wiki 59i08ktj12m4v8mioplm2r4yar0phn3 2455610 2455609 2026-09-08T12:52:15Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-esams 2455610 wikitext text/x-wiki 0hqvxdxqenu1kc7qsao4kvzpzyzp802 2455611 2455610 2026-09-08T12:52:18Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr2-drmrs 2455611 wikitext text/x-wiki l3yox9227xqffhjohtcp6zv74try9re 2455612 2455611 2026-09-08T12:52:38Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-drmrs 2455612 wikitext text/x-wiki bmp5sc2zb8xaqysnp3z2rfrnyez8gad 2455613 2455612 2026-09-08T12:52:41Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device asw1-bw27-esams 2455613 wikitext text/x-wiki 7fira0jcx6pqy471o73otyg5v5b1yb3 2455614 2455613 2026-09-08T12:53:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-bw27-esams 2455614 wikitext text/x-wiki sylhrna0rhy5c1qx0zx5gq1bgdt65d0 2455615 2455614 2026-09-08T12:53:04Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr2-esams 2455615 wikitext text/x-wiki 9w1brzvl8rk0dculwlwrvjcsudjeb58 2455616 2455615 2026-09-08T12:53:24Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-esams 2455616 wikitext text/x-wiki 6tvh1wgx380rdbv0vix64w0wf8qi1rq 2455617 2455616 2026-09-08T12:53:27Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device asw1-b12-drmrs 2455617 wikitext text/x-wiki 577fdv2a3sfrf0kbyw6s8pq881w7z5p 2455618 2455617 2026-09-08T12:53:46Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b12-drmrs 2455618 wikitext text/x-wiki gsz6941beewgu1ehkjv467cw9r7l53v 2455619 2455618 2026-09-08T12:53:50Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device asw1-b13-drmrs 2455619 wikitext text/x-wiki 1at5gl4fg1yrq1zfo3g4s3iv7xbt802 2455620 2455619 2026-09-08T12:54:09Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b13-drmrs 2455620 wikitext text/x-wiki 15r8sqwekcti5b202bww64o18mftfha 2455621 2455620 2026-09-08T12:54:11Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2455621 wikitext text/x-wiki gzlqqlbgtlsu7hh2qc9d6gq432k6mu0 2455622 2455621 2026-09-08T12:54:12Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device asw1-by27-esams 2455622 wikitext text/x-wiki no6e3bk0ctj6bqbqktsjs8j548nek3m 2455623 2455622 2026-09-08T12:54:32Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-by27-esams 2455623 wikitext text/x-wiki 47wt86nohqiym6gebfk93iitj4rwzta 2455624 2455623 2026-09-08T12:54:34Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr2-codfw 2455624 wikitext text/x-wiki 05m1mrtf1ehrrs3q7ejt87abyr73sdk 2455625 2455624 2026-09-08T12:54:46Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2011.codfw.wmnet with OS bookworm 2455625 wikitext text/x-wiki 4wibbnls550zo9675depr8rfiee7m4g 2455626 2455625 2026-09-08T12:54:50Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-codfw 2455626 wikitext text/x-wiki tgmst8uzp5qcz9y0wxxj1nnxf9qalzr 2455627 2455626 2026-09-08T12:54:52Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device cr1-codfw 2455627 wikitext text/x-wiki 9lmlr6tdwog6n3lxx2nz2mfb89dhsim 2455628 2455627 2026-09-08T12:55:07Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-codfw 2455628 wikitext text/x-wiki ik5u3nqtvj5pnf3de3e2qun5m3lr859 2455629 2455628 2026-09-08T12:55:09Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2455629 wikitext text/x-wiki n3rx0xer5potdzpxqhnzhffzsk1ucwm 2455630 2455629 2026-09-08T12:55:10Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw 2455630 wikitext text/x-wiki 0sbyg7b7g7x9fvq4vwv99kywh0pd29o 2455631 2455630 2026-09-08T12:55:28Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet 2455631 wikitext text/x-wiki 3xcu5g4a1ny31fo5rslzx8dvmrm9pb2 2455632 2455631 2026-09-08T12:55:40Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'clear' for AS: 35320 2455632 wikitext text/x-wiki oo01idved5paryve8qdknaioiqq099y 2455633 2455632 2026-09-08T12:55:53Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 35320 2455633 wikitext text/x-wiki kv1bjfcbbfhzzv3dyir9k1muqzlfhcl 2455634 2455633 2026-09-08T12:56:14Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reboot-single for host aqs2012.codfw.wmnet 2455634 wikitext text/x-wiki qf8vja2tao6hccbsfrz740am3isf8zt 2455635 2455634 2026-09-08T12:58:02Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 34655 2455635 wikitext text/x-wiki n7d5ehg5ggqbxuuifkm1j7jzcbgopwm 2455636 2455635 2026-09-08T12:58:41Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 34655 2455636 wikitext text/x-wiki bnqf025tbxwdbz9xdusb1z4zjqd9lmy 2455637 2455636 2026-09-08T13:01:45Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet 2455637 wikitext text/x-wiki 1zxf3cqbtgq1h93v4teytrd3x1sv7p6 2455638 2455637 2026-09-08T13:01:46Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet 2455638 wikitext text/x-wiki 3ihqvx2jt1kx80ihjw2tl3ahwtmdg4r 2455639 2455638 2026-09-08T13:01:51Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet 2455639 wikitext text/x-wiki acryz7sg6197wkgi1zb2kuwp9065q28 2455640 2455639 2026-09-08T13:04:34Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host aqs2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2455640 wikitext text/x-wiki s368po7pxsdtztoncmbjl0sxufope17 2455641 2455640 2026-09-08T13:05:22Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad 2455641 wikitext text/x-wiki lgd9yq9xovxjgja0na1222gyhn66bq4 2455642 2455641 2026-09-08T13:06:17Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2012.codfw.wmnet 2455642 wikitext text/x-wiki b95401o9w9t1qw81fbxnktxwwx7ncqa 2455643 2455642 2026-09-08T13:06:18Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2012.codfw.wmnet 2455643 wikitext text/x-wiki 0qvpiusyjfeysitl9trm20tctwcbf3a 2455644 2455643 2026-09-08T13:06:20Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2455644 wikitext text/x-wiki f8n58ku0p4ale1cerphnmmokpebvuv9 2455645 2455644 2026-09-08T13:06:42Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host aqs2012.codfw.wmnet with OS bookworm 2455645 wikitext text/x-wiki bc1hmxgnqp7ll9mcysot67yko3o24vm 2455646 2455645 2026-09-08T13:06:54Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet 2455646 wikitext text/x-wiki dq0j4j48sdbucr87n47tpudctb5ju1n 2455647 2455646 2026-09-08T13:08:52Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 14593 2455647 wikitext text/x-wiki kf2p36u5dzvdaaibbrsfh58nq7gt3zl 2455648 2455647 2026-09-08T13:09:02Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2455648 wikitext text/x-wiki rew9z3ujhrgqsfeo0piyn0ymtxvkdfa 2455649 2455648 2026-09-08T13:10:03Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1337610|Split out edit and block-based filters from activity filters (T436508)]] 2455649 wikitext text/x-wiki hipp08bk1ihdaq1gw16ckr2e6l174xx 2455650 2455649 2026-09-08T13:10:07Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2455650 wikitext text/x-wiki 42wx5d78tfthzj1py9ebb1uoj4d3j5v 2455651 2455650 2026-09-08T13:10:08Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad 2455651 wikitext text/x-wiki rsewukh1elhxs92xh10lt5v5pbc8mj5 2455652 2455651 2026-09-08T13:10:20Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 14593 2455652 wikitext text/x-wiki 8dnnlbq1lbwtkducxlmozzrzf4wyj1p 2455653 2455652 2026-09-08T13:11:20Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 2519 2455653 wikitext text/x-wiki dqdqgli61rc5qv3f7zjswbqtlobr1na 2455654 2455653 2026-09-08T13:11:29Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 2519 2455654 wikitext text/x-wiki fa7hyzl1431avphmd7716yw79gbyzmi 2455655 2455654 2026-09-08T13:13:24Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet 2455655 wikitext text/x-wiki g74kty8pdskv9y5zrv04lg8g5zthcxs 2455656 2455655 2026-09-08T13:13:26Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet 2455656 wikitext text/x-wiki ohthztanchwct6av8klue1ljndgfi0b 2455657 2455656 2026-09-08T13:13:32Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet 2455657 wikitext text/x-wiki pelaak6r1umwd50uez4f0xxltj6dtzu 2455658 2455657 2026-09-08T13:15:29Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 139628 2455658 wikitext text/x-wiki l5zdotrych0lylbficvk7y9b85hib1z 2455659 2455658 2026-09-08T13:16:57Z Stashbot 7414 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 139628 2455659 wikitext text/x-wiki 6jcgdeptwal8bnf5eo9u2v6nptlipp2 2455660 2455659 2026-09-08T13:18:34Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet 2455660 wikitext text/x-wiki ke0vhao6cnpgxjh3krs49o5j2yb2zdf 2455661 2455660 2026-09-08T13:21:04Z Stashbot 7414 moritzm: installing qemu security updates 2455661 wikitext text/x-wiki dzlhh75w7dbicu18snpevmpi1p7qpda 2455663 2455661 2026-09-08T13:23:26Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2012.codfw.wmnet with reason: host reimage 2455663 wikitext text/x-wiki kyee8l6ave2641cqae06ehoftgd56yn 2455664 2455663 2026-09-08T13:23:29Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet 2455664 wikitext text/x-wiki rkn0oawycw8h3w61g3fpzdbr9vleat9 2455665 2455664 2026-09-08T13:23:30Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet 2455665 wikitext text/x-wiki am441sc0zwbf4j5epf4ddbi33dsqp9x 2455666 2455665 2026-09-08T13:23:35Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet 2455666 wikitext text/x-wiki bt9fkjov8i96smni9lblhftu7sz91tw 2455670 2455666 2026-09-08T13:28:38Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet 2455670 wikitext text/x-wiki 4dgddouks979liv1e15pdgn2seg08tn 2455671 2455670 2026-09-08T13:28:44Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2012.codfw.wmnet with reason: host reimage 2455671 wikitext text/x-wiki lttrf8465c07fdtdx3k72d1grj82624 2455672 2455671 2026-09-08T13:29:59Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1337610|Split out edit and block-based filters from activity filters (T436508)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455672 wikitext text/x-wiki gbsxocpw8lkbbj59hdj3bwsrozqthpx 2455673 2455672 2026-09-08T13:32:03Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2455673 wikitext text/x-wiki 95vbclmw6bbqky2snamm95p35np6nyk 2455674 2455673 2026-09-08T13:35:03Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet 2455674 wikitext text/x-wiki l02raz0ewx3filv97x3fgtv6dsth6xn 2455675 2455674 2026-09-08T13:35:04Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet 2455675 wikitext text/x-wiki o4ty9191s9xhrn2ima22hf9a754mf3n 2455676 2455675 2026-09-08T13:35:11Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet 2455676 wikitext text/x-wiki b0v91nrwm7thi02zj5rmgbi8hdfpkwy 2455677 2455676 2026-09-08T13:37:21Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/services/mw-debug: apply 2455677 wikitext text/x-wiki 3s86zbjtls7jegbd3alp21cbs2jy5nn 2455678 2455677 2026-09-08T13:40:12Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet 2455678 wikitext text/x-wiki ihlv8inixxvpwh4l7twsetvruxevmo4 2455679 2455678 2026-09-08T13:44:03Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337610|Split out edit and block-based filters from activity filters (T436508)]] (duration: 34m 00s) 2455679 wikitext text/x-wiki ahe6h8mvqh12ltu78ebnt7yeelgw94w 2455681 2455679 2026-09-08T13:46:32Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2012.codfw.wmnet with OS bookworm 2455681 wikitext text/x-wiki 6uwc612ar2zniizlavrkl3zon04cdb0 2455682 2455681 2026-09-08T13:46:33Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet 2455682 wikitext text/x-wiki 39adzzto6l7onoxzzvtt3wbeebgzb6y 2455683 2455682 2026-09-08T13:46:34Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet 2455683 wikitext text/x-wiki gzo0w7kpm0ravzkfjaq6bmgazc83wat 2455684 2455683 2026-09-08T13:46:40Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet 2455684 wikitext text/x-wiki 8q73imdq2r2c51dlamh1nwc1m501tai 2455685 2455684 2026-09-08T13:46:56Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox 2455685 wikitext text/x-wiki jcd68s3scmfvgx52mwews7nvv0v3uav 2455688 2455685 2026-09-08T13:50:06Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/services/mw-debug: apply 2455688 wikitext text/x-wiki 9900qcfpgy8x9p4niie5drnjvqpyhr0 2455689 2455688 2026-09-08T13:51:45Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet 2455689 wikitext text/x-wiki 45v2c96huhtqtb02mmt3s45zah2n98m 2455690 2455689 2026-09-08T13:52:57Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/services/mw-debug: apply 2455690 wikitext text/x-wiki izufl8ovx4fprspog242jxn9botb5be 2455691 2455690 2026-09-08T13:54:30Z Stashbot 7414 stran@deploy1003: mwscript-k8s job started: foreachwikiindblist checkuser-suggested-investigations extensions/CheckUser/maintenance/populateSiCaseProperties.php # T435066 2455691 wikitext text/x-wiki fetsrzwai47kmj03pib6hdzwtrlbed5 2455692 2455691 2026-09-08T13:55:30Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/services/mw-debug: apply 2455692 wikitext text/x-wiki o9f5ydoqn05yc0hi38e1h9567st08fz 2455693 2455692 2026-09-08T13:56:08Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/services/mw-debug: apply 2455693 wikitext text/x-wiki mvxs5lah4garhbxr8zporlai7kt0uy8 2455694 2455693 2026-09-08T13:57:45Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet 2455694 wikitext text/x-wiki ebvm6lcw7m9ql2ouezdi5reaejjhu69 2455695 2455694 2026-09-08T13:57:46Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet 2455695 wikitext text/x-wiki ey7wmpv50bvpjeglmye73je06ibu0l4 2455696 2455695 2026-09-08T13:57:51Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet 2455696 wikitext text/x-wiki 7yenn68pd8vcguyha8bbs61fj4qk92k 2455697 2455696 2026-09-08T13:58:08Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/services/mw-debug: apply 2455697 wikitext text/x-wiki f7mmqtdleymvxawdazyooarg867bklb 2455699 2455697 2026-09-08T14:02:54Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet 2455699 wikitext text/x-wiki d0ub3qa96p6yapjz1jm7d1e2yre81vd 2455700 2455699 2026-09-08T14:03:15Z Stashbot 7414 topranks: drain traffic from ssw1-a1-codfw before JunOS upgrade T426197 2455700 wikitext text/x-wiki 6erdnm7gmve3954xfks6dmcn6kdnqqf 2455701 2455700 2026-09-08T14:12:35Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet 2455701 wikitext text/x-wiki lwbzgzidkfucq532ofkagaczncob71c 2455702 2455701 2026-09-08T14:12:36Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet 2455702 wikitext text/x-wiki ktvc773lmp7ekqw2gwoed0wxnmvl1on 2455703 2455702 2026-09-08T14:12:41Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet 2455703 wikitext text/x-wiki d92jsdx5rs3a64dz3nj9w413a4v6rhl 2455704 2455703 2026-09-08T14:13:30Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan 1788220800 --verbose # T437158 2455704 wikitext text/x-wiki tdnko72jpv4j56k8dtej2av8ze86p8k 2455709 2455704 2026-09-08T14:22:45Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet 2455709 wikitext text/x-wiki nlld9mimdnqwayuso4x7h7000tanxz7 2455710 2455709 2026-09-08T14:28:54Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet 2455710 wikitext text/x-wiki azdef0w8nbljum3w0r2e1mrqw8l6xu6 2455711 2455710 2026-09-08T14:28:55Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet 2455711 wikitext text/x-wiki 0ty4td1uvz90mzo44ks2o97y9uopfsa 2455712 2455711 2026-09-08T14:28:56Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad 2455712 wikitext text/x-wiki 8oji5pxqndzzwq433zo8fcy9erp8cep 2455714 2455712 2026-09-08T14:30:47Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw 2455714 wikitext text/x-wiki 5wdsb46zmm61tu4e80so8oh8ak1qeqz 2455715 2455714 2026-09-08T14:33:26Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-codfw 2455715 wikitext text/x-wiki ed9n66s3bait4a3hwa0bs26fc1s21h4 2455716 2455715 2026-09-08T14:34:20Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host cephosd2001.codfw.wmnet 2455716 wikitext text/x-wiki 3owx822wabnjkgxjv4g8uv30wgymc1s 2455720 2455716 2026-09-08T14:43:21Z Stashbot 7414 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: upgrade ssw1-a1-eqiad 2455720 wikitext text/x-wiki teeniju7nnc3c087tkiojtj4ax7hbcc 2455721 2455720 2026-09-08T14:44:22Z Stashbot 7414 topranks: shutdown et-1/1/5 on cr1-codfw to shift traffic off ssw1-a1-codfw 2455721 wikitext text/x-wiki 1w1leruonjgcbhc1f2nu5208vrpbwtl 2455723 2455721 2026-09-08T14:45:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd2001.codfw.wmnet 2455723 wikitext text/x-wiki 26l2iv5dapzwxadn0xlnq3tz3nizvkz 2455724 2455723 2026-09-08T14:46:10Z Stashbot 7414 logmsgbot: dreamyjazz Deployed security patch for T436429 2455724 wikitext text/x-wiki i32qglzph0utz07tzegkffy84lmctyu 2455726 2455724 2026-09-08T14:55:09Z Stashbot 7414 logmsgbot: dreamyjazz Deployed security patch for T436429 2455726 wikitext text/x-wiki e0yi9zkdzpgfpdoub5xbv0ru6vtfbpv 2455728 2455726 2026-09-08T15:19:42Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host cephosd2002.codfw.wmnet 2455728 wikitext text/x-wiki itvx0si3gnssmmoquf9wv3pb233kbj9 2455730 2455728 2026-09-08T15:29:21Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd2002.codfw.wmnet 2455730 wikitext text/x-wiki 8ok5t7q14trzkrp02q1k90a4ih6n1r5 2455732 2455730 2026-09-08T15:44:00Z Stashbot 7414 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2455732 wikitext text/x-wiki jxa6x3bbvjtv5qhxsi6irvo5mwhj44f 2455733 2455732 2026-09-08T15:54:26Z Stashbot 7414 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2455733 wikitext text/x-wiki 2vqibx37w98k6vek7k2mk2xl7wkghyn 2455734 2455733 2026-09-08T15:55:27Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host cephosd2003.codfw.wmnet 2455734 wikitext text/x-wiki 3tcftl8h1etsg8j65mewf85ejj9b2jx 2455735 2455734 2026-09-08T15:58:12Z Stashbot 7414 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2455735 wikitext text/x-wiki j037025hzbj43hlkm4afi67h7g3kh2c 2455736 2455735 2026-09-08T16:04:50Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd2003.codfw.wmnet 2455736 wikitext text/x-wiki 89yqfqs2224ovcuhtpbf31ug8nuoqz2 2455737 2455736 2026-09-08T16:05:31Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2455737 wikitext text/x-wiki 37r1izcgq0ygv63rum65fj86sfpl0d9 2455738 2455737 2026-09-08T16:05:40Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2455738 wikitext text/x-wiki edosrcjlyhzaeswqs3e0y1gwykf8jnj 2455739 2455738 2026-09-08T16:08:41Z Stashbot 7414 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2455739 wikitext text/x-wiki qcquplhmgmj6zv20o4s0qv9rshc6nic 2455740 2455739 2026-09-08T16:12:29Z Stashbot 7414 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie 2455740 wikitext text/x-wiki n3lbb4os7m3dw4wz1r9zux8oujqzqpf 2455743 2455740 2026-09-08T16:24:45Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2004.codfw.wmnet 2455743 wikitext text/x-wiki mge8usvq8j2pdsrees85pvtphp9j5ic 2455744 2455743 2026-09-08T16:24:50Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2005.codfw.wmnet 2455744 wikitext text/x-wiki 957b32c793krpgdxd0waicxzg5r6omk 2455745 2455744 2026-09-08T16:25:02Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2004.codfw.wmnet 2455745 wikitext text/x-wiki itw6ip1mexusa42fn5ptk5k2o6s1i59 2455746 2455745 2026-09-08T16:25:06Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2005.codfw.wmnet 2455746 wikitext text/x-wiki knit4bze2ezb6ks4bwh8dj2vg23m2ti 2455747 2455746 2026-09-08T16:25:09Z Stashbot 7414 jhancock@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage 2455747 wikitext text/x-wiki jgb6vsquj8c7l12w2c33do10mm9cmkv 2455748 2455747 2026-09-08T16:29:18Z Stashbot 7414 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage 2455748 wikitext text/x-wiki gagizfdqzedcn7rrctilrz6a5bz84er 2455750 2455748 2026-09-08T16:41:14Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1337946|Use local database for category table in SpecialWantedCategories (T437286)]], [[gerrit:1337947|Use local database for category table in SpecialWantedCategories (T437286)]] 2455750 wikitext text/x-wiki k68lzfzm875n96bld686nakrpo47h17 2455751 2455750 2026-09-08T16:45:46Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1337946|Use local database for category table in SpecialWantedCategories (T437286)]], [[gerrit:1337947|Use local database for category table in SpecialWantedCategories (T437286)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455751 wikitext text/x-wiki kfxa6kst56zx02yed07a9x5bf2xgbjp 2455752 2455751 2026-09-08T16:46:54Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with deployment 2455752 wikitext text/x-wiki 371h92l04hemcapdkhsuoaugnjmonh7 2455753 2455752 2026-09-08T16:51:35Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337946|Use local database for category table in SpecialWantedCategories (T437286)]], [[gerrit:1337947|Use local database for category table in SpecialWantedCategories (T437286)]] (duration: 10m 19s) 2455753 wikitext text/x-wiki 7srdk5t8fj8rirb58nk7225m4nsc68q 2455755 2455753 2026-09-08T17:10:54Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Noop deployment to test mediawiki_runtime_image cleanup 2455755 wikitext text/x-wiki jv3zd7kxqttkudvz0mq7g2x4n72rdz2 2455756 2455755 2026-09-08T17:11:56Z Stashbot 7414 Amir1: dropping links tables from db1247 (s4 replica) - (T437278) 2455756 wikitext text/x-wiki eqvknzwj2tcyljwwfqj8f2ciogvc0d2 2455757 2455756 2026-09-08T17:15:13Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Noop deployment to test mediawiki_runtime_image cleanup (duration: 04m 18s) 2455757 wikitext text/x-wiki ru133ygyopa8ppdwg07b7mwcgy7kch1 2455758 2455757 2026-09-08T17:25:33Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet 2455758 wikitext text/x-wiki nl8zoh3wqa8y98qbgrl65y76qqsalm4 2455759 2455758 2026-09-08T17:30:53Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet 2455759 wikitext text/x-wiki s4x7fn4uyf9vxyj3tq2ziczh2y5yus9 2455760 2455759 2026-09-08T17:36:30Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox 2455760 wikitext text/x-wiki clzaimsl0d2iestuuu2qjghibq0jkk6 2455761 2455760 2026-09-08T17:36:45Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet 2455761 wikitext text/x-wiki k3ls5f0jhd18ep6vken0egg3slxvztp 2455762 2455761 2026-09-08T17:43:47Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet 2455762 wikitext text/x-wiki syngal1eag16bt5zw9xxllb5okylevr 2455768 2455762 2026-09-08T18:18:46Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.19 refs T430838 2455768 wikitext text/x-wiki 4ld0c1dkierig1ea9huav1jf2v16hkf 2455769 2455768 2026-09-08T18:19:01Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1275.eqiad.wmnet 2455769 wikitext text/x-wiki htgbminpikvmx6uqusups6a5ggdeign 2455770 2455769 2026-09-08T18:19:05Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1275.eqiad.wmnet 2455770 wikitext text/x-wiki ia9dug15uqm6xt3liqspalaafiov0w4 2455771 2455770 2026-09-08T18:19:40Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1275.eqiad.wmnet 2455771 wikitext text/x-wiki d8o87ois4loqq9a595bi922hfh5i4k8 2455772 2455771 2026-09-08T18:20:01Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1275.eqiad.wmnet with OS trixie 2455772 wikitext text/x-wiki qspsjc6go05d44sto8ovapo2m3xic89 2455773 2455772 2026-09-08T18:20:17Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1275 2455773 wikitext text/x-wiki 4zj4bve8ki5glu5qw63k9do4qp882u3 2455774 2455773 2026-09-08T18:20:45Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.netbox 2455774 wikitext text/x-wiki a73gya9brvowcm0hzkzymtb3wswhbix 2455775 2455774 2026-09-08T18:25:20Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1275 - swfrench@cumin1003" 2455775 wikitext text/x-wiki 293xg1vvktj0dhry9wv1n5m02z8kh1w 2455776 2455775 2026-09-08T18:33:10Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1275 - swfrench@cumin1003" 2455776 wikitext text/x-wiki 60qvlshmfx54ezcolmer74741hfjbiq 2455777 2455776 2026-09-08T18:33:11Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2455777 wikitext text/x-wiki pe9nc6mquozekn8kdxh415nhxjsgqt7 2455778 2455777 2026-09-08T18:33:12Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1275.eqiad.wmnet 168.48.64.10.in-addr.arpa 8.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455778 wikitext text/x-wiki 0vx892y1817rpw2ly7reof83anwtivn 2455779 2455778 2026-09-08T18:33:14Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1275.eqiad.wmnet 168.48.64.10.in-addr.arpa 8.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455779 wikitext text/x-wiki kpkokpip9rozky87r7r84uzy2tm1nzl 2455780 2455779 2026-09-08T18:33:14Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1275 2455780 wikitext text/x-wiki nqg8iw4zg656o9jv7jsxrfd0wvww82n 2455782 2455780 2026-09-08T18:34:06Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1275 2455782 wikitext text/x-wiki gfe7xz7i1ti1lw0p0ibt8nuk8elw8oq 2455783 2455782 2026-09-08T18:34:07Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1275 2455783 wikitext text/x-wiki 55317lpk3vx8e500yzr2wt83atdeeb2 2455784 2455783 2026-09-08T18:43:04Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie 2455784 wikitext text/x-wiki bzxlt80ch1rbl3tjyw09fpdaem223k0 2455788 2455784 2026-09-08T18:53:00Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1275.eqiad.wmnet with reason: host reimage 2455788 wikitext text/x-wiki jflts2h47u57wm2p973i2a2hv8muwxy 2455791 2455788 2026-09-08T18:58:51Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1275.eqiad.wmnet with reason: host reimage 2455791 wikitext text/x-wiki dfmyx81r9qwww2kvmlu26x68qfbuid4 2455792 2455791 2026-09-08T19:04:32Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie 2455792 wikitext text/x-wiki duw9kwdqjkbqgpw73boci25nbiwdqsy 2455793 2455792 2026-09-08T19:04:44Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie 2455793 wikitext text/x-wiki 8cbpo4qwc00ji4fdrne2h6k6q54162f 2455795 2455793 2026-09-08T19:12:05Z Stashbot 7414 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2013.codfw.wmnet with OS trixie 2455795 wikitext text/x-wiki taqxym2s3f7y3q5gg6ytpfhh9c5vyp2 2455796 2455795 2026-09-08T19:12:18Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie 2455796 wikitext text/x-wiki nipwmgsfgbkmq9mq4ueluoxdmv4mfbj 2455797 2455796 2026-09-08T19:17:24Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie 2455797 wikitext text/x-wiki l8n6sa0txmnjywl955a5cnkwqcb3yo8 2455798 2455797 2026-09-08T19:19:58Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1275.eqiad.wmnet with OS trixie 2455798 wikitext text/x-wiki 2y76e19qeae4rvj62vqw7o2p3g847mf 2455799 2455798 2026-09-08T19:23:40Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie 2455799 wikitext text/x-wiki iji3xw7xdkw3hc0psitj9zhbx2jr1ej 2455801 2455799 2026-09-08T19:31:41Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1275.eqiad.wmnet 2455801 wikitext text/x-wiki abwirdmdniwlc0krp6196ig68ya3545 2455802 2455801 2026-09-08T19:31:42Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1275.eqiad.wmnet 2455802 wikitext text/x-wiki 8mdc4bt5wiaagzrub3mk6kpvqvlgt91 2455803 2455802 2026-09-08T19:31:44Z Stashbot 7414 swfrench@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1275.eqiad.wmnet 2455803 wikitext text/x-wiki ivl9cprvi4ywkbu8pq9i8jd4kn3fl76 2455808 2455803 2026-09-08T19:37:23Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1305.eqiad.wmnet 2455808 wikitext text/x-wiki p3ffc13ddoa5e0i5ovx6ko5vemhi4pa 2455809 2455808 2026-09-08T19:37:27Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1305.eqiad.wmnet 2455809 wikitext text/x-wiki fn6n3yxgmtvlqpxzrtegzvqc7u5i6sy 2455810 2455809 2026-09-08T19:37:58Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.provision for host aqs1023.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2455810 wikitext text/x-wiki my9ksg5nicuyyyn7m33650cc5w3obbh 2455811 2455810 2026-09-08T19:37:59Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1305.eqiad.wmnet 2455811 wikitext text/x-wiki 63lz4ngub3r4p9b0tcfzkxugr8ivefn 2455812 2455811 2026-09-08T19:38:33Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1305.eqiad.wmnet with OS trixie 2455812 wikitext text/x-wiki e4dztqskai616uj7wycy7lf7cxwhek4 2455813 2455812 2026-09-08T19:39:00Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1305 2455813 wikitext text/x-wiki lrvgv4qlv6czt4h49d309ci3sbiueb0 2455814 2455813 2026-09-08T19:39:17Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.netbox 2455814 wikitext text/x-wiki c2sh8shxvfzzjfhimc4zc6p23427edm 2455815 2455814 2026-09-08T19:43:15Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1305 - swfrench@cumin1003" 2455815 wikitext text/x-wiki e943f0y3gbe81o3udw8t1zcb1gs7cuv 2455816 2455815 2026-09-08T19:43:19Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1305 - swfrench@cumin1003" 2455816 wikitext text/x-wiki kr968fe71oorlo4vcsmle6fwj1nv80u 2455817 2455816 2026-09-08T19:43:20Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2455817 wikitext text/x-wiki t83axyycvsqq72k2nwezxzkp1qvsq03 2455818 2455817 2026-09-08T19:43:21Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1305.eqiad.wmnet 138.32.64.10.in-addr.arpa 8.3.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455818 wikitext text/x-wiki elpc8zwh9pb4cicdedkrvm7vb60p6wp 2455819 2455818 2026-09-08T19:43:22Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1305.eqiad.wmnet 138.32.64.10.in-addr.arpa 8.3.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455819 wikitext text/x-wiki a46fxv587gu0alu4pwp34x35y24r8wj 2455820 2455819 2026-09-08T19:43:24Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1305 2455820 wikitext text/x-wiki 587hmyz4c8kpdcljl2mjdg0ushflac2 2455821 2455820 2026-09-08T19:44:08Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1305 2455821 wikitext text/x-wiki izirfo8ykny7dehlebkp2mvm9r541f7 2455822 2455821 2026-09-08T19:44:09Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1305 2455822 wikitext text/x-wiki b1nfsjgy4819ruk1tagpq2qmjljifye 2455823 2455822 2026-09-08T19:45:05Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs1023.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2455823 wikitext text/x-wiki evdqtg735kyalu4cutbpzhl1susxlku 2455825 2455823 2026-09-08T19:51:09Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1023.eqiad.wmnet with OS bookworm 2455825 wikitext text/x-wiki 1hralromutznnd0mja6rivbiq24d8q3 2455828 2455825 2026-09-08T20:01:13Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1023.eqiad.wmnet with reason: host reimage 2455828 wikitext text/x-wiki 7rtiyxahaza0ze621l78b05rybfb8e7 2455830 2455828 2026-09-08T20:05:19Z Stashbot 7414 aaron@deploy1003: Started scap sync-world: Backport for [[gerrit:1326425|Add wmf-analytics-commons external module to commonswiki (T434927)]] 2455830 wikitext text/x-wiki p2e0xgy5cm9b8h1ggygcsqm1gwmiqt9 2455831 2455830 2026-09-08T20:05:27Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1305.eqiad.wmnet with reason: host reimage 2455831 wikitext text/x-wiki luhz7mwyqegoocjhz8vvz08uecrk330 2455832 2455831 2026-09-08T20:09:24Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1023.eqiad.wmnet with reason: host reimage 2455832 wikitext text/x-wiki tgjjkycj6s1j3hgkhsk9c9l250u8thx 2455833 2455832 2026-09-08T20:09:59Z Stashbot 7414 aaron@deploy1003: aaron: Backport for [[gerrit:1326425|Add wmf-analytics-commons external module to commonswiki (T434927)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455833 wikitext text/x-wiki faz59ze1dbt4bcsllpqx9c09iosu2dd 2455834 2455833 2026-09-08T20:11:00Z Stashbot 7414 aaron@deploy1003: aaron: Continuing with deployment 2455834 wikitext text/x-wiki hczkph3kqkoe0msn3m4emrfudpz9mjf 2455835 2455834 2026-09-08T20:13:15Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1305.eqiad.wmnet with reason: host reimage 2455835 wikitext text/x-wiki c9k5i38d5it76at64qlyqvcs30gps7o 2455836 2455835 2026-09-08T20:15:36Z Stashbot 7414 aaron@deploy1003: Finished scap sync-world: Backport for [[gerrit:1326425|Add wmf-analytics-commons external module to commonswiki (T434927)]] (duration: 10m 16s) 2455836 wikitext text/x-wiki cdejdowzhcccrtk24nmnbuy1285w500 2455837 2455836 2026-09-08T20:23:07Z Stashbot 7414 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1338023|Filter out non-http(s) license urls (T435999)]], [[gerrit:1337997|Filter out non-http(s) license urls (T435999)]], [[gerrit:1338024|Filter out non-http(s) license urls (T435999)]], [[gerrit:1337998|Filter out non-http(s) license urls (T435999)]], [[gerrit:1338025|Filter out non-http(s) license urls (T435999)]], [[gerrit:1337996|Filter out non- 2455837 wikitext text/x-wiki pizdlgd8ecbhgi9isunoczd1pa4w9ta 2455838 2455837 2026-09-08T20:27:20Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1023.eqiad.wmnet with OS bookworm 2455838 wikitext text/x-wiki 4ww2s2ozaerx62x91he5ylarmu7gbgd 2455839 2455838 2026-09-08T20:27:31Z Stashbot 7414 sbassett@deploy1003: sbassett: Backport for [[gerrit:1338023|Filter out non-http(s) license urls (T435999)]], [[gerrit:1337997|Filter out non-http(s) license urls (T435999)]], [[gerrit:1338024|Filter out non-http(s) license urls (T435999)]], [[gerrit:1337998|Filter out non-http(s) license urls (T435999)]], [[gerrit:1338025|Filter out non-http(s) license urls (T435999)]], [[gerrit:1337996|Filter out non-http(s) license 2455839 wikitext text/x-wiki io2alxfzy7vbalri1jmh5yfg42v33qq 2455840 2455839 2026-09-08T20:28:40Z Stashbot 7414 sbassett@deploy1003: sbassett: Continuing with deployment 2455840 wikitext text/x-wiki lmzpyloxpdr083oukla510ue2h34u60 2455841 2455840 2026-09-08T20:34:22Z Stashbot 7414 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2013.codfw.wmnet with OS trixie 2455841 wikitext text/x-wiki fbdk6yppwh2133ledc9m5m6qkvr38bf 2455842 2455841 2026-09-08T20:35:08Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1305.eqiad.wmnet with OS trixie 2455842 wikitext text/x-wiki 35zkjszm6roinvlsp6lz87yf8kywpfq 2455843 2455842 2026-09-08T20:50:44Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1305.eqiad.wmnet 2455843 wikitext text/x-wiki 51lqylxda0raqyokzvnlhr6s2xje1aj 2455844 2455843 2026-09-08T20:50:45Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1305.eqiad.wmnet 2455844 wikitext text/x-wiki 7v6oa2b78xvd0qilnxfey5ga3yk4enw 2455845 2455844 2026-09-08T20:50:46Z Stashbot 7414 swfrench@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1305.eqiad.wmnet 2455845 wikitext text/x-wiki cbo02r0ms7pevktras1ycpd3zzaolkm 2455846 2455845 2026-09-08T20:53:29Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie 2455846 wikitext text/x-wiki 4m5dvksbaajllj2ay9deypsjvagpk28 2455847 2455846 2026-09-08T20:58:49Z Stashbot 7414 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1338037|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338034|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338035|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338036|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338032|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338033|Revert "Filter out n 2455847 wikitext text/x-wiki pscjynnz0ywq90wezzt0rs3d3fa9v0i 2455848 2455847 2026-09-08T20:59:01Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reboot-single for host aqs1023.eqiad.wmnet 2455848 wikitext text/x-wiki b1ahy3g24b3o5q28afv1qpfu091y5ki 2455849 2455848 2026-09-08T21:03:24Z Stashbot 7414 sbassett@deploy1003: sbassett: Backport for [[gerrit:1338037|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338034|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338035|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338036|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338032|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338033|Revert "Filter out non-http(s) lice 2455849 wikitext text/x-wiki 4wgwcfajys7r8k2j43q4lz58g78snzv 2455850 2455849 2026-09-08T21:04:27Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1023.eqiad.wmnet 2455850 wikitext text/x-wiki qyn3i59ku4btroerlb9fl9elwzavbge 2455851 2455850 2026-09-08T21:04:54Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie 2455851 wikitext text/x-wiki avliot1xqr8m4p3wjk1gpbopigvdy4j 2455852 2455851 2026-09-08T21:05:39Z Stashbot 7414 sbassett@deploy1003: sbassett: Continuing with deployment 2455852 wikitext text/x-wiki 8wbos16xw5da5d3jpuf1o7uczen0v5g 2455853 2455852 2026-09-08T21:05:49Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1024.eqiad.wmnet with OS bookworm 2455853 wikitext text/x-wiki ihk1e048tz86zqgh4pni0m83eeg2jbt 2455854 2455853 2026-09-08T21:10:12Z Stashbot 7414 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1338037|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338034|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338035|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338036|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338032|Revert "Filter out non-http(s) license urls"]], [[gerrit:1338033|Revert "Filter out 2455854 wikitext text/x-wiki 7y4fwkbtceshwe1e3xwn85pft5d29zi 2455855 2455854 2026-09-08T21:15:15Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1324753|InitialiseSettings: Enable 2FA enforcement on remaining private wikis (T428103)]] 2455855 wikitext text/x-wiki 273pu3jz64nfvioye26eotoaylr2t6z 2455856 2455855 2026-09-08T21:15:44Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1024.eqiad.wmnet with reason: host reimage 2455856 wikitext text/x-wiki huhxtwhrnyujg2g9ihs4ekl3oplxbmm 2455857 2455856 2026-09-08T21:18:40Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1306.eqiad.wmnet 2455857 wikitext text/x-wiki pjt7p90j1ykicsii2u4jtybah7u9yz5 2455858 2455857 2026-09-08T21:18:44Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1306.eqiad.wmnet 2455858 wikitext text/x-wiki 6ubj6m0s14wlnw3u1be88vf040609cg 2455859 2455858 2026-09-08T21:19:21Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1306.eqiad.wmnet 2455859 wikitext text/x-wiki bbz2es76f8cflpj51a1un8zdsei8no8 2455860 2455859 2026-09-08T21:19:26Z Stashbot 7414 reedy@deploy1003: reedy: Backport for [[gerrit:1324753|InitialiseSettings: Enable 2FA enforcement on remaining private wikis (T428103)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455860 wikitext text/x-wiki o3l66pnbvwdls1hrerbeayrjhsuej0w 2455861 2455860 2026-09-08T21:19:33Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1024.eqiad.wmnet with reason: host reimage 2455861 wikitext text/x-wiki j6a4k9c4x3fh7b9uzo0hcrofx5ohf6p 2455862 2455861 2026-09-08T21:19:44Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1306.eqiad.wmnet with OS trixie 2455862 wikitext text/x-wiki ren2t7t2d2m9sbzzxggv6kkzcqunacv 2455863 2455862 2026-09-08T21:19:53Z Stashbot 7414 reedy@deploy1003: reedy: Continuing with deployment 2455863 wikitext text/x-wiki jwtqym3q8sf8ivc44xxpdvqnh07lkjg 2455864 2455863 2026-09-08T21:20:11Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1306 2455864 wikitext text/x-wiki bxlk6d6ptkjwr4vqq2dtbl94wvq7w2l 2455865 2455864 2026-09-08T21:20:22Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.netbox 2455865 wikitext text/x-wiki g0u22gcr8s8hm4ux7lh6rngqx2r9u1q 2455866 2455865 2026-09-08T21:24:27Z Stashbot 7414 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324753|InitialiseSettings: Enable 2FA enforcement on remaining private wikis (T428103)]] (duration: 09m 12s) 2455866 wikitext text/x-wiki 0k595o35p3t2kugve1fsy69anushybp 2455867 2455866 2026-09-08T21:25:19Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1306 - swfrench@cumin1003" 2455867 wikitext text/x-wiki 58f2u7wvymlzghrbohewksgif7ibixj 2455868 2455867 2026-09-08T21:30:22Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1306 - swfrench@cumin1003" 2455868 wikitext text/x-wiki is2qkvlfagxh83es62sqqyewgnahf8b 2455869 2455868 2026-09-08T21:30:23Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2455869 wikitext text/x-wiki eyykwcn210ksv92ummtvx8q0yltiog7 2455870 2455869 2026-09-08T21:30:24Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1306.eqiad.wmnet 146.32.64.10.in-addr.arpa 6.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455870 wikitext text/x-wiki mhoonbetdu2bxczduhn5yv8p0ny55ln 2455871 2455870 2026-09-08T21:30:25Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1306.eqiad.wmnet 146.32.64.10.in-addr.arpa 6.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455871 wikitext text/x-wiki pdkjgcrhqjm0b2ehka4lx1ksml3xx2w 2455872 2455871 2026-09-08T21:30:26Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1306 2455872 wikitext text/x-wiki ateotg9tqx2foodgrexfei4tv4nuy5y 2455873 2455872 2026-09-08T21:31:03Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1306 2455873 wikitext text/x-wiki 6r538urrt9esfoj9qlyjgzq0s0lhlkg 2455874 2455873 2026-09-08T21:31:05Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1306 2455874 wikitext text/x-wiki 8ssaf818fpfivnefqw138gly2bbe2t8 2455875 2455874 2026-09-08T21:33:51Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp6008.drmrs.wmnet with OS trixie 2455875 wikitext text/x-wiki bdtuhvcx1fvwge4muhx1l5ytvabfilr 2455876 2455875 2026-09-08T21:35:58Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1024.eqiad.wmnet with OS bookworm 2455876 wikitext text/x-wiki dx82md6muo5zfty81aztnhrynf79tsb 2455877 2455876 2026-09-08T21:39:33Z Stashbot 7414 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1338049|Assets build - 2026-09-08 21:25:19+00:00]] 2455877 wikitext text/x-wiki kswop60lu4wecow5vaphxzjn147qm5t 2455878 2455877 2026-09-08T21:40:52Z Stashbot 7414 jdrewniak@deploy1003: portalsbuilder, jdrewniak: Backport for [[gerrit:1338049|Assets build - 2026-09-08 21:25:19+00:00]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455878 wikitext text/x-wiki huz8ba2tmc0pjq8ram0y5ala8qsh2ta 2455879 2455878 2026-09-08T21:43:16Z Stashbot 7414 jdrewniak@deploy1003: portalsbuilder, jdrewniak: Continuing with deployment 2455879 wikitext text/x-wiki nav5mdvsf5r87uxfc3z7fkc6loqzx1r 2455880 2455879 2026-09-08T21:45:01Z Stashbot 7414 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1338049|Assets build - 2026-09-08 21:25:19+00:00]] (duration: 05m 27s) 2455880 wikitext text/x-wiki tg5jkmo8kqpdr542hhurq4ixfga3wtd 2455881 2455880 2026-09-08T21:48:35Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1025.eqiad.wmnet with OS bookworm 2455881 wikitext text/x-wiki osabw7xa2kdcb6d572l5wusyjgcgymt 2455882 2455881 2026-09-08T21:51:47Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1306.eqiad.wmnet with reason: host reimage 2455882 wikitext text/x-wiki tsec4plcpvxkqvu712v18xw10ct7geq 2455883 2455882 2026-09-08T21:51:51Z Stashbot 7414 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1338053|Bumping portals to master (T128546)]] 2455883 wikitext text/x-wiki 96pcomgcc7o3kwv2wu5tvmushmv0rv3 2455884 2455883 2026-09-08T21:52:46Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp6008.drmrs.wmnet with reason: host reimage 2455884 wikitext text/x-wiki quipo3kzfnqp7l9qtszduka9lyjudhw 2455885 2455884 2026-09-08T21:56:20Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1338053|Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455885 wikitext text/x-wiki ba4kc4smhvckdsamqn57lqi3dstdajl 2455886 2455885 2026-09-08T21:57:03Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1306.eqiad.wmnet with reason: host reimage 2455886 wikitext text/x-wiki 96ud0l877nkmejaxc14h3z22kqmrvb9 2455887 2455886 2026-09-08T21:57:06Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Continuing with deployment 2455887 wikitext text/x-wiki a3n8ekfmhfqz0s4un7uuva4trndgojd 2455888 2455887 2026-09-08T21:58:41Z Stashbot 7414 Amir1: drop links tables from db2210 (T437278) 2455888 wikitext text/x-wiki 2dkwdrqpu89xnpukqwsem44yz5i895t 2455889 2455888 2026-09-08T21:59:49Z Stashbot 7414 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1025.eqiad.wmnet with OS bookworm 2455889 wikitext text/x-wiki kohprgxvhejtqzrknujyz8f0yavpyx9 2455890 2455889 2026-09-08T22:00:04Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1025.eqiad.wmnet with OS bookworm 2455890 wikitext text/x-wiki jf1bp3ola26s3s6ehnyuyye7sv7q7kv 2455891 2455890 2026-09-08T22:01:45Z Stashbot 7414 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1338053|Bumping portals to master (T128546)]] (duration: 09m 53s) 2455891 wikitext text/x-wiki 5a0wnvbuhrvirr6n7cds6lr0vdt8d8k 2455892 2455891 2026-09-08T22:03:18Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6008.drmrs.wmnet with reason: host reimage 2455892 wikitext text/x-wiki ao5floqj9ejsv7mk7ufuks1fkuecjqu 2455893 2455892 2026-09-08T22:18:36Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1306.eqiad.wmnet with OS trixie 2455893 wikitext text/x-wiki 3ydlxqa6u9cyaavk1gz5xf2fvvg296l 2455894 2455893 2026-09-08T22:27:18Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6008.drmrs.wmnet with OS trixie 2455894 wikitext text/x-wiki igj6k0msez5crqhosuoi6wh587win7p 2455895 2455894 2026-09-08T22:30:24Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1306.eqiad.wmnet 2455895 wikitext text/x-wiki 6w4z7qq5g8q350r87n0a3zsyobwzxfl 2455896 2455895 2026-09-08T22:30:25Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1306.eqiad.wmnet 2455896 wikitext text/x-wiki olskonmxpiumr8gcq31tf89e1031ztr 2455897 2455896 2026-09-08T22:30:29Z Stashbot 7414 swfrench@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1306.eqiad.wmnet 2455897 wikitext text/x-wiki stajjpnqdlo1socxcg5jscwztpusj9v 2455898 2455897 2026-09-08T22:31:44Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1313.eqiad.wmnet 2455898 wikitext text/x-wiki l6xxlt3vfhzd7qx2vw5acv1ilbgptru 2455899 2455898 2026-09-08T22:31:48Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1313.eqiad.wmnet 2455899 wikitext text/x-wiki iidjvsx48cnbvctbe9znx6m2uhwtbln 2455900 2455899 2026-09-08T22:32:23Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1313.eqiad.wmnet 2455900 wikitext text/x-wiki rfovcgiqaaeqeiy95m50s29bo9sr93y 2455901 2455900 2026-09-08T22:32:41Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1313.eqiad.wmnet with OS trixie 2455901 wikitext text/x-wiki p0sfno0t25affomyru2uksho51apkvw 2455902 2455901 2026-09-08T22:33:09Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1313 2455902 wikitext text/x-wiki ofpanpnegrm7745t7vp41rgt9kv49pc 2455903 2455902 2026-09-08T22:33:21Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.netbox 2455903 wikitext text/x-wiki 222d4lkq6ie4v26l0wdtdf8nmxa01m4 2455904 2455903 2026-09-08T22:37:13Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1313 - swfrench@cumin1003" 2455904 wikitext text/x-wiki 5d85ho6haeohh7ch6dxq3blxdopzg6o 2455905 2455904 2026-09-08T22:37:17Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1313 - swfrench@cumin1003" 2455905 wikitext text/x-wiki 1s1f5dmadmyzolsxdywldzvfp61aodx 2455906 2455905 2026-09-08T22:37:18Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2455906 wikitext text/x-wiki 70b47hxqgafjdo8xopctujdojwil6wk 2455907 2455906 2026-09-08T22:37:19Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1313.eqiad.wmnet 149.32.64.10.in-addr.arpa 9.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455907 wikitext text/x-wiki p2tzdih7sbxgvdpu6dw0qrmty49ctai 2455908 2455907 2026-09-08T22:37:20Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1313.eqiad.wmnet 149.32.64.10.in-addr.arpa 9.4.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455908 wikitext text/x-wiki 6fyzljobemghsgk5yc92z4ml11ug8qe 2455909 2455908 2026-09-08T22:37:21Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1313 2455909 wikitext text/x-wiki 5cg4j1ng24f8duwa338b0b9yg4i84qm 2455910 2455909 2026-09-08T22:38:44Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1313 2455910 wikitext text/x-wiki myuthdrckceqabrc3a9mojg39ki4o51 2455911 2455910 2026-09-08T22:38:45Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1313 2455911 wikitext text/x-wiki mqpxjw0ie2igrg7cw6k1dt8ujl5g8i2 2455912 2455911 2026-09-08T22:57:34Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1313.eqiad.wmnet with reason: host reimage 2455912 wikitext text/x-wiki iur0lq36hyojq6xtw0hiyl0bfykv6zb 2455913 2455912 2026-09-08T23:02:58Z Stashbot 7414 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1025.eqiad.wmnet with OS bookworm 2455913 wikitext text/x-wiki d1iwwp64nq09ocww9lvx01a5sivjv2k 2455914 2455913 2026-09-08T23:03:20Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1025.eqiad.wmnet with OS bookworm 2455914 wikitext text/x-wiki ad1wvqcodcfjwi6chupk8lelv46rd2b 2455915 2455914 2026-09-08T23:03:51Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1313.eqiad.wmnet with reason: host reimage 2455915 wikitext text/x-wiki qvkuekew719vtvtzyssxequg4cer8jd 2455916 2455915 2026-09-08T23:05:59Z Stashbot 7414 Amir1: dropped 57 tables on db1260 (T437278) 2455916 wikitext text/x-wiki 0bynjas6fhepczchj2n5qwtgf2v320l 2455917 2455916 2026-09-08T23:07:15Z Stashbot 7414 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1025.eqiad.wmnet with OS bookworm 2455917 wikitext text/x-wiki a0u1dntl8ped49jppviatsm2sdr97fu 2455918 2455917 2026-09-08T23:07:46Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.provision for host aqs1025.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2455918 wikitext text/x-wiki dwkp3vfmmapp5wxpyq8ody2dd676n5a 2455919 2455918 2026-09-08T23:14:56Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs1025.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2455919 wikitext text/x-wiki 4m86jfr37vl19gt5g46opzufadcx5ul 2455920 2455919 2026-09-08T23:15:43Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1025.eqiad.wmnet with OS bookworm 2455920 wikitext text/x-wiki tais1pe73j0ajzfv0097logagcx7wr5 2455921 2455920 2026-09-08T23:19:11Z Stashbot 7414 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1025.eqiad.wmnet with OS bookworm 2455921 wikitext text/x-wiki ieynik9ifpltnqazg3nmf1zritj7f30 2455922 2455921 2026-09-08T23:19:24Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1025.eqiad.wmnet with OS bookworm 2455922 wikitext text/x-wiki 3pba3z33j5k9w4j3w46rjv5mjrutik6 2455923 2455922 2026-09-08T23:25:12Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1313.eqiad.wmnet with OS trixie 2455923 wikitext text/x-wiki hrusy16df3p7o87z2ok2el5haptogda 2455924 2455923 2026-09-08T23:30:28Z Stashbot 7414 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1025.eqiad.wmnet with OS bookworm 2455924 wikitext text/x-wiki sgfrok4cwk123meiago5tp9hk46qfjo 2455925 2455924 2026-09-08T23:30:47Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1025.eqiad.wmnet with OS bookworm 2455925 wikitext text/x-wiki 7l3ci4bghmi0klcdvigcad6cfazuqac 2455926 2455925 2026-09-08T23:35:20Z Stashbot 7414 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1025.eqiad.wmnet with OS bookworm 2455926 wikitext text/x-wiki 8c5hyi8z8t3yzcyy10obim8vx3ewond 2455927 2455926 2026-09-08T23:37:36Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1313.eqiad.wmnet 2455927 wikitext text/x-wiki es6ev6ncmoat6rakekt5xocr7wt4nl7 2455928 2455927 2026-09-08T23:37:37Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1313.eqiad.wmnet 2455928 wikitext text/x-wiki e0c3iqy1wm1xmhnbux5f5bsz32djb6l 2455929 2455928 2026-09-08T23:37:38Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1313.eqiad.wmnet 2455929 wikitext text/x-wiki n0pxvh4z8o2ae79t53dlu842arn7vwk 2455930 2455929 2026-09-08T23:39:48Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1025.eqiad.wmnet with OS bookworm 2455930 wikitext text/x-wiki 10hd9xp9yff0fr3v34tdhkrwxnf6gaj 2455931 2455930 2026-09-08T23:48:59Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1025.eqiad.wmnet with reason: host reimage 2455931 wikitext text/x-wiki jgprq413bd740p7dvl91canvhwckcbe 2455932 2455931 2026-09-08T23:51:53Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1025.eqiad.wmnet with reason: host reimage 2455932 wikitext text/x-wiki o85e57lqp7wycer741z73juc3q8ehbs 2455934 2455932 2026-09-09T00:09:52Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1025.eqiad.wmnet with OS bookworm 2455934 wikitext text/x-wiki i39kvl8oyvcwug31kre5nkzfy4b7oav 2455936 2455934 2026-09-09T02:00:43Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2455936 wikitext text/x-wiki 2wxfllmxwrr5nujp9efs522n737dtyq 2455937 2455936 2026-09-09T02:08:28Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s) 2455937 wikitext text/x-wiki q6y3wipl45wpp2hz4eg0vmum8jn5rvt 2455946 2455937 2026-09-09T07:15:46Z Stashbot 7414 chlod@deploy1003: Started scap sync-world: Backport for [[gerrit:1335706|thwikibooks: update tagline and wordmark (T436426)]] 2455946 wikitext text/x-wiki o6pmhxcqyipqbkunivr217yzs6lmtwk 2455947 2455946 2026-09-09T07:20:04Z Stashbot 7414 chlod@deploy1003: chlod, hamishz: Backport for [[gerrit:1335706|thwikibooks: update tagline and wordmark (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455947 wikitext text/x-wiki mkpi3nf7hagkjmrqgj9s3z32k6g7jxw 2455948 2455947 2026-09-09T07:32:36Z Stashbot 7414 chlod@deploy1003: chlod, hamishz: Continuing with deployment 2455948 wikitext text/x-wiki g52plpm6352m3o8spfmn9qd6h9oxre4 2455950 2455948 2026-09-09T07:37:22Z Stashbot 7414 chlod@deploy1003: Finished scap sync-world: Backport for [[gerrit:1335706|thwikibooks: update tagline and wordmark (T436426)]] (duration: 21m 36s) 2455950 wikitext text/x-wiki jpgtkh5742uy2ezvq1mhhyy5e62rdba 2455951 2455950 2026-09-09T07:40:54Z Stashbot 7414 chlod: UTC morning backport window done 2455951 wikitext text/x-wiki hrp6mm04o89ui72molqr9ou7l4t9pnu 2455952 2455951 2026-09-09T08:00:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1198.eqiad.wmnet 2455952 wikitext text/x-wiki avcl9xymrqghp5sfg9ypyzvnba1e4tb 2455953 2455952 2026-09-09T08:01:03Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1198.eqiad.wmnet 2455953 wikitext text/x-wiki gwudtsckx3x3scbarasi3vp1ysxkjps 2455954 2455953 2026-09-09T08:01:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1198.eqiad.wmnet 2455954 wikitext text/x-wiki s7kz81nkfa9uvi57oeyuurancc9h01g 2455955 2455954 2026-09-09T08:01:28Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1198.eqiad.wmnet 2455955 wikitext text/x-wiki t85rja4nzsv3kobm8qk8l1p1nyzicr9 2455956 2455955 2026-09-09T08:03:48Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1260.eqiad.wmnet with reason: Cloning sanitarium 2455956 wikitext text/x-wiki 8se2ifbalxhksqtkbpapzne30uaisay 2455957 2455956 2026-09-09T08:06:23Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1260: Needs to clone another host from this one 2455957 wikitext text/x-wiki 9hzc5jwjwp8l20t3uxo70fpjjhu03sk 2455958 2455957 2026-09-09T08:06:47Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1260: Needs to clone another host from this one 2455958 wikitext text/x-wiki 9qimebm8gbj1pvvicsbf4x8t2n1ip5y 2455959 2455958 2026-09-09T08:28:24Z Stashbot 7414 brouberol@dns1004: START - running authdns-update 2455959 wikitext text/x-wiki k0lhgvi9yfwxmpe61b34dv0arpfcxl0 2455960 2455959 2026-09-09T08:28:37Z Stashbot 7414 moritzm: pruned obsolete Bullseye image golang1.15 from the docker registry T416452 2455960 wikitext text/x-wiki beffbky4bql6wmtsc5c165md8yiq58d 2455961 2455960 2026-09-09T08:30:34Z Stashbot 7414 brouberol@dns1004: END - running authdns-update 2455961 wikitext text/x-wiki dm6kxjf62hqt2fbzf0px2yxeeq4xrm7 2455964 2455961 2026-09-09T08:36:05Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1155.eqiad.wmnet with reason: Cloning sanitarium 2455964 wikitext text/x-wiki 1rd1n0fdb2mf5kiedvf5dc51c62p6ph 2455965 2455964 2026-09-09T08:48:27Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-d1-codfw 2455965 wikitext text/x-wiki r6vqp1rsicmp35jlw36ginqrj8ztl66 2455966 2455965 2026-09-09T08:48:45Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d1-codfw 2455966 wikitext text/x-wiki gnc7ruh1cys7yjz46az19uu24jb70pw 2455967 2455966 2026-09-09T08:49:01Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d3-codfw 2455967 wikitext text/x-wiki 7nkro8bwx539dd4bcj3a3g9zg8lzcav 2455968 2455967 2026-09-09T08:49:19Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d3-codfw 2455968 wikitext text/x-wiki 9wew15msjz5hfhh8ofkylutxdow4vl4 2455969 2455968 2026-09-09T08:55:54Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2455969 wikitext text/x-wiki bz1kwdn5aoc28mw8ql9e4meb2wae6z6 2455970 2455969 2026-09-09T08:56:29Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2455970 wikitext text/x-wiki ditcl3xjxlvhqvxi723y333xv597xec 2455971 2455970 2026-09-09T08:58:59Z Stashbot 7414 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 22 hosts with reason: upgrade ssw1-a1-eqiad 2455971 wikitext text/x-wiki o478aapxodn0kyzzmxmkgvwqoo3g64m 2455972 2455971 2026-09-09T09:01:18Z Stashbot 7414 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: upgrade ssw1-a1-eqiad 2455972 wikitext text/x-wiki s4n8a0hv2pucndi0cfcsxsegvwjxcpb 2455973 2455972 2026-09-09T09:02:49Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2455973 wikitext text/x-wiki sej5gzcfyutfxk5ar8tebgeuef78g6n 2455974 2455973 2026-09-09T09:03:48Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2455974 wikitext text/x-wiki od8v47k72fq2ipzkervxqz5tq9p1bck 2455975 2455974 2026-09-09T09:27:59Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1198.eqiad.wmnet 2455975 wikitext text/x-wiki kzgggnp59y54ptntsfx9o35klugb50s 2455976 2455975 2026-09-09T09:28:02Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1198.eqiad.wmnet 2455976 wikitext text/x-wiki cami8gozcb34prtwlfxhsgi1lt9te29 2455978 2455976 2026-09-09T09:43:23Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply 2455978 wikitext text/x-wiki eezeavchynsl9ajriqc8ozizdgffw8k 2455979 2455978 2026-09-09T09:51:56Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply 2455979 wikitext text/x-wiki m6ciithmfmppjcfvpf77ebi8el7uasa 2455980 2455979 2026-09-09T09:52:36Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d3-codfw 2455980 wikitext text/x-wiki rob6ji336w735u4mcsz7x5fy3tyena7 2455981 2455980 2026-09-09T09:52:54Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d3-codfw 2455981 wikitext text/x-wiki samwp9gwcm6k98f8osxx9rfyc0hls6b 2455982 2455981 2026-09-09T09:53:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1260: Repooling after cloning 2455982 wikitext text/x-wiki jjksrjmnjzloepksz0ip863snohnpyx 2455983 2455982 2026-09-09T09:55:56Z Stashbot 7414 moritzm: pruned obsolete Bullseye images openjdk-8-jdk/openjdk-8-jre/openjdk-11-jre/openjdk-11-jdk from the docker registry T416452 2455983 wikitext text/x-wiki 0ibcwt3dr4c2d7hd70pp5ys20j2lzac 2455985 2455983 2026-09-09T10:06:45Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2455985 wikitext text/x-wiki 3wsdrltw3d28lnnin196de9iv9njpub 2455986 2455985 2026-09-09T10:07:09Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2455986 wikitext text/x-wiki axrx8usanjehftig8kxxwqf2bs1z5k1 2455988 2455986 2026-09-09T10:09:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Repooling db2196 2455988 wikitext text/x-wiki b7aea6jlmx2snjetui6hbbdc6t2rny4 2455990 2455988 2026-09-09T10:38:42Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1260: Repooling after cloning 2455990 wikitext text/x-wiki g3hdyeovvcyt1bsd1fjeu4t1kkgcm6r 2455991 2455990 2026-09-09T10:43:10Z Stashbot 7414 moritzm: installing Bird security updates 2455991 wikitext text/x-wiki c75nh0w1eoyv643cstwsy6momjmp780 2455992 2455991 2026-09-09T10:47:02Z Stashbot 7414 moritzm: pruned obsolete Bullseye images nodejs12-slim/nodejs12-devel/nodejs14-slim/nodejs16-slim from the docker registry T416452 2455992 wikitext text/x-wiki 89lbpwccta4eohdla8jx8w5fxx8t0gk 2455993 2455992 2026-09-09T10:55:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Repooling db2196 2455993 wikitext text/x-wiki pl48euidpl1agd40xe01sj17xjav1z4 2455995 2455993 2026-09-09T11:13:30Z Taavi 13997 archive to [[Server Admin Log/Archive 109]] 2455995 wikitext text/x-wiki 457sg3g1o9q5ggjh44ncsealhi7zlk5 2455998 2455995 2026-09-09T11:31:05Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]] 2455998 wikitext text/x-wiki 2sfwtmqdq2lojvw48rwoqlvpro9e8dv 2455999 2455998 2026-09-09T11:35:28Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455999 wikitext text/x-wiki dheauu8v0kymi4tlwa285xmcuc8npzq 2456000 2455999 2026-09-09T11:48:16Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Continuing with deployment 2456000 wikitext text/x-wiki 6inj42lm9g0b6jkdeeab8ie1qav0z64 2456001 2456000 2026-09-09T11:53:03Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]] (duration: 21m 58s) 2456001 wikitext text/x-wiki s3q7gwll5znet6jk63zbgd93lca4w64 498 9925 2455805 2455374 2026-09-08T19:35:51Z Stashbot 7414 Southparkfan: set profile::mediawiki::httpd::enable_forensic_log to false, to avoid disk exhaustion during high-traffic load 2455805 wikitext text/x-wiki 20z6bjo8288vco94yu7q0d34m7ax65d 2455807 2455805 2026-09-08T19:36:28Z Stashbot 7414 Southparkfan: truncate Apache2 forensic log on deployment-mediawiki14, disk almost full 2455807 wikitext text/x-wiki 9bvjlppcbo770oh7ckutq4fwuskv2px 2456003 2455807 2026-09-09T11:58:46Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'denisse' in role 'member' 2456003 wikitext text/x-wiki 9okrey9fwdqw78f2gf01b2xlxbb74tm 2456005 2456003 2026-09-09T11:58:51Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'denisse' in role 'member' 2456005 wikitext text/x-wiki beoymk10lguzhudm65ihd1v1p88db27 0 11557 2455963 2387010 2026-09-09T08:33:58Z ADenisse-WMF 31462 Update to GitLab repo. 2455963 wikitext text/x-wiki h570n3oepm0md96huk10ja2v7wgns1u 0 17290 2455727 2455373 2026-09-08T15:06:24Z Stashbot 7414 hashar: Tag Quibble @ b692707e5213abf5e16894b1fb21cea41cd72c9e # T429715 T303270 T437134 T300727 2455727 wikitext text/x-wiki hitbt25a9ri6yjgol52wm2ic26x1tcj 2455731 2455727 2026-09-08T15:42:30Z Stashbot 7414 hashar: Updating Quibble jobs to 1.21.0 # T429715 T303270 T437134 T300727 2455731 wikitext text/x-wiki his2ef47djaa2j9lqomwtqc1pb2ldd6 2455742 2455731 2026-09-08T16:21:52Z Stashbot 7414 thcipriani: hard reboot deployment-prep:deployment-cache-text08 horizon logs showing OOMs 2455742 wikitext text/x-wiki 4fvqsipodxg1bggcuhh6n73jmzxvp81 2455804 2455742 2026-09-08T19:35:50Z Stashbot 7414 Southparkfan: set profile::mediawiki::httpd::enable_forensic_log to false, to avoid disk exhaustion during high-traffic load 2455804 wikitext text/x-wiki 8ogym5ko7fnb68erfqsuryljkyt6xei 2455806 2455804 2026-09-08T19:36:28Z Stashbot 7414 Southparkfan: truncate Apache2 forensic log on deployment-mediawiki14, disk almost full 2455806 wikitext text/x-wiki fzmgaz1b5t36njvt71vui0hhs97fo9m 2455977 2455806 2026-09-09T09:42:17Z Stashbot 7414 phedenskog: jjb: update quibble-with-gated-extensions-selenium-php83 to run npm install ahead of the browser tests T436376 2455977 wikitext text/x-wiki 2sut3s94k7ey7zmwwmphv4q7fp3hu6n 2456002 2455977 2026-09-09T11:58:45Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'denisse' in role 'member' 2456002 wikitext text/x-wiki a57idp79out79grasg46oh539v6lvxw 2456004 2456002 2026-09-09T11:58:50Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'denisse' in role 'member' 2456004 wikitext text/x-wiki 1wm7dsnbepubg7jjukql860z15k9ykj 498 30942 2455786 2455281 2026-09-08T18:49:12Z Stashbot 7414 andrewbogott: restarting haproxy on proxy-5.project-proxy due to port exhausting. This will likely happen again in a couple hours. 2455786 wikitext text/x-wiki ttarsyrzp85nda4m48agar1oclwh8z1 0 49472 2455767 2445765 2026-09-08T18:18:14Z KReid-WMF 49588 Add preference for Test Kitchen over Matomo for new sites. 2455767 wikitext text/x-wiki 5qrod62x8dntenctfatd249v98l9jhl 0 122303 2455662 2413462 2026-09-08T13:21:42Z SSapaty (WMF) 28955 /* Modify quotas */ 2455662 wikitext text/x-wiki t92ri71qbh7qg5oblu1furp6x4w53rc 0 441753 2455687 2394291 2026-09-08T13:49:15Z Taavi-WMF 41365 document adding new sections 2455687 wikitext text/x-wiki k4o08dd2y5n3zwg38950fg7zeep74gm 0 442233 2455719 2360999 2026-09-08T14:43:16Z Taavi-WMF 41365 +x4 2455719 wikitext text/x-wiki f1mq956l8l852fyeybn246y7xw4uah1 2455725 2455719 2026-09-08T14:52:56Z Taavi-WMF 41365 fix name 2455725 wikitext text/x-wiki 2ae7wakka0akbq725juqj3kgiosrowv 2455729 2455725 2026-09-08T15:19:46Z JJMC89 7474 +[[News/2025 Cloud VPS Bullseye deprecation]] 2455729 wikitext text/x-wiki sbpmy11c7612a5di2cpxs5agtg5414t 498 445024 2455800 2438632 2026-09-08T19:25:38Z Stashbot 7414 andrewbogott: stopping apache2 on wsexport-main01.wikisource.eqiad1.wikimedia.cloud 2455800 wikitext text/x-wiki s71eylbl2z3ouj3ye7ace2nacm7xkm6 116 447011 2455935 1892982 2026-09-09T00:10:30Z ValerioBoz-WMCH 45376 2455935 wikitext text/x-wiki 99l7kf3pq9nlnevu8srhgfuhccv096f 0 447017 2455667 2453455 2026-09-08T13:24:00Z DMburugu (WMF) 25135 /* Published SLOs */ add CommunityConfiguration and GrowthExperiments SLOs 2455667 wikitext text/x-wiki si1nm9qu72ojqmimoswstvgquk3shzo 0 447310 2455766 2439985 2026-09-08T18:15:59Z KReid-WMF 49588 /* Guides */ Removed link to TKAA documents, as it is deprecated 2455766 wikitext text/x-wiki l91eo6zadfdb3cgl1um772sf3rcynjs 0 447352 2455962 2454971 2026-09-09T08:33:53Z MMuhlenhoff (WMF) 5268 /* Deleting images */ Some context around how images are pruned 2455962 wikitext text/x-wiki leay8bjwossv6m03usost5gv723x2uv 0 448386 2455741 2454338 2026-09-08T16:14:37Z CDanis (WMF) 13326 /* Goal */ 2455741 wikitext text/x-wiki be26vwjngaam9ktfv4vorx7lvfh2pzc 0 449160 2455933 2455473 2026-09-09T00:02:10Z Dexbot 30554 Bot: Updating the report 2455933 wikitext text/x-wiki s4saz7ve72o86uw1b0ivr6ktpvuk2zg 0 450212 2455785 2119671 2026-09-08T18:46:53Z RLazarus (WMF) 15215 2455785 wikitext text/x-wiki jphhd728hxhsgdf92rjsz763d5xoj4p 0 451075 2455826 2451550 2026-09-08T20:00:21Z DAdedoyin (WMF) 39150 /* Historical */ 2455826 wikitext text/x-wiki oyot93rxsrwjl6c16phznmjobk2lmt4 2455827 2455826 2026-09-08T20:00:49Z DAdedoyin (WMF) 39150 /* Current */ 2455827 wikitext text/x-wiki k97j9bgrom2tqtvuza6etlmuooyq4ul 2455829 2455827 2026-09-08T20:01:58Z DAdedoyin (WMF) 39150 /* Current */ 2455829 wikitext text/x-wiki ejz4uytjij5lkreih7vya2kyliu8kyw 0 453382 2455984 2442217 2026-09-09T09:56:10Z CMooney (WMF) 24288 /* Show juniper's gRPC deamon's status */ 2455984 wikitext text/x-wiki reztufropijxwn3z5m0hsyco1uek22q 2455987 2455984 2026-09-09T10:09:46Z CMooney (WMF) 24288 /* Check Juniper "Analytics Agent" is running correctly on a target device */ 2455987 wikitext text/x-wiki m7l4wzsx7m0w1e2n5f8e25rcr98mv2d 2455989 2455987 2026-09-09T10:38:10Z CMooney (WMF) 24288 /* Check Juniper "Analytics Agent" is running correctly on a target device */ 2455989 wikitext text/x-wiki k8o9f4i3qc8adjlxhdlye0uyxk7s5g4 0 454144 2455945 2451938 2026-09-09T06:29:40Z Filippo Giunchedi 2939 Link nfsd grafana dashboard 2455945 wikitext text/x-wiki 3j44xnuwxwh1u1fclgemo1ux1y2j6az 0 459290 2455707 2453551 2026-09-08T14:17:48Z SNwachukwu (WMF) 39521 2455707 wikitext text/x-wiki nyddm4ykrem6ejkp8kis8wfogupgr03 0 459511 2455718 2442911 2026-09-08T14:41:16Z Jdforrester (WMF) 468 Add dones. 2455718 wikitext text/x-wiki jqvijt0ciz1sbr2hji80wu62nrh7r82 0 459699 2455749 2454644 2026-09-08T16:38:15Z Stashbot 7414 larssandergreen: tools upgraded from 14ec70fa to c7be1099 2455749 wikitext text/x-wiki td734d5ao4ujlcbmvmqffm1mml15seu 2455754 2455749 2026-09-08T17:04:19Z Stashbot 7414 larssandergreen: civicrm upgraded from e66e5440 to 13b49ffe 2455754 wikitext text/x-wiki 4qxeb495b23eovyciqe6nklv2wzu4s7 2455939 2455754 2026-09-09T04:44:14Z Stashbot 7414 eileen: civicrm upgraded from 13b49ffe to 36a5aa98 2455939 wikitext text/x-wiki ioergjn2xt8y9jokfzoo4cf186nzafc 0 460194 2455949 2446269 2026-09-09T07:36:38Z MLechvien-WMF 49197 Make sure the FAQ is easy to discover 2455949 wikitext text/x-wiki goq2cpvee0d99yg96xapzd6m6wwchqt 0 460286 2455668 2423810 2026-09-08T13:25:23Z DMburugu (WMF) 25135 Remove todos and update status 2455668 wikitext text/x-wiki k9d7upbjku2lcce4z7suchqv452nd93 0 460287 2455669 2423804 2026-09-08T13:25:55Z DMburugu (WMF) 25135 Remove todos and update status 2455669 wikitext text/x-wiki ddw3oe40f8jxd7rdsd2x9e74t4dong0 0 460330 2455722 2447560 2026-09-08T14:45:06Z BLiviero-WMF 53603 2455722 wikitext text/x-wiki o3dq28pjetd87f8617bue1hyyud2q0i 2 460586 2455942 2451851 2026-09-09T05:53:08Z SSapaty (WMF) 28955 2455942 wikitext text/x-wiki kgxq5ahcox76jlflkxi3mslptnjr7r4 2455943 2455942 2026-09-09T06:03:15Z SSapaty (WMF) 28955 2455943 wikitext text/x-wiki dqnodc6iksr1goq7daf32h9lhr7fkij 2 460587 2455944 2451198 2026-09-09T06:05:24Z SSapaty (WMF) 28955 Use actual repo(ie recent-changes) for tutorial 2455944 wikitext text/x-wiki 3sj98woczf94xoe6pxdjamuvijx5npa 0 460601 2455996 2453111 2026-09-09T11:13:49Z Taavi 13997 archive rest of august 2455996 wikitext text/x-wiki g07dppbone3pxmhd50luk0ayfmu4tkb 0 460641 2455698 2455359 2026-09-08T13:59:17Z Taavi-WMF 41365 update for done items + tense 2455698 wikitext text/x-wiki na5hthqali06lo0j246lbfdru19qjx2 2455713 2455698 2026-09-08T14:30:36Z Taavi-WMF 41365 not a draft anymore 2455713 wikitext text/x-wiki hge5xfa9g43qqblflvayyx97ded9o5s 0 460643 2455680 2026-09-08T13:44:52Z Taavi-WMF 41365 Redirected page to [[Portal:Cloud Services/Admin/cloud-announce]] 2455680 wikitext text/x-wiki r5ypy7zjrio99qjf6zc7it2ay8um84v 0 460644 2455686 2026-09-08T13:47:56Z Taavi-WMF 41365 Redirected page to [[Portal:Data Services/Admin/Wiki Replicas/DNS]] 2455686 wikitext text/x-wiki jd1jom9u54cm3uyeg8v69iybdmj8d6r 2 460645 2455940 2026-09-09T05:51:20Z SSapaty (WMF) 28955 first draft: migrate build service job to ptd 2455940 wikitext text/x-wiki ca2mea4c3f07qr2zbcjnfgvhh9bc9k2 2455941 2455940 2026-09-09T05:51:49Z SSapaty (WMF) 28955 /* See also */ 2455941 wikitext text/x-wiki 3plmifkjavhzyucpu8un1wscopxlwdp