Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.3 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2417479 2416333 2026-05-20T23:54:01Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2417479 wikitext text/x-wiki nb3nclathsh657m5rwlvldhlsxa3qbu 2417481 2417479 2026-05-21T00:03:18Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2417481 wikitext text/x-wiki e4bncr3679w4zzy9x825z0a5na8mazi 2417654 2417481 2026-05-21T09:48:39Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers 2417654 wikitext text/x-wiki t7xi70ov40agpk4v6tsaunxfikngue0 2417705 2417654 2026-05-21T10:10:46Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers 2417705 wikitext text/x-wiki qbktcf2g7zpsedlapmddh1gtpnujskg 2417750 2417705 2026-05-21T10:48:35Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 2417750 wikitext text/x-wiki bhb0ni909s0qxrytfxmi9w7tgtupazj 2417753 2417750 2026-05-21T10:50:24Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 2417753 wikitext text/x-wiki muy6hnzm98a158rpeyur8emm5f77hyu 2417762 2417753 2026-05-21T10:54:21Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 2417762 wikitext text/x-wiki enripcv07vkzcsisitwhk7hpxe00ko9 2417763 2417762 2026-05-21T10:55:07Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 2417763 wikitext text/x-wiki jgatk0f6d146esudayjwrbkvqtcync7 2417765 2417763 2026-05-21T10:55:40Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 2417765 wikitext text/x-wiki kpe3bx8mytwf47ran6tng4eziw5ayvm 2417766 2417765 2026-05-21T10:56:20Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 2417766 wikitext text/x-wiki 4vh42gth0lcdce4tyhaz2txwaxbd8h6 2417769 2417766 2026-05-21T10:57:11Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 2417769 wikitext text/x-wiki 6kxqfyuzz2bea9gna24lrhlpenamcq7 2417771 2417769 2026-05-21T10:57:57Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 2417771 wikitext text/x-wiki izt75doodvz81n4lew9cskvioim6asz 0 3387 2417154 2387032 2026-05-20T15:36:43Z EEggleston (WMF) 41072 /* External-facing */ 2417154 wikitext text/x-wiki 221vwdu8jlsgvpic9be2xvfgdjuhn0u 2417418 2417154 2026-05-20T20:59:13Z Star walker 40724 Tidied up article 2417418 wikitext text/x-wiki cwis0okcw0xyyoyltzam5f62u0noqnm 0 4108 2417220 2416573 2026-05-20T16:03:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1290014]] to Thursday, May 21 UTC afternoon backport window 2417220 wikitext text/x-wiki t2qujgx2kdca85viwwb6a4u5xwxhn80 2417239 2417220 2026-05-20T16:11:42Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-05-20}} */ UTC-late infra window 2417239 wikitext text/x-wiki 8dxs3k2z802xs5h1o260t0xyncq1x1w 2417251 2417239 2026-05-20T16:27:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1287433]] to Wednesday, May 20 UTC late backport window 2417251 wikitext text/x-wiki pjmesurhsmd7z0797s5xi0srsa8o8kg 2417354 2417251 2026-05-20T18:54:06Z EEggleston (WMF) 41072 Add one-off window 2417354 wikitext text/x-wiki 1a5jza3rc9j15i4jgylbn9ridvcgbgo 2417355 2417354 2026-05-20T18:54:41Z EEggleston (WMF) 41072 Fix time 2417355 wikitext text/x-wiki 9siloj6pdke3n53gl5hs6d805dmqc99 2417375 2417355 2026-05-20T20:04:25Z ScheduleDeploymentBot 37566 Add [[gerrit:1281901]] to Thursday, May 21 UTC afternoon backport window 2417375 wikitext text/x-wiki 94twvotb34sdzpkw3d51l18p2bwvhu8 2417483 2417375 2026-05-21T00:29:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1290035]] to Thursday, May 21 UTC afternoon backport window 2417483 wikitext text/x-wiki m31mpma5izry27qtox31yjlu27scec1 2417774 2417483 2026-05-21T10:59:15Z ScheduleDeploymentBot 37566 Add [[gerrit:1290088]] to Thursday, May 21 UTC afternoon backport window 2417774 wikitext text/x-wiki 7giv3xljcjh636wz017jh1sqp60n73a 2417842 2417774 2026-05-21T11:58:23Z ScheduleDeploymentBot 37566 Add [[gerrit:1290032]] to Thursday, May 21 UTC afternoon backport window 2417842 wikitext text/x-wiki ie2xed0kewc5xh7t6g0x4xhv0o6ubve 0 6479 2417687 2415349 2026-05-21T10:01:04Z Meno25 1446 update redirected link 2417687 wikitext text/x-wiki g4tm3ltaxg4ltx825jnq1ajsjbcwhrb 2417690 2417687 2026-05-21T10:01:37Z Meno25 1446 update redirected link 2417690 wikitext text/x-wiki fi9j9bow7f3m39hruo8krn7lpoaks04 498 6726 2417475 2416459 2026-05-20T23:35:29Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2417475 wikitext text/x-wiki f9tor88wntk6j0sbwum3q0r8oxuhllc 2417476 2417475 2026-05-20T23:35:32Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api 2417476 wikitext text/x-wiki 5mxpnk67z65umz6yxbpyp3sfqo0w2bj 2417477 2417476 2026-05-20T23:42:08Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2417477 wikitext text/x-wiki qrdnyhotjzwalq8bangd5g6yfycb2gf 2417478 2417477 2026-05-20T23:50:16Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2417478 wikitext text/x-wiki sqip59go5jtunsx9axbor4622sgnl50 2417566 2417478 2026-05-21T09:09:53Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers 2417566 wikitext text/x-wiki 35mlenolc83pv9m8uwahexrqa1b19xp 2417567 2417566 2026-05-21T09:09:56Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers 2417567 wikitext text/x-wiki dh4hzl2fisk1lqpn1kcgoo624m03yt4 2417570 2417567 2026-05-21T09:13:46Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers 2417570 wikitext text/x-wiki 7w1cl1bw5pa2y33tm7fd37ctnypc74x 2417596 2417570 2026-05-21T09:29:07Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers 2417596 wikitext text/x-wiki c9lvo83ja4g9ceo0blbo5yv6n4apuaq 2417612 2417596 2026-05-21T09:33:18Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers 2417612 wikitext text/x-wiki rxxghxwmb7s3b7dqtfvkmhxubb7hl9z 2417649 2417612 2026-05-21T09:48:15Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers 2417649 wikitext text/x-wiki 2hymvk1vzno9wekfhpe8do8pz1ky8zm 0 7919 2416730 2416729 2026-05-20T11:59:24Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7008.magru.wmnet 2416730 wikitext text/x-wiki 4fn0jzsx9h2vljttyrslnq08y8g8qnm 2416731 2416730 2026-05-20T11:59:26Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp700[7-8].magru.wmnet} and A:cp 2416731 wikitext text/x-wiki 7p8n6xvulgut7cl267go7pa2n66653v 2416732 2416731 2026-05-20T12:01:33Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1265-1268].eqiad.wmnet 2416732 wikitext text/x-wiki lb3yoyfd426e9as03hwfbff6ym2dlwf 2416733 2416732 2026-05-20T12:01:35Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1265-1268].eqiad.wmnet 2416733 wikitext text/x-wiki cwfa4kq1hlowxm2rtxjie4yjwip5zb1 2416734 2416733 2026-05-20T12:01:45Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1269-1272].eqiad.wmnet 2416734 wikitext text/x-wiki 2u1b7mpvabqmovn3ghrnfpigwdvok5h 2416735 2416734 2026-05-20T12:02:46Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1011.eqiad.wmnet with OS trixie 2416735 wikitext text/x-wiki iln0h9mc2bv95tmea0qvvk8478pyex0 2416736 2416735 2026-05-20T12:04:32Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1269-1272].eqiad.wmnet 2416736 wikitext text/x-wiki a37xog2dt0god0c576sr9hpp2oug4wh 2416737 2416736 2026-05-20T12:04:34Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp701[5-6].magru.wmnet} and A:cp 2416737 wikitext text/x-wiki nqhj00s228psgghnltwkbmlkuc05em5 2416738 2416737 2026-05-20T12:05:01Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=99) rolling reboot on A:cephosd-eqiad 2416738 wikitext text/x-wiki cixn88703qjmujw69w3zy4krrdazz7g 2416739 2416738 2026-05-20T12:05:43Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet 2416739 wikitext text/x-wiki i5w62lz48ksqso2ngeisu2emz9yvfkd 2416740 2416739 2026-05-20T12:05:46Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet 2416740 wikitext text/x-wiki if4kx2entvxqmsiljtcvml5l1od68ty 2416741 2416740 2026-05-20T12:05:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet 2416741 wikitext text/x-wiki 94yawjo52a0pdxj5q61sh2zzqjc6mru 2416742 2416741 2026-05-20T12:06:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2038.codfw.wmnet 2416742 wikitext text/x-wiki ihp69scybdn39ujphkaqip3mpvnl9nq 2416743 2416742 2026-05-20T12:09:57Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2162: Migration of db2162.codfw.wmnet completed 2416743 wikitext text/x-wiki iw1gqwlscagcdb4e5p99rfqvht0nd42 2416744 2416743 2026-05-20T12:10:00Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2416744 wikitext text/x-wiki 0ues1q0bzh8k9f4yd6qxqcns04xgzas 2416745 2416744 2026-05-20T12:10:22Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2038.codfw.wmnet 2416745 wikitext text/x-wiki 5hw9yb0xd0bfn4m2682qu29z6yveuot 2416746 2416745 2026-05-20T12:11:55Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1269-1272].eqiad.wmnet 2416746 wikitext text/x-wiki eu4bswyyxr0v5k6565i19fr59v6mfdq 2416747 2416746 2026-05-20T12:11:57Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1269-1272].eqiad.wmnet 2416747 wikitext text/x-wiki aqx9g1tr3j8vmfmf5ss1pp14nqz4gyq 2416748 2416747 2026-05-20T12:12:07Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1273-1276].eqiad.wmnet 2416748 wikitext text/x-wiki niaa14oumq9j5t5j57chjmtq9tkh7it 2416749 2416748 2026-05-20T12:14:21Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1273-1276].eqiad.wmnet 2416749 wikitext text/x-wiki dx5vn2vd12h4whel53yq5vkls5boyut 2416751 2416749 2026-05-20T12:15:09Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid public cluster: Reboot Druid nodes 2416751 wikitext text/x-wiki l0z99weh3dajb6koikv6kehtmaj4sqc 2416752 2416751 2026-05-20T12:15:44Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2038.codfw.wmnet 2416752 wikitext text/x-wiki m36whwxweu67lux67q8r5afj27lkatr 2416753 2416752 2026-05-20T12:15:55Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2038.codfw.wmnet 2416753 wikitext text/x-wiki hnnsw0vifxhahi1kgxe90rulztvyzmm 2416755 2416753 2026-05-20T12:16:38Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7015.magru.wmnet 2416755 wikitext text/x-wiki 8v2z072e03wj43djy0dxnjknvfo61gj 2416757 2416755 2026-05-20T12:17:10Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1011.eqiad.wmnet with reason: host reimage 2416757 wikitext text/x-wiki 5gx6ixzmphxhmoqtj77b833lv7j93lo 2416758 2416757 2026-05-20T12:17:50Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1289938|Revert "ApiEditPage: Update request in main context before calling attemptSave()" (T426751)]] 2416758 wikitext text/x-wiki e9kdaql8la7u0ylrxmancb00e1sof7i 2416759 2416758 2026-05-20T12:19:44Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1289938|Revert "ApiEditPage: Update request in main context before calling attemptSave()" (T426751)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416759 wikitext text/x-wiki piewwg6jb3kqrgb5fmkrlmh67ht8hi0 2416760 2416759 2026-05-20T12:21:48Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1273-1276].eqiad.wmnet 2416760 wikitext text/x-wiki hq03fqxvkanvlaxgzi7kdrhbic4jy02 2416761 2416760 2026-05-20T12:21:51Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1273-1276].eqiad.wmnet 2416761 wikitext text/x-wiki rpniqns2n36ey3vwy9k9nrr72cudgzg 2416762 2416761 2026-05-20T12:22:01Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1277-1280].eqiad.wmnet 2416762 wikitext text/x-wiki lti84e8ouheya5l2eujai1t2cktq3ha 2416763 2416762 2026-05-20T12:22:15Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2416763 wikitext text/x-wiki kzkj5owep7qtny68hg93jlnpmsc6w0u 2416764 2416763 2026-05-20T12:24:16Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1277-1280].eqiad.wmnet 2416764 wikitext text/x-wiki c5skriui8p0h82eme4d1ng4esa1e6nn 2416765 2416764 2026-05-20T12:25:14Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1011.eqiad.wmnet with reason: host reimage 2416765 wikitext text/x-wiki fi6c1s2eqlyl4bbiddxl5kmgt7dy4mj 2416766 2416765 2026-05-20T12:25:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2039.codfw.wmnet 2416766 wikitext text/x-wiki kw8smi8rdbo7lt4yc4j64xo239fye6b 2416767 2416766 2026-05-20T12:26:28Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289938|Revert "ApiEditPage: Update request in main context before calling attemptSave()" (T426751)]] (duration: 08m 37s) 2416767 wikitext text/x-wiki 2nkzg3kqv8a4adcntfdt5nkqt6by681 2416768 2416767 2026-05-20T12:26:36Z Stashbot 7414 taavi@cumin1003: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org 2416768 wikitext text/x-wiki 3cyhkmzain09kbsvubjyqy1unlux6or 2416769 2416768 2026-05-20T12:31:06Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2416769 wikitext text/x-wiki e8es55dca6woxoh5s0nhmeqe9f2dwae 2416770 2416769 2026-05-20T12:31:08Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2416770 wikitext text/x-wiki 2zqkmng2vad05b3guxomjgev4rht3av 2416771 2416770 2026-05-20T12:32:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2416771 wikitext text/x-wiki fx9chqkv9uz0mirbbcppixulo1ozaux 2416772 2416771 2026-05-20T12:33:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2416772 wikitext text/x-wiki 9tg9hwqcp04enr5455rlb5gkhup1t2l 2416773 2416772 2026-05-20T12:33:15Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2416773 wikitext text/x-wiki 3v7edhqyaydus6snpfu39t87h5fz4ju 2416774 2416773 2026-05-20T12:33:32Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2416774 wikitext text/x-wiki f6s1kyz621ixfuvf5o6x2olbmy8waal 2416775 2416774 2026-05-20T12:33:44Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1258: Upgrading db1258.eqiad.wmnet 2416775 wikitext text/x-wiki jnkdyq7f846fisz7vm2jam1dl83nx2m 2416776 2416775 2026-05-20T12:33:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1258: Upgrading db1258.eqiad.wmnet 2416776 wikitext text/x-wiki 8eq1klru4ti2i7nfvmlgd2k8h29hhsq 2416777 2416776 2026-05-20T12:34:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1277-1280].eqiad.wmnet 2416777 wikitext text/x-wiki muflvveem93srmtpv7i7imjn32efp90 2416778 2416777 2026-05-20T12:34:51Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1277-1280].eqiad.wmnet 2416778 wikitext text/x-wiki mizufc9e8rjdmclyq8ujg55xw2dqoid 2416779 2416778 2026-05-20T12:35:00Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1281-1284].eqiad.wmnet 2416779 wikitext text/x-wiki jaoixhqebktujoxu4bx032soa2ptbep 2416780 2416779 2026-05-20T12:35:48Z Stashbot 7414 taavi@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1002.wikimedia.org 2416780 wikitext text/x-wiki t7nw4rysn9hak8vvixu0zc7h36vuvw1 2416781 2416780 2026-05-20T12:35:54Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet 2416781 wikitext text/x-wiki ddrts824yjet3bukzrqzmfn82awvvjy 2416782 2416781 2026-05-20T12:36:05Z Stashbot 7414 taavi@cumin1003: START - Cookbook sre.hosts.reboot-single for host clouddumps1001.wikimedia.org 2416782 wikitext text/x-wiki mggatbm3cs4v9g12tryx0tj1dpxyox0 2416783 2416782 2026-05-20T12:36:50Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1258.eqiad.wmnet with OS trixie 2416783 wikitext text/x-wiki 5cxbsahpmok6j7pq9nz64on8q8g54ee 2416784 2416783 2026-05-20T12:37:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2039.codfw.wmnet 2416784 wikitext text/x-wiki 6i1p82gmjrfprbz7afgx1baxkc4b40o 2416785 2416784 2026-05-20T12:37:32Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2416785 wikitext text/x-wiki azjc5d15blxug30r5f874y2j112041p 2416786 2416785 2026-05-20T12:37:48Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1281-1284].eqiad.wmnet 2416786 wikitext text/x-wiki 5glpfawzofbd04p1ldi43ual6lna6bn 2416790 2416786 2026-05-20T12:40:55Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2416790 wikitext text/x-wiki p8ff47qnpqt8h5zn2vrekcaeuo8318d 2416791 2416790 2026-05-20T12:41:06Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2416791 wikitext text/x-wiki io3xe3ioqf366jkqc3ahz12cxki02i7 2416794 2416791 2026-05-20T12:42:10Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1011.eqiad.wmnet with OS trixie 2416794 wikitext text/x-wiki p5suvewor5xfwzskp1odzavsm3y4z38 2416795 2416794 2026-05-20T12:42:32Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet 2416795 wikitext text/x-wiki hxon02igqj9dwb7s1bup4ca4diom1mw 2416797 2416795 2026-05-20T12:42:38Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet 2416797 wikitext text/x-wiki l9cyl9deceadhw7z5ngpxtbx7osojep 2416798 2416797 2026-05-20T12:42:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2039.codfw.wmnet 2416798 wikitext text/x-wiki j7djr40p605k0a1dtwk5w179siug0q0 2416799 2416798 2026-05-20T12:42:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet 2416799 wikitext text/x-wiki rcf91twofcakt860xspopr15xrtmu0z 2416800 2416799 2026-05-20T12:42:43Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2039.codfw.wmnet 2416800 wikitext text/x-wiki lwzamhpqiiudbjq0s1jeawsoi0kclv7 2416803 2416800 2026-05-20T12:43:19Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet 2416803 wikitext text/x-wiki 168ylwsb043d2k0kexdqwzjd2e0sylc 2416806 2416803 2026-05-20T12:44:32Z Stashbot 7414 taavi@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1001.wikimedia.org 2416806 wikitext text/x-wiki n0inj85pj9ga2tfu3671iczgepyqgbg 2416807 2416806 2026-05-20T12:45:23Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1281-1284].eqiad.wmnet 2416807 wikitext text/x-wiki pxnl2xmltuew1zz7esz27uoof41mwd7 2416808 2416807 2026-05-20T12:45:25Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1281-1284].eqiad.wmnet 2416808 wikitext text/x-wiki gdyt8c4ulus1jqectullh9qky29q0e6 2416809 2416808 2026-05-20T12:45:34Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1285-1288].eqiad.wmnet 2416809 wikitext text/x-wiki nl4x4531d6y460ec3qt3rbjfo3zhdo8 2416812 2416809 2026-05-20T12:47:52Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1285-1288].eqiad.wmnet 2416812 wikitext text/x-wiki 5liijs83aua7rizyv0rrbi2024djfku 2416814 2416812 2026-05-20T12:48:19Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[2183-2184].codfw.wmnet with reason: restart 2416814 wikitext text/x-wiki ofhaicwxjmaxu3rn6rx01lexfxya3eg 2416815 2416814 2026-05-20T12:49:58Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet 2416815 wikitext text/x-wiki norikz2v84xu5zsgr7jy37psaejm7dw 2416816 2416815 2026-05-20T12:50:00Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet 2416816 wikitext text/x-wiki 0bfjwoli0dqsauwrgg3hit3w4k0vhb3 2416817 2416816 2026-05-20T12:50:04Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet 2416817 wikitext text/x-wiki piqqy2680ngvsc2rtk5d8un2e9enyyp 2416820 2416817 2026-05-20T12:52:16Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1289902|hCaptcha: Exempt Wikibase entity namespaces from edit/create triggers (T426829)]] 2416820 wikitext text/x-wiki 9pi9b9ckcllt1gpcf792xrfv67txz2f 2416821 2416820 2026-05-20T12:52:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2040.codfw.wmnet 2416821 wikitext text/x-wiki go2dztu5epviuv3ul2n8xsso47vj6kh 2416822 2416821 2026-05-20T12:53:20Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1258.eqiad.wmnet with reason: host reimage 2416822 wikitext text/x-wiki 3k908vyiswrku71m0an8zqdsy0rgc0a 2416823 2416822 2026-05-20T12:54:13Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1289902|hCaptcha: Exempt Wikibase entity namespaces from edit/create triggers (T426829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416823 wikitext text/x-wiki ryed0hrmtclv98eufubmz8h0ddhuwx7 2416824 2416823 2026-05-20T12:55:08Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet 2416824 wikitext text/x-wiki nuh89x7aqpfr9v8f5nfl681pf3ydn6h 2416826 2416824 2026-05-20T12:55:47Z Stashbot 7414 fnegri@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddb1015.eqiad.wmnet with OS trixie 2416826 wikitext text/x-wiki pjwuwtn0jqe5lxzp1kqs4n75qnf0vbn 2416827 2416826 2026-05-20T12:55:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1285-1288].eqiad.wmnet 2416827 wikitext text/x-wiki fb5inmy0o9lwyut9jo9msf7pbdjoc7u 2416828 2416827 2026-05-20T12:55:51Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1285-1288].eqiad.wmnet 2416828 wikitext text/x-wiki b8xwxlodsrhmtfcxk0awue11qqn8wjw 2416829 2416828 2026-05-20T12:56:02Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1289,1291-1293].eqiad.wmnet 2416829 wikitext text/x-wiki jlnxxc9dxau6dfjpoorme19ltnb4221 2416830 2416829 2026-05-20T12:56:31Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2040.codfw.wmnet 2416830 wikitext text/x-wiki oq1aen3c6ftmm1pmi4389ownx85pg4t 2416832 2416830 2026-05-20T12:57:33Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2416832 wikitext text/x-wiki 7klnw53ho5d1judpddvy24w0eal2kp1 2416833 2416832 2026-05-20T12:57:47Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet 2416833 wikitext text/x-wiki 4bhm88g7d0n3f70fq8x13dz6mwva5v4 2416836 2416833 2026-05-20T12:58:05Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1015.eqiad.wmnet 2416836 wikitext text/x-wiki g3vyw6e74r33sr60utjj8a1ft4eqedw 2416837 2416836 2026-05-20T12:58:08Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1015.eqiad.wmnet 2416837 wikitext text/x-wiki 4ca05ijqzz34ybzrw43wk4ej9pbp3p2 2416839 2416837 2026-05-20T12:58:12Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1289,1291-1293].eqiad.wmnet 2416839 wikitext text/x-wiki ftf22uzg9028lbl818eek3m1t15yirw 2416840 2416839 2026-05-20T12:58:23Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet 2416840 wikitext text/x-wiki py514h3izp6wfkqhjy9mh6h1ttg5kkc 2416841 2416840 2026-05-20T12:58:25Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp701[5-6].magru.wmnet} and A:cp 2416841 wikitext text/x-wiki 9xh3b8eoq8ll68dj35t2vsbrysrrwng 2416842 2416841 2026-05-20T12:59:03Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1258.eqiad.wmnet with reason: host reimage 2416842 wikitext text/x-wiki t298qw416ubpse1pa6twdj769kinz7x 2416846 2416842 2026-05-20T13:00:02Z Stashbot 7414 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2416846 wikitext text/x-wiki j5ap97hn7vm4e7wbcbb27v64cv2ytq8 2416850 2416846 2026-05-20T13:01:32Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet 2416850 wikitext text/x-wiki ccbsuinfi8ca6r5ql8gvw778v7rx6b1 2416851 2416850 2026-05-20T13:01:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet 2416851 wikitext text/x-wiki t8b5oy637awych8iigkj3dp9hkd8xhp 2416853 2416851 2026-05-20T13:01:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet 2416853 wikitext text/x-wiki ofbazr7s8zc9vg5zlcyo5ij9w9v8pwr 2416854 2416853 2026-05-20T13:01:46Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289902|hCaptcha: Exempt Wikibase entity namespaces from edit/create triggers (T426829)]] (duration: 09m 31s) 2416854 wikitext text/x-wiki 54z50187xp2gopy8wuthldsvwnp0zyb 2416856 2416854 2026-05-20T13:01:53Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2040.codfw.wmnet 2416856 wikitext text/x-wiki gjw9qpxkapaio5brkunz8tsv3dh6qso 2416857 2416856 2026-05-20T13:02:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2040.codfw.wmnet 2416857 wikitext text/x-wiki njbie3957g2j9fqdlld329qt2qjmb3u 2416858 2416857 2026-05-20T13:02:25Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp403[7-8].ulsfo.wmnet} and A:cp 2416858 wikitext text/x-wiki 3or6lehkx7kkig41326iaatbh7kn3af 2416859 2416858 2026-05-20T13:02:43Z Stashbot 7414 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2416859 wikitext text/x-wiki prdwzv17gsm51idsinfudim4dpdc2f7 2416861 2416859 2026-05-20T13:03:44Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet 2416861 wikitext text/x-wiki mz1hlnb3qkn2l9ruj2jip8v5p6kgajv 2416864 2416861 2026-05-20T13:04:54Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1012.eqiad.wmnet with OS trixie 2416864 wikitext text/x-wiki lea77mbybvg1ga0w7heruayxdxku450 2416866 2416864 2026-05-20T13:06:03Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2416866 wikitext text/x-wiki ewqfzd1xs9kdcxh9xzjkdyoc1ts10gf 2416868 2416866 2026-05-20T13:06:18Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet 2416868 wikitext text/x-wiki 6wvf2sm5he4y3yv5jpyl6e8w27k7nkf 2416871 2416868 2026-05-20T13:07:47Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1289446|Log editing_start and article_saved events for control group (T422146)]] 2416871 wikitext text/x-wiki aj841ndamiwpov3s7etgv3d2akwqlpy 2416872 2416871 2026-05-20T13:08:38Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2416872 wikitext text/x-wiki 1xk3s5gntm2559gneerkhamxbb4wooy 2416873 2416872 2026-05-20T13:09:43Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1289446|Log editing_start and article_saved events for control group (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416873 wikitext text/x-wiki 6jnpsmk2soffl2hzdj29omlmh3m298m 2416874 2416873 2026-05-20T13:11:30Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2416874 wikitext text/x-wiki dnj6lyoz5doz86cb9w5relf1qsqbfw6 2416875 2416874 2026-05-20T13:12:14Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4037.ulsfo.wmnet 2416875 wikitext text/x-wiki qc0jbic22zh8gyl2lr32egur87c83m4 2416876 2416875 2026-05-20T13:13:06Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2416876 wikitext text/x-wiki 1ob9uxq33b8blx4vpmqcyne25fv9rlt 2416881 2416876 2026-05-20T13:14:57Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet 2416881 wikitext text/x-wiki omtwwpclvaqzakbl72viksmy4oqt20i 2416882 2416881 2026-05-20T13:14:59Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet 2416882 wikitext text/x-wiki 1l3rvzz3szga4rtc3o74ml7zbssyikv 2416883 2416882 2026-05-20T13:15:05Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet 2416883 wikitext text/x-wiki q8996c9sqolj0wum53kqudv53r8qclr 2416884 2416883 2026-05-20T13:15:37Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet 2416884 wikitext text/x-wiki sv1ej2iftqtnrdniezr9u9lkwm3fkta 2416885 2416884 2026-05-20T13:15:43Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289446|Log editing_start and article_saved events for control group (T422146)]] (duration: 07m 55s) 2416885 wikitext text/x-wiki anogy7u574afspaxsalzq1eclvam0bg 2416887 2416885 2026-05-20T13:16:22Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet 2416887 wikitext text/x-wiki m5koqg0dx9jkgvrlua9igz5hse4z0ep 2416888 2416887 2026-05-20T13:17:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1055.eqiad.wmnet to cluster codfw and group A 2416888 wikitext text/x-wiki het2aare4mz0t8naiis1tjq3f7u56nv 2416889 2416888 2026-05-20T13:17:10Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1055.eqiad.wmnet to cluster codfw and group A 2416889 wikitext text/x-wiki h91um4k7uo6bbp9l9ql8x3rokt5wekp 2416890 2416889 2026-05-20T13:17:12Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1258.eqiad.wmnet with OS trixie 2416890 wikitext text/x-wiki cpo2u6mks0ev8fssm9arnvqcb0upfie 2416891 2416890 2026-05-20T13:17:18Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1055.eqiad.wmnet to cluster eqiad and group A 2416891 wikitext text/x-wiki axkpwwcz50oh59s8uof90kpaixtnslw 2416892 2416891 2026-05-20T13:18:00Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2416892 wikitext text/x-wiki aba5948dixyu6tg4s7ejg8b6jj2lb9e 2416894 2416892 2026-05-20T13:18:14Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2416894 wikitext text/x-wiki c9j3hrrrjq800ixe61c41ia8qpow4id 2416895 2416894 2026-05-20T13:18:18Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2416895 wikitext text/x-wiki 7n3a9uoyoxwid1iwflq4ol57tamdwm1 2416897 2416895 2026-05-20T13:18:30Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.reimage for host db2183.codfw.wmnet with OS trixie 2416897 wikitext text/x-wiki 328rm8zhuncozzavot1zxxvcp33xgtp 2416899 2416897 2026-05-20T13:18:35Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2416899 wikitext text/x-wiki h0z3br7y3va8la27hyb39dk4rj9d6lu 2416900 2416899 2026-05-20T13:18:40Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2416900 wikitext text/x-wiki hiuluw6tzupegzwia0sxq4emdy862el 2416901 2416900 2026-05-20T13:18:50Z Stashbot 7414 arthurtaylor@deploy1003: Started scap sync-world: Backport for [[gerrit:1289736|Disable support for PHP-serialized EntityData on Beta / Test Wikidata (T98035)]] 2416901 wikitext text/x-wiki lckc4mav7kk57s8bj1b4xu9rcp12376 2416902 2416901 2026-05-20T13:18:52Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.move-vlan for host db2183 2416902 wikitext text/x-wiki cdudz01akcsf3k4vqwrtjb6s87aydh5 2416903 2416902 2026-05-20T13:19:18Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1012.eqiad.wmnet with reason: host reimage 2416903 wikitext text/x-wiki kp0vdjfr5sjizl7e0zgcm9rlvj3lhlh 2416904 2416903 2026-05-20T13:19:24Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1258: Migration of db1258.eqiad.wmnet completed 2416904 wikitext text/x-wiki fuyuwl7rvhuih7mwlqrspjt0l87w78f 2416905 2416904 2026-05-20T13:19:38Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2416905 wikitext text/x-wiki et4202q45v735228v7vpuolvyqw1lkb 2416906 2416905 2026-05-20T13:19:48Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2416906 wikitext text/x-wiki h51a98idhuohxtgojsjcpta6bci7cfa 2416907 2416906 2026-05-20T13:20:01Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2416907 wikitext text/x-wiki 9z4ieojzvbhx5iw9pln1epanjwy8nax 2416908 2416907 2026-05-20T13:20:09Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2416908 wikitext text/x-wiki m3avpehgagugc0snac0gpyq1olt9qmo 2416911 2416908 2026-05-20T13:20:45Z Stashbot 7414 arthurtaylor@deploy1003: arthurtaylor: Backport for [[gerrit:1289736|Disable support for PHP-serialized EntityData on Beta / Test Wikidata (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416911 wikitext text/x-wiki a8bbomeomc4547ud36jzmf3wv5l3fg7 2416913 2416911 2026-05-20T13:22:03Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet 2416913 wikitext text/x-wiki 5s6obq11y6px57nhjvkwhiz3th7v6t0 2416914 2416913 2026-05-20T13:22:05Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet 2416914 wikitext text/x-wiki 1w6xg0b82j9s460vexxirw6wts1joos 2416915 2416914 2026-05-20T13:22:08Z Stashbot 7414 arthurtaylor@deploy1003: arthurtaylor: Continuing with deployment 2416915 wikitext text/x-wiki 5dyvf24mefxolhzrqmg66pn4ps0hyvl 2416916 2416915 2026-05-20T13:22:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet 2416916 wikitext text/x-wiki rffpnn5wbegq7hpivjbk8ibin24nt8l 2416917 2416916 2026-05-20T13:22:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet 2416917 wikitext text/x-wiki ii9htn7dxg9vfrb1mxdsicc5cx32qxm 2416918 2416917 2026-05-20T13:22:37Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2416918 wikitext text/x-wiki nuxzrutm1p1cfdvodatf0lnfx2dveuu 2416919 2416918 2026-05-20T13:22:39Z Stashbot 7414 root@cumin1003: START - Cookbook sre.dns.netbox 2416919 wikitext text/x-wiki 98hfm5yp9elc88e77ae7zl79tnwdvxf 2416920 2416919 2026-05-20T13:22:42Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2025.codfw.wmnet 2416920 wikitext text/x-wiki j11ivv5aumn2u4usm5zvvoctuljs67p 2416921 2416920 2026-05-20T13:22:46Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2416921 wikitext text/x-wiki i9jbhnoq18m9yq6m5rwmtx93hedk27e 2416922 2416921 2026-05-20T13:22:50Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2416922 wikitext text/x-wiki 2urzzn2c0h1miv0wutx33p2mji25y77 2416923 2416922 2026-05-20T13:23:00Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2416923 wikitext text/x-wiki f2hqg5invnwi66uzc19vb0tjw9b6y4k 2416924 2416923 2026-05-20T13:23:22Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1012.eqiad.wmnet with reason: host reimage 2416924 wikitext text/x-wiki o5dxmpeb7c1houou14dukirtmbrf0hk 2416926 2416924 2026-05-20T13:24:16Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet 2416926 wikitext text/x-wiki 2mqvzs33ziwnwtga8x1dfu1758fqxgk 2416927 2416926 2026-05-20T13:25:00Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2416927 wikitext text/x-wiki 8qj9myqtyqqxcbwgy296w579i5bstid 2416928 2416927 2026-05-20T13:25:27Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2416928 wikitext text/x-wiki 1j30dtiznyjqcio6vsyxmi6c6eq7vbs 2416930 2416928 2026-05-20T13:26:03Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1055.eqiad.wmnet to cluster eqiad and group A 2416930 wikitext text/x-wiki etsi3b53dg9cjrndkx5modj0paf9ak1 2416931 2416930 2026-05-20T13:26:17Z Stashbot 7414 arthurtaylor@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289736|Disable support for PHP-serialized EntityData on Beta / Test Wikidata (T98035)]] (duration: 07m 26s) 2416931 wikitext text/x-wiki 98kr5gq2h7wnxp0qsw5suymwpgicw5v 2416932 2416931 2026-05-20T13:26:57Z Stashbot 7414 root@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host db2183 - root@cumin1003" 2416932 wikitext text/x-wiki blmcmvhpot6oaa700hzpo6zbgl86azr 2416933 2416932 2026-05-20T13:27:02Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host db2183 - root@cumin1003" 2416933 wikitext text/x-wiki fozzrmtj3vsutzfw5tmfeg3iy7sfxog 2416934 2416933 2026-05-20T13:27:04Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2416934 wikitext text/x-wiki l52ib2yeinpnxj4w56yq43yttdqbhid 2416935 2416934 2026-05-20T13:27:06Z Stashbot 7414 root@cumin1003: START - Cookbook sre.dns.wipe-cache db2183.codfw.wmnet 6.0.192.10.in-addr.arpa 6.0.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2416935 wikitext text/x-wiki c8yx9yfvfbmnzp1e7zueio3j6gs9si5 2416936 2416935 2026-05-20T13:27:08Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db2183.codfw.wmnet 6.0.192.10.in-addr.arpa 6.0.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2416936 wikitext text/x-wiki 0cqs7l1fqsxork75lmsdd1cgnpgssou 2416937 2416936 2026-05-20T13:27:10Z Stashbot 7414 root@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db2183 2416937 wikitext text/x-wiki 408k0aqytvuqf8ujn7drhmz8opdtuum 2416938 2416937 2026-05-20T13:28:55Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudelastic1012.eqiad.wmnet 2416938 wikitext text/x-wiki hf1e0jxc0f2d1dah5tg3f9r8haqacaj 2416939 2416938 2026-05-20T13:29:37Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2416939 wikitext text/x-wiki 2fr4j79uepdvrugtpi68qofw3o8udil 2416940 2416939 2026-05-20T13:29:53Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2416940 wikitext text/x-wiki ifwz03bi0hr66qd8scag7hs4uibsx5b 2416941 2416940 2026-05-20T13:30:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1056.eqiad.wmnet 2416941 wikitext text/x-wiki borarstjj4yl7brl4lmsk5e6s03b7kb 2416942 2416941 2026-05-20T13:31:24Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2026.codfw.wmnet 2416942 wikitext text/x-wiki 0zz003l58q5rrl48856n0utntke823e 2416943 2416942 2026-05-20T13:31:45Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2183 2416943 wikitext text/x-wiki ttsukt86gxkoipndgwh59tayk1bjydf 2416944 2416943 2026-05-20T13:31:47Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host db2183 2416944 wikitext text/x-wiki d3v773qd7yam5cokuxkwie9vbktjbnt 2416945 2416944 2026-05-20T13:35:11Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudelastic1012.eqiad.wmnet 2416945 wikitext text/x-wiki r64ggoud6cg3nqbbnkl2by9b9saal2u 2416946 2416945 2026-05-20T13:35:59Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1056 2416946 wikitext text/x-wiki 6lxlze36q3xsrse59l2k57rhthb73qm 2416947 2416946 2026-05-20T13:36:28Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1289953|composer.json: Upgrading symfony/yaml (v7.4.6 => v7.4.12) (T426845)]] 2416947 wikitext text/x-wiki juq4krzm7projkpmqllstpwhkspfv3l 2416948 2416947 2026-05-20T13:37:31Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1056 2416948 wikitext text/x-wiki kpk9c2vp6cu3nyqfx4f235xmtc6gw0j 2416949 2416948 2026-05-20T13:37:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet 2416949 wikitext text/x-wiki c3hgxjafvzutqmhnpintocwwmz0vqpe 2416950 2416949 2026-05-20T13:38:17Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1056.eqiad.wmnet 2416950 wikitext text/x-wiki bhz99t6djrg8uuzpupkwy5itu7zbtw1 2416951 2416950 2026-05-20T13:38:26Z Stashbot 7414 reedy@deploy1003: reedy: Backport for [[gerrit:1289953|composer.json: Upgrading symfony/yaml (v7.4.6 => v7.4.12) (T426845)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2416951 wikitext text/x-wiki mgq63tnugpxbt5z2awcc29q7mqv3i92 2416952 2416951 2026-05-20T13:38:29Z Stashbot 7414 moritzm: installing krb5 security updates 2416952 wikitext text/x-wiki m507x3sb14kx32dszfwqog5nt5fo9iu 2416953 2416952 2026-05-20T13:38:56Z Stashbot 7414 reedy@deploy1003: reedy: Continuing with deployment 2416953 wikitext text/x-wiki kt2491qxy3krguxdx04qmudad2drl9w 2416954 2416953 2026-05-20T13:41:43Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet 2416954 wikitext text/x-wiki tm5g3zwntqsmbin5evcxxbikli5ebwj 2416955 2416954 2026-05-20T13:41:45Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet 2416955 wikitext text/x-wiki q0lj459po8fvgvqaobfiv9bw23258gd 2416956 2416955 2026-05-20T13:41:49Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet 2416956 wikitext text/x-wiki kyhe6vjiza6s5rjog9f86m4wzxc8nam 2416957 2416956 2026-05-20T13:41:54Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1012.eqiad.wmnet with OS trixie 2416957 wikitext text/x-wiki jky4u9ptfpjm86jqvclle69byxgm1ln 2416958 2416957 2026-05-20T13:42:25Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet 2416958 wikitext text/x-wiki cud6uwxlnlj1qlc6m057sxsii6fq2s1 2416959 2416958 2026-05-20T13:42:27Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet 2416959 wikitext text/x-wiki 9kwp51untjlnjcc215lbf9253t2fy61 2416960 2416959 2026-05-20T13:43:08Z Stashbot 7414 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289953|composer.json: Upgrading symfony/yaml (v7.4.6 => v7.4.12) (T426845)]] (duration: 06m 39s) 2416960 wikitext text/x-wiki oxw37ucnnd08p1bkw86jcj3kw3rcd3m 2416961 2416960 2026-05-20T13:43:15Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudelastic1011.eqiad.wmnet 2416961 wikitext text/x-wiki 5jcvpkwfaaj6bymd97e4fp5vep5bxd2 2416962 2416961 2026-05-20T13:43:50Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2416962 wikitext text/x-wiki iy7cdp78oeyja3b2mo2my0t55smjn3x 2416963 2416962 2026-05-20T13:43:53Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet 2416963 wikitext text/x-wiki b9t7hc3vk0wr66kmyjfq5gvj6w2wnv0 2416964 2416963 2026-05-20T13:44:20Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2026.codfw.wmnet 2416964 wikitext text/x-wiki 1h5e3c01e84ywdsqpwhls0hjvbja1yb 2416965 2416964 2026-05-20T13:47:38Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2041.codfw.wmnet 2416965 wikitext text/x-wiki agkmcci2lvvbxmsuplhjvk0wnm2l6hg 2416966 2416965 2026-05-20T13:48:14Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet 2416966 wikitext text/x-wiki r3sqqu6u93izmp2fsno1p3ukf1kbq5i 2416967 2416966 2026-05-20T13:48:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet 2416967 wikitext text/x-wiki l9n1u5x8ja493acqoz5h9pl2qu4jk1a 2416968 2416967 2026-05-20T13:48:47Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet 2416968 wikitext text/x-wiki m55jkyek9bcfhfv8xe2ky2ok8dkx0mq 2416969 2416968 2026-05-20T13:48:52Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet 2416969 wikitext text/x-wiki hbknuk05f4hpmiksybxfh8x2wt39g4p 2416970 2416969 2026-05-20T13:49:27Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet 2416970 wikitext text/x-wiki 2lyspns07yrfkzujtgkwv3bmgcdxlzq 2416971 2416970 2026-05-20T13:49:34Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1013.eqiad.wmnet with OS trixie 2416971 wikitext text/x-wiki rxp00rbaraymhl4wauv7s95krnaa8ko 2416972 2416971 2026-05-20T13:49:39Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2183.codfw.wmnet with reason: host reimage 2416972 wikitext text/x-wiki je2lqik4vmp79dk8mu1l759v67xihgg 2416973 2416972 2026-05-20T13:49:47Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudelastic1011.eqiad.wmnet 2416973 wikitext text/x-wiki bju74tw7mo0lu0nc33zpuqv9dinbfv8 2416975 2416973 2026-05-20T13:52:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes 2416975 wikitext text/x-wiki pq7jzpgqfio0b8hk92o41j19rdjkwvr 2416976 2416975 2026-05-20T13:53:34Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2183.codfw.wmnet with reason: host reimage 2416976 wikitext text/x-wiki mfbj33cbl0w8xtr9dgmyvblqbc4w6ec 2416978 2416976 2026-05-20T13:53:56Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4038.ulsfo.wmnet 2416978 wikitext text/x-wiki oaa4hb8rfjny6i36t3s3789ax5ajx4j 2416979 2416978 2026-05-20T13:53:58Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp403[7-8].ulsfo.wmnet} and A:cp 2416979 wikitext text/x-wiki tsr4ceujyhefsdsqob59rlo56nzoezb 2416980 2416979 2026-05-20T13:56:08Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet 2416980 wikitext text/x-wiki 5mnefoy834w2eiln5igrxkwho1louu9 2416981 2416980 2026-05-20T13:56:11Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet 2416981 wikitext text/x-wiki pvcho8sz0hp2qiv45svramglkoifac9 2416982 2416981 2026-05-20T13:56:15Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet 2416982 wikitext text/x-wiki gab322xykqnubo2d6u6py3p3eftku09 2416983 2416982 2026-05-20T13:56:50Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet 2416983 wikitext text/x-wiki avxzsgml5is3cujkdagkfmc7z2nf04l 2416984 2416983 2026-05-20T14:00:22Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on P{cephosd100[4-5].eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad) 2416984 wikitext text/x-wiki fi06kiwtk6vaiqfvupc826cjuc7n8ve 2416985 2416984 2026-05-20T14:00:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2041.codfw.wmnet 2416985 wikitext text/x-wiki g3duz4uvboznveodzfa9is95hsj36gx 2416986 2416985 2026-05-20T14:01:41Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudelastic1010.eqiad.wmnet 2416986 wikitext text/x-wiki 90sceyginlil7d36fm64xanqpp9fhwp 2416987 2416986 2026-05-20T14:03:07Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet 2416987 wikitext text/x-wiki g2qsommo3z8e6dsldmw6f18lfew01a8 2416988 2416987 2026-05-20T14:03:10Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet 2416988 wikitext text/x-wiki kmhvmkrqdzpk0aiu53clsl20rjg41c1 2416989 2416988 2026-05-20T14:03:14Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet 2416989 wikitext text/x-wiki 3rhb2yralfsq7cj5f8oowfvqp2msrx1 2416990 2416989 2026-05-20T14:04:25Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1013.eqiad.wmnet with reason: host reimage 2416990 wikitext text/x-wiki 0gf1keqkfyw5rymxp7v6m1cpqf5fich 2416991 2416990 2026-05-20T14:04:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1258: Migration of db1258.eqiad.wmnet completed 2416991 wikitext text/x-wiki mbcknm6l68gvfskmgaveo672g8jutff 2416992 2416991 2026-05-20T14:04:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2416992 wikitext text/x-wiki nou3n9okgkplb6p62h4o64yj7jvxomd 2416993 2416992 2026-05-20T14:06:26Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2041.codfw.wmnet 2416993 wikitext text/x-wiki 11q9x2hkja3bdy60r7uzbk01m0rcf9g 2416994 2416993 2026-05-20T14:06:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2041.codfw.wmnet 2416994 wikitext text/x-wiki jp1hzdav2niuayhqcdz0fwk3449troi 2416995 2416994 2026-05-20T14:07:34Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2416995 wikitext text/x-wiki k1i42cbunca7dp1gvqmg6l4imd78dkz 2416996 2416995 2026-05-20T14:08:09Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2416996 wikitext text/x-wiki 5pbfdnxumkr59s8us4g7uhpgfhl16ur 2416997 2416996 2026-05-20T14:08:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet 2416997 wikitext text/x-wiki 4ifu6lxqgmio9b7u9ii0kdhgf6d43zp 2416998 2416997 2026-05-20T14:08:33Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2416998 wikitext text/x-wiki fpx81rjhnz7v24fc5mit8xqo67lo9w1 2416999 2416998 2026-05-20T14:08:43Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test2001.codfw.wmnet with OS bookworm 2416999 wikitext text/x-wiki lchrc2tnzwv8jq8hk3471rs1bveo9mq 2417000 2416999 2026-05-20T14:08:48Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2417000 wikitext text/x-wiki 6zal4369nrbtnhzhmzgov8k6ekh38vl 2417002 2417000 2026-05-20T14:09:28Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1013.eqiad.wmnet with reason: host reimage 2417002 wikitext text/x-wiki 1vu62bey3zqjzv1hm5scag9uzw3oox2 2417003 2417002 2026-05-20T14:10:08Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2417003 wikitext text/x-wiki 1zd28q7mqpp15uu58f7dbr6o4xntvhl 2417005 2417003 2026-05-20T14:10:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2417005 wikitext text/x-wiki nd5ibhntm5fncy3mooso2b6gtil65yd 2417006 2417005 2026-05-20T14:10:46Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2417006 wikitext text/x-wiki hs9uwoquqyhz7kci7g78kd04vzg804j 2417007 2417006 2026-05-20T14:11:37Z Stashbot 7414 bjensen: uploaded trixie-packaged memkeys on apt1002 2417007 wikitext text/x-wiki ghl1vtxitckhend84jpkfhd27ekjhdv 2417008 2417007 2026-05-20T14:11:57Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2417008 wikitext text/x-wiki 8x0hy6kg953jbpu1bidr8iul68bbng2 2417009 2417008 2026-05-20T14:12:04Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2417009 wikitext text/x-wiki 3aymf7cjijynrm9bj791pmge8kxqdil 2417010 2417009 2026-05-20T14:12:06Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2042.codfw.wmnet 2417010 wikitext text/x-wiki fu6t0ti7h44mz6mx86kfzutcvns6isx 2417014 2417010 2026-05-20T14:14:19Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet 2417014 wikitext text/x-wiki dacd23wk937f2r5196hlfjve1d90ute 2417015 2417014 2026-05-20T14:14:22Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet 2417015 wikitext text/x-wiki giix71wdldtj48fmwqqshfugl1t267u 2417016 2417015 2026-05-20T14:14:26Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet 2417016 wikitext text/x-wiki 737l2whio0tsuj6whgjw50se1o6aqcm 2417017 2417016 2026-05-20T14:15:18Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2417017 wikitext text/x-wiki of72q99cfwzuu0r76wi83wkqi5dobk4 2417020 2417017 2026-05-20T14:16:25Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2183.codfw.wmnet with OS trixie 2417020 wikitext text/x-wiki fxnzudjbhqsy6df0brsfwssc23p3vmk 2417022 2417020 2026-05-20T14:17:39Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudelastic1010.eqiad.wmnet 2417022 wikitext text/x-wiki 8x8asx2daenghg5pkkylytn26g1gufk 2417026 2417022 2026-05-20T14:19:19Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2042.codfw.wmnet 2417026 wikitext text/x-wiki 3r2o7uuod1d0nsg1rwlfv2r75dpzbc6 2417027 2417026 2026-05-20T14:19:32Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet 2417027 wikitext text/x-wiki lvzu0omrbvm6ll9fr2g6rexvve1jt6m 2417031 2417027 2026-05-20T14:21:02Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.266.0" for 2 host(s) 2417031 wikitext text/x-wiki bwt1fu35pmk677oiub0vw2ir7ki28ug 2417032 2417031 2026-05-20T14:21:30Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.reboot-nodes (exit_code=1) rolling reboot on P{wikikube-worker[1249-1289,1291-1327,1375-1384].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2417032 wikitext text/x-wiki pa36ksujl4pjreq3kik65qv3pe03ul5 2417034 2417032 2026-05-20T14:22:47Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2417034 wikitext text/x-wiki h600yase82as3uind5ptrtcrytp784p 2417035 2417034 2026-05-20T14:22:53Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.266.0" completed for 2 hosts 2417035 wikitext text/x-wiki kgnc2uxvqs53kf3rk78axl17tjzacy0 2417036 2417035 2026-05-20T14:23:45Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417036 wikitext text/x-wiki k38ejbcgxhs9fawiz4do7q8vpw31xac 2417038 2417036 2026-05-20T14:24:41Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2042.codfw.wmnet 2417038 wikitext text/x-wiki 1d14qp61odni8d876l1jpw5mstq4al3 2417039 2417038 2026-05-20T14:25:10Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2042.codfw.wmnet 2417039 wikitext text/x-wiki cezpeivi5opo4iwfmr3jsen7alh5iaw 2417040 2417039 2026-05-20T14:26:21Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on P{cephosd100[4-5].eqiad.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad) 2417040 wikitext text/x-wiki apjgp35ajqq2f545qxbu2zg29vn9j5y 2417041 2417040 2026-05-20T14:26:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs-test2001.codfw.wmnet with reason: host reimage 2417041 wikitext text/x-wiki 56egi2zeg9mp1jo6uva444uutrg7spt 2417043 2417041 2026-05-20T14:27:56Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1013.eqiad.wmnet with OS trixie 2417043 wikitext text/x-wiki khiy7qve7wfj67fx6oissupgry593m4 2417044 2417043 2026-05-20T14:28:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs-test1001.eqiad.wmnet with reason: host reimage 2417044 wikitext text/x-wiki cvj8klu5uemgk89vr77rgkmlha30yen 2417046 2417044 2026-05-20T14:29:34Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2043.codfw.wmnet 2417046 wikitext text/x-wiki 4oizpgtkvwdpiikhyepvviw8ndw0zza 2417047 2417046 2026-05-20T14:29:57Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs-test2001.codfw.wmnet with reason: host reimage 2417047 wikitext text/x-wiki qhqul1li9fwmsqlgp6agzx3fu9o94a2 2417049 2417047 2026-05-20T14:33:09Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2043.codfw.wmnet 2417049 wikitext text/x-wiki n6gvoh90u97zigx5t5ey0w620k8y6wu 2417050 2417049 2026-05-20T14:33:37Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs-test1001.eqiad.wmnet with reason: host reimage 2417050 wikitext text/x-wiki 6di3h0ee9zpg03cg5uq7t8cfm4u4xxo 2417051 2417050 2026-05-20T14:35:06Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417051 wikitext text/x-wiki 1ehouog3bahisztwm80ojvia6nw5275 2417052 2417051 2026-05-20T14:36:44Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet 2417052 wikitext text/x-wiki 4zdwm6m69l0r7t9h5h6z9zjoboqcwvy 2417053 2417052 2026-05-20T14:36:47Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet 2417053 wikitext text/x-wiki 0sn4ujjs914otx5k220qigww501r1ix 2417054 2417053 2026-05-20T14:36:52Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet 2417054 wikitext text/x-wiki 6rrl2thoxaq2pbjlvsovz76nkardisn 2417056 2417054 2026-05-20T14:38:28Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2417056 wikitext text/x-wiki jdt7xjm30eh520wod6dbotyw7roaar6 2417057 2417056 2026-05-20T14:38:31Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2043.codfw.wmnet 2417057 wikitext text/x-wiki n3tgd2q5vmz4lbw63xbclkcpyqpqwkl 2417058 2417057 2026-05-20T14:38:38Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2043.codfw.wmnet 2417058 wikitext text/x-wiki cgqg63s22twhtja8osvnttxuq75n0sy 2417059 2417058 2026-05-20T14:39:25Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2417059 wikitext text/x-wiki lv10om0ja2vgk4i8735hvvhwh7n6b6a 2417060 2417059 2026-05-20T14:39:35Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2044.codfw.wmnet 2417060 wikitext text/x-wiki i84ie9q7q6w4zfna5d49b8tkqaycwyt 2417061 2417060 2026-05-20T14:40:55Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1024.eqiad.wmnet 2417061 wikitext text/x-wiki qxjiehojcuiomxtdnj1vr7sl4crlo3r 2417062 2417061 2026-05-20T14:41:55Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet 2417062 wikitext text/x-wiki 37aq7ekaismnzj8kask2cgmhcdrcby7 2417063 2417062 2026-05-20T14:42:29Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool ulsfo [reason: router upgrade, T416562] 2417063 wikitext text/x-wiki 25a0h4na7cnzujhw5h4lg1zdxleiet3 2417064 2417063 2026-05-20T14:42:36Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool ulsfo [reason: router upgrade, T416562] 2417064 wikitext text/x-wiki 47zr1g8qja64cph4tq58gu6okp8xhth 2417065 2417064 2026-05-20T14:42:50Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1014.eqiad.wmnet with OS trixie 2417065 wikitext text/x-wiki t1m5swvnnsa21lxwgnhtiyopon1x33v 2417066 2417065 2026-05-20T14:42:54Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2044.codfw.wmnet 2417066 wikitext text/x-wiki c9ofik9bm1klavpvbt2ued7okihpha3 2417067 2417066 2026-05-20T14:43:01Z Stashbot 7414 moritzm: installing rsync security updates 2417067 wikitext text/x-wiki 3nrc0xi8b4uf3vs2m8i5htfbsx7hglc 2417068 2417067 2026-05-20T14:43:39Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417068 wikitext text/x-wiki ge0f04dzzzvjj1gab4qty2ykgssbrfp 2417069 2417068 2026-05-20T14:43:52Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2417069 wikitext text/x-wiki cph0en4m9ycd1n1ocluyszymu6qr7f2 2417070 2417069 2026-05-20T14:43:54Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt with reason: switch refresh 2417070 wikitext text/x-wiki fwfuap2sd10jkjurr9ze1p95gs8dkct 2417071 2417070 2026-05-20T14:44:13Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1257: Upgrading db1257.eqiad.wmnet 2417071 wikitext text/x-wiki 9nn29aca9ibgnsasr8hiqsnfz7d89db 2417072 2417071 2026-05-20T14:44:42Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1257: Upgrading db1257.eqiad.wmnet 2417072 wikitext text/x-wiki fmhsf7o9w4j9kyd2941schuny6p096i 2417073 2417072 2026-05-20T14:45:21Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-master1004.eqiad.wmnet 2417073 wikitext text/x-wiki q282q3fbf1z4vclf0o39ebrl9i4q6py 2417077 2417073 2026-05-20T14:47:31Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet 2417077 wikitext text/x-wiki n0z6lyeb99i28dz6113feu1exye6wn7 2417078 2417077 2026-05-20T14:47:43Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs-test2001.codfw.wmnet with OS bookworm 2417078 wikitext text/x-wiki n3r31f8zcj2xtfp1cte8i0etm9euzt0 2417079 2417078 2026-05-20T14:48:10Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417079 wikitext text/x-wiki k1ur4x8aei2d6mjpe8dczlz834tyc81 2417080 2417079 2026-05-20T14:48:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2044.codfw.wmnet 2417080 wikitext text/x-wiki ix827xl3h1859v06u2jy4mkm74olcoe 2417081 2417080 2026-05-20T14:48:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2044.codfw.wmnet 2417081 wikitext text/x-wiki mdop9w4gq0klybhx4ruertrbgzw1dtj 2417082 2417081 2026-05-20T14:49:49Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1257.eqiad.wmnet with OS trixie 2417082 wikitext text/x-wiki gwrto1xtt5tg88gg8fcjke4f0ao5cpa 2417083 2417082 2026-05-20T14:50:06Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417083 wikitext text/x-wiki jf23tg5iwejjc7u6cuuzntv7jrv5xlb 2417085 2417083 2026-05-20T14:51:16Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417085 wikitext text/x-wiki l69ljuesng1poheyv3yg1d85rutcd1x 2417086 2417085 2026-05-20T14:51:58Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1004.eqiad.wmnet 2417086 wikitext text/x-wiki 3lalc36piksn6dvwb5uyyt5ahxt9s9b 2417087 2417086 2026-05-20T14:52:21Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417087 wikitext text/x-wiki gt4cimxkmisdvdgsv6gogs7bebmgzlz 2417088 2417087 2026-05-20T14:52:56Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2417088 wikitext text/x-wiki orxwz7d5qj79sn7r8qy15hu2s6hmrwl 2417089 2417088 2026-05-20T14:53:29Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417089 wikitext text/x-wiki 7e4q346cpfjaj9lbdxyxv18ijyqn98l 2417090 2417089 2026-05-20T14:54:38Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet 2417090 wikitext text/x-wiki iveph7p4zwmcrdptrm2xpgrck9ojglg 2417091 2417090 2026-05-20T14:54:40Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet 2417091 wikitext text/x-wiki 0ppw7g3hx4kcjizj40pf8ck98qkzq3q 2417092 2417091 2026-05-20T14:54:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet 2417092 wikitext text/x-wiki s1b9vzvhyl70qkfpwihpzgaeurfmxf3 2417093 2417092 2026-05-20T14:54:48Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2417093 wikitext text/x-wiki ba0qbrufjg8ekg3mbwhl63wd6d4c4zy 2417094 2417093 2026-05-20T14:54:55Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2417094 wikitext text/x-wiki me9fy2e3gxmpd1ea6ja08ajeo8643lq 2417095 2417094 2026-05-20T14:56:14Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417095 wikitext text/x-wiki dqh0e1ww5sdcrin3ww6m96nbmgv814y 2417096 2417095 2026-05-20T14:56:48Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1025.eqiad.wmnet 2417096 wikitext text/x-wiki mr5qr1gfumwqw07rnofuz84n99fj1if 2417097 2417096 2026-05-20T14:57:07Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1014.eqiad.wmnet with reason: host reimage 2417097 wikitext text/x-wiki 11aib2gcmr0xytymhcn7vhmb01c78p9 2417098 2417097 2026-05-20T14:57:14Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 14 hosts with reason: restart 2417098 wikitext text/x-wiki raosub1ykb0019pif44mb6ozzyuzip0 2417101 2417098 2026-05-20T15:00:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet 2417101 wikitext text/x-wiki gpq5rkhbxylwbkrxvgnxlmoua753hvq 2417102 2417101 2026-05-20T15:00:10Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2033.codfw.wmnet 2417102 wikitext text/x-wiki 7d76pyuep61biq9b1bokupdfzs6d357 2417103 2417102 2026-05-20T15:00:21Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{wikikube-worker[1294-1327].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2417103 wikitext text/x-wiki quc2qgnaupuppixvok4l20kw7kviq1i 2417104 2417103 2026-05-20T15:00:30Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1294-1297].eqiad.wmnet 2417104 wikitext text/x-wiki 8nzkqvo5gw38pdw7wb23lsdyzh7ibyn 2417105 2417104 2026-05-20T15:00:58Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet 2417105 wikitext text/x-wiki 0mtobofkowrnw6fi9itprsdp91uw6ub 2417106 2417105 2026-05-20T15:02:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1257.eqiad.wmnet with reason: host reimage 2417106 wikitext text/x-wiki 33n60a8goi6k0h03lrujndqdbdhvpo4 2417107 2417106 2026-05-20T15:02:43Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1294-1297].eqiad.wmnet 2417107 wikitext text/x-wiki j38i0ha7l91m27q78glwketmhw49okt 2417108 2417107 2026-05-20T15:04:45Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1014.eqiad.wmnet with reason: host reimage 2417108 wikitext text/x-wiki enp3s4bf5u4lbq52vctcrz07hxusgne 2417109 2417108 2026-05-20T15:04:53Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid analytics cluster: Reboot Druid nodes 2417109 wikitext text/x-wiki bkgterj5mvujjb14j8hen3twruoeakc 2417110 2417109 2026-05-20T15:06:57Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2033.codfw.wmnet 2417110 wikitext text/x-wiki 9e3cefbij4od39wtbcgiybebmze5tpt 2417111 2417110 2026-05-20T15:07:42Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2417111 wikitext text/x-wiki rqveqiifna5xyb3rgamv1plc96scv15 2417112 2417111 2026-05-20T15:08:08Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet 2417112 wikitext text/x-wiki 5zezcmado2zqjxvvtgal0cjqmkn71rr 2417113 2417112 2026-05-20T15:08:10Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet 2417113 wikitext text/x-wiki o0ftd0lyxgj2gato5rvdx18sx85vpyw 2417114 2417113 2026-05-20T15:08:15Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet 2417114 wikitext text/x-wiki 8ka4gluz41naehi10t26l9jzxnhjid2 2417115 2417114 2026-05-20T15:08:48Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1257.eqiad.wmnet with reason: host reimage 2417115 wikitext text/x-wiki g30y0pi0c6m6r6ffzrjacaze00xtq3h 2417116 2417115 2026-05-20T15:10:00Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=codfw 2417116 wikitext text/x-wiki syv8dzmtw48jfuc24kftqk730r3e5s8 2417117 2417116 2026-05-20T15:13:17Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1294-1297].eqiad.wmnet 2417117 wikitext text/x-wiki pdts8et3ocvsfdkg9h040bsdn779a7e 2417118 2417117 2026-05-20T15:13:19Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1294-1297].eqiad.wmnet 2417118 wikitext text/x-wiki iw6xjma1thi1i1ml972xto3cxfce1mj 2417119 2417118 2026-05-20T15:13:30Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1298-1301].eqiad.wmnet 2417119 wikitext text/x-wiki 6cd158ug2z711knfusfbf4ezslsdrt5 2417122 2417119 2026-05-20T15:14:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2033.codfw.wmnet 2417122 wikitext text/x-wiki 9sznkauzvvbrgq5pezmrfuazafwkjxl 2417123 2417122 2026-05-20T15:15:09Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet 2417123 wikitext text/x-wiki icepn7m7jpy6xwveoq4kxexk0xlegqr 2417124 2417123 2026-05-20T15:15:42Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1298-1301].eqiad.wmnet 2417124 wikitext text/x-wiki aikkb0rvvprnom9ik46fy6yny50fv7p 2417125 2417124 2026-05-20T15:17:19Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2417125 wikitext text/x-wiki o6ji77gtryjql2k6cjlyy4rl5bo6wcg 2417128 2417125 2026-05-20T15:20:47Z Stashbot 7414 hashar: Restarted Jenkins CI due to Java upgrade which causes integration/pipelinelib to not be loadable. 2417128 wikitext text/x-wiki gwuhwwglpsn1y2diyc4dxlzx1z9nfe7 2417129 2417128 2026-05-20T15:23:08Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1014.eqiad.wmnet with OS trixie 2417129 wikitext text/x-wiki o14eyvo2140ffkcbalhm6rpcsd3lj63 2417130 2417129 2026-05-20T15:24:00Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2417130 wikitext text/x-wiki 6fqy703cpq4bu0uceczj2xj00giacpd 2417131 2417130 2026-05-20T15:25:02Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1257.eqiad.wmnet with OS trixie 2417131 wikitext text/x-wiki 74zxmoyte7rk74dl54co0qfavw5dzh5 2417132 2417131 2026-05-20T15:25:29Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.wdqs.reboot 2417132 wikitext text/x-wiki 77zbt5iuaha8581jcfcr8cuohlxorsa 2417133 2417132 2026-05-20T15:25:58Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2417133 wikitext text/x-wiki bql2hdr2sd8dkx08zb33ofykdpmd9e6 2417134 2417133 2026-05-20T15:26:17Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1298-1301].eqiad.wmnet 2417134 wikitext text/x-wiki mydisr76youfztu9mp1hruj6z4wfvb2 2417135 2417134 2026-05-20T15:26:19Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1298-1301].eqiad.wmnet 2417135 wikitext text/x-wiki lb98zy2ifxpyedx87r9ga15wzyhl5zb 2417136 2417135 2026-05-20T15:26:29Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1302-1305].eqiad.wmnet 2417136 wikitext text/x-wiki d70cx5z2r5y0hn6ykiws47zxk60vvmm 2417137 2417136 2026-05-20T15:27:10Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1257: Migration of db1257.eqiad.wmnet completed 2417137 wikitext text/x-wiki 6jwjkik1hiba5iym8fmvu5enq7sp5ux 2417138 2417137 2026-05-20T15:27:51Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) 2417138 wikitext text/x-wiki 39ygtfqgkgpnk70oxtz4tk9in486kvl 2417141 2417138 2026-05-20T15:29:16Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1302-1305].eqiad.wmnet 2417141 wikitext text/x-wiki aubh7wxjw4h5u6thtd281dvjmxfu1rw 2417142 2417141 2026-05-20T15:29:32Z Stashbot 7414 moritzm: failover Ganeti master in codfw02 to ganeti2033 2417142 wikitext text/x-wiki nef6f6tgt053vnxp4hl3z2vf8v4pojf 2417144 2417142 2026-05-20T15:30:09Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1015.eqiad.wmnet with OS trixie 2417144 wikitext text/x-wiki 5oaw0kskfw1ifu4ok70z6k4cavv5y5n 2417145 2417144 2026-05-20T15:30:39Z Stashbot 7414 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply 2417145 wikitext text/x-wiki ragvrc6twrfyh1zybutzlwipq326jpt 2417146 2417145 2026-05-20T15:32:05Z Stashbot 7414 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply 2417146 wikitext text/x-wiki outap29081s683y9fl0cdq7p4m0wx03 2417148 2417146 2026-05-20T15:32:46Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.decommission for hosts cp[2041-2042].codfw.wmnet 2417148 wikitext text/x-wiki gec7swt8kkk1lqd9q9uh6rxb0xlppfq 2417151 2417148 2026-05-20T15:35:06Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2016.codfw.wmnet 2417151 wikitext text/x-wiki afoel12hwaay0gtou2uhekez595z4dg 2417152 2417151 2026-05-20T15:36:32Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1302-1305].eqiad.wmnet 2417152 wikitext text/x-wiki 4lglic828svauac864thic0i0inpe9k 2417153 2417152 2026-05-20T15:36:34Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1302-1305].eqiad.wmnet 2417153 wikitext text/x-wiki owjcuc0urin4zv7usgdqkhvenxx4vtb 2417155 2417153 2026-05-20T15:36:45Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1306-1309].eqiad.wmnet 2417155 wikitext text/x-wiki e4orw8x4nazdkx692adbsqggxd7yii7 2417156 2417155 2026-05-20T15:37:14Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2023.codfw.wmnet 2417156 wikitext text/x-wiki ch3iue10nw6s9l6yrkir5s4wd5a5nlv 2417157 2417156 2026-05-20T15:37:20Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2024.codfw.wmnet 2417157 wikitext text/x-wiki 5kn9hkizvgpx0bilv47uut0s9652kqs 2417158 2417157 2026-05-20T15:38:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet 2417158 wikitext text/x-wiki gkqd36cieg3sg0nrq7qam899j34zhgm 2417159 2417158 2026-05-20T15:38:32Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.dns.netbox 2417159 wikitext text/x-wiki suh6eagdvagbr7yb6yelvxo8o8wt0qx 2417161 2417159 2026-05-20T15:39:36Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1306-1309].eqiad.wmnet 2417161 wikitext text/x-wiki bs2rg1du8oe4rwsmotz8y0v6bc94rwb 2417163 2417161 2026-05-20T15:41:10Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2016.codfw.wmnet 2417163 wikitext text/x-wiki dpu2jucm0py2pt3517feubp8eky5y7k 2417165 2417163 2026-05-20T15:41:49Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.dns.roll-reboot rolling reboot on P{dns6002.wikimedia.org} and (A:dnsbox) 2417165 wikitext text/x-wiki 6w4agab3z0dsckpst7ewhd1bgryiq67 2417166 2417165 2026-05-20T15:41:52Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns6002.wikimedia.org 2417166 wikitext text/x-wiki b6766llwb1d1xcn09gfin8hzz7hu6k1 2417168 2417166 2026-05-20T15:43:19Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema 2417168 wikitext text/x-wiki 7cceb3iwg7h61l0zcpzlyff0v82w8pd 2417169 2417168 2026-05-20T15:44:12Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2023.codfw.wmnet 2417169 wikitext text/x-wiki cmyc85q7h43jep06093qfyavs5s2lms 2417171 2417169 2026-05-20T15:44:24Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2024.codfw.wmnet 2417171 wikitext text/x-wiki 9cnw249vsghqtdmdor1e6dn48zqnmzy 2417172 2417171 2026-05-20T15:44:28Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1015.eqiad.wmnet with reason: host reimage 2417172 wikitext text/x-wiki 6m96tlebi5hsc40jkj0d3mh1eqs8a9t 2417173 2417172 2026-05-20T15:44:30Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=codfw 2417173 wikitext text/x-wiki tfjjcydv1tk7sp01of0lmigsamsglpd 2417174 2417173 2026-05-20T15:44:54Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=eqiad 2417174 wikitext text/x-wiki celvrdk6p994n2melv9aucrb3nl38ga 2417175 2417174 2026-05-20T15:44:58Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host matomo1003.eqiad.wmnet 2417175 wikitext text/x-wiki 9onpg45imwrq5010sml3ovvbz61bhbg 2417177 2417175 2026-05-20T15:45:11Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1024.eqiad.wmnet 2417177 wikitext text/x-wiki cychyvlczita32xc6ee01vj55ud93uq 2417178 2417177 2026-05-20T15:45:20Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2417178 wikitext text/x-wiki g3puu4u7spl9tgqk5aizv3ut0u3doia 2417179 2417178 2026-05-20T15:45:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet 2417179 wikitext text/x-wiki mwdqd3t4yaofy5x2j8cfgf0s533w0w0 2417180 2417179 2026-05-20T15:45:25Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet 2417180 wikitext text/x-wiki 372xlwogxa3pcdkqj9vgxblclcb3wch 2417181 2417180 2026-05-20T15:45:29Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1023.eqiad.wmnet 2417181 wikitext text/x-wiki mnzok7m0a4cxstaa5ji5yd5o1qxyo10 2417182 2417181 2026-05-20T15:45:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet 2417182 wikitext text/x-wiki kopav5pahcw7yl0ktj1931iqtvjojf4 2417183 2417182 2026-05-20T15:45:36Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp[2041-2042].codfw.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" 2417183 wikitext text/x-wiki kky3cvpk5sfryaandih11k8pjpeck2p 2417184 2417183 2026-05-20T15:46:07Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet 2417184 wikitext text/x-wiki 7kkwyojuw3qi7zt2t26cloy7ditcig2 2417185 2417184 2026-05-20T15:46:32Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp[2041-2042].codfw.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" 2417185 wikitext text/x-wiki re13vflp8y8556yujro8u7ckznxel4l 2417186 2417185 2026-05-20T15:46:34Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2417186 wikitext text/x-wiki nten92i4biu4p230q6c5uhkhms41whg 2417187 2417186 2026-05-20T15:46:36Z Stashbot 7414 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cp[2041-2042].codfw.wmnet 2417187 wikitext text/x-wiki sj3ir02w1j6dws5qfwtbjjrpn3pimf9 2417189 2417187 2026-05-20T15:48:55Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1306-1309].eqiad.wmnet 2417189 wikitext text/x-wiki nz4rk69u38db1h4rmybr0qh7qw4fphy 2417190 2417189 2026-05-20T15:48:57Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1306-1309].eqiad.wmnet 2417190 wikitext text/x-wiki o5vsxh11tos6ytvceea7824t2r6h3z3 2417191 2417190 2026-05-20T15:48:59Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1003.eqiad.wmnet 2417191 wikitext text/x-wiki t4neevdgs4d9y3daxdi07a2ys38fj42 2417192 2417191 2026-05-20T15:49:06Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1310-1313].eqiad.wmnet 2417192 wikitext text/x-wiki lt15hprkg02vku37soug56815tanpvh 2417193 2417192 2026-05-20T15:49:23Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1015.eqiad.wmnet with reason: host reimage 2417193 wikitext text/x-wiki tp6ob369nggl3jtwn097ac23zzpj71u 2417195 2417193 2026-05-20T15:50:20Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2417195 wikitext text/x-wiki 06be5wzm3k923wfkeis2l21llfmp8qv 2417196 2417195 2026-05-20T15:50:35Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns6002.wikimedia.org 2417196 wikitext text/x-wiki bmxqse6e15722c2uwzxh9yxpr6odab5 2417197 2417196 2026-05-20T15:50:37Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P{dns6002.wikimedia.org} and (A:dnsbox) 2417197 wikitext text/x-wiki n3umpow3o8u6nw8gas4vs67bvfuqka8 2417198 2417197 2026-05-20T15:51:18Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1310-1313].eqiad.wmnet 2417198 wikitext text/x-wiki ssj3k7mio87j3sppebv7yw3i9zzrcnj 2417199 2417198 2026-05-20T15:51:38Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox and not P{dns6002.wikimedia.org} and not A:magru and (A:dnsbox) 2417199 wikitext text/x-wiki 6nhmi0htidg4o3tybnii8phtx3bjjy7 2417200 2417199 2026-05-20T15:51:39Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns1004.wikimedia.org 2417200 wikitext text/x-wiki 9i9le7r0p1qxjgrq73wha0yq8voyxn1 2417201 2417200 2026-05-20T15:52:03Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1024.eqiad.wmnet 2417201 wikitext text/x-wiki 1ju8bstsau2iiu45fmxfg9obj7mrzlu 2417202 2417201 2026-05-20T15:52:20Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1023.eqiad.wmnet 2417202 wikitext text/x-wiki 71b9p372yozbc31xry56x1ridj7javv 2417203 2417202 2026-05-20T15:52:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet 2417203 wikitext text/x-wiki qbi9z49lp0br75do1451h4g61lf2tna 2417204 2417203 2026-05-20T15:52:47Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet 2417204 wikitext text/x-wiki 81l32wyyxendvga4so76lzkjzm466kc 2417205 2417204 2026-05-20T15:52:52Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet 2417205 wikitext text/x-wiki p2yuweuta2gqt7k0ijthex21j6yo3md 2417209 2417205 2026-05-20T15:54:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2417209 wikitext text/x-wiki mjizfdmjsx0xxzdgh5zpkv21meym6hr 2417210 2417209 2026-05-20T15:54:39Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2417210 wikitext text/x-wiki 79manehoda9k6wuyw6scyxduyhkp9sc 2417211 2417210 2026-05-20T15:56:43Z Stashbot 7414 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2417211 wikitext text/x-wiki 0cm859f5sqcg6srtli5zh8uadagdwou 2417212 2417211 2026-05-20T15:57:48Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=eqiad 2417212 wikitext text/x-wiki iwxp9io81vxfaomtobvbrov87j77x63 2417213 2417212 2026-05-20T15:59:52Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema 2417213 wikitext text/x-wiki mjufljx2dakw4ozo0ybbtqgzqw6tv9r 2417214 2417213 2026-05-20T15:59:54Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1310-1313].eqiad.wmnet 2417214 wikitext text/x-wiki p2bszmi84rslkrwnp9lwf695wukgfx2 2417215 2417214 2026-05-20T15:59:56Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1310-1313].eqiad.wmnet 2417215 wikitext text/x-wiki 2zzqepd25fb3gygyuge2m3w1fqdxgp2 2417216 2417215 2026-05-20T16:00:05Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1314-1317].eqiad.wmnet 2417216 wikitext text/x-wiki auze2hh8yuydff26zd1d4zjl0ixaz8s 2417217 2417216 2026-05-20T16:00:53Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1289980|Fix newFromUserIdentity calls with interwiki users (T426832)]] 2417217 wikitext text/x-wiki 6kbkc3w41itr1pm5lgrepimmlkbd4sa 2417218 2417217 2026-05-20T16:02:17Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1314-1317].eqiad.wmnet 2417218 wikitext text/x-wiki s0kcxzw63wi8q3ilawqf1knyogc0f94 2417219 2417218 2026-05-20T16:02:57Z Stashbot 7414 urbanecm@deploy1003: urbanecm, mszwarc: Backport for [[gerrit:1289980|Fix newFromUserIdentity calls with interwiki users (T426832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417219 wikitext text/x-wiki s3btcjwmt5ky0iv5oguh0ulndd4chu8 2417223 2417219 2026-05-20T16:05:14Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:cassandra-dev 2417223 wikitext text/x-wiki 1qo1i9zis8qj17i8vsq4joppvzwvbrp 2417224 2417223 2026-05-20T16:05:45Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns1004.wikimedia.org 2417224 wikitext text/x-wiki nyyhmswfv0pocmbczyzsa7elzf7ncq5 2417225 2417224 2026-05-20T16:05:52Z Stashbot 7414 urbanecm@deploy1003: urbanecm, mszwarc: Continuing with deployment 2417225 wikitext text/x-wiki 6targ34e2xqkq507mgwcdukii55bks2 2417228 2417225 2026-05-20T16:07:28Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt 2417228 wikitext text/x-wiki oeqifh9hobj3lo67wzjxzzj1xguboet 2417229 2417228 2026-05-20T16:07:30Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt 2417229 wikitext text/x-wiki bexkbjmzwr135r3d4osipo8y84nr06o 2417230 2417229 2026-05-20T16:07:53Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1015.eqiad.wmnet with OS trixie 2417230 wikitext text/x-wiki 5f120pv73alg4xrpjdbst0zo5lssda4 2417231 2417230 2026-05-20T16:08:04Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet 2417231 wikitext text/x-wiki 45rnatvwnviuvoqce8zjbwp3uq66f50 2417234 2417231 2026-05-20T16:09:53Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1314-1317].eqiad.wmnet 2417234 wikitext text/x-wiki 4rda2a4r4rukqymyksyltrux87k0or9 2417235 2417234 2026-05-20T16:09:55Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1314-1317].eqiad.wmnet 2417235 wikitext text/x-wiki bas8q6pslqhudwa3v0qhgbnz4e7gyi5 2417236 2417235 2026-05-20T16:10:04Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289980|Fix newFromUserIdentity calls with interwiki users (T426832)]] (duration: 09m 12s) 2417236 wikitext text/x-wiki 6refombrhk4bu5bjg4q8bckdaeq5uuz 2417237 2417236 2026-05-20T16:10:07Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1318-1321].eqiad.wmnet 2417237 wikitext text/x-wiki ird46m04ywmgaf4bdx1plbr44h1yrto 2417238 2417237 2026-05-20T16:10:36Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: switch refresh 2417238 wikitext text/x-wiki gmayh10bhkjcivyzttw7rsl4sqi3yhz 2417240 2417238 2026-05-20T16:12:17Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1318-1321].eqiad.wmnet 2417240 wikitext text/x-wiki 9oqieiduxukqpjp03rq0dlkf7v935s9 2417241 2417240 2026-05-20T16:12:39Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1257: Migration of db1257.eqiad.wmnet completed 2417241 wikitext text/x-wiki b4p6cbuqwa14rc3meugw2jxadkx64ti 2417242 2417241 2026-05-20T16:12:41Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2417242 wikitext text/x-wiki dhq67d9kdkrlcipfoea4kdhrrw5g54d 2417244 2417242 2026-05-20T16:14:26Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet 2417244 wikitext text/x-wiki dn0s7qff0cpb8m4ujlg5ive766e6j2q 2417245 2417244 2026-05-20T16:19:55Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1318-1321].eqiad.wmnet 2417245 wikitext text/x-wiki qlzyv66a4iw1jqd7e6z9up7jgojjlfs 2417246 2417245 2026-05-20T16:19:57Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1318-1321].eqiad.wmnet 2417246 wikitext text/x-wiki 6fmqquy7ofyz55l8ts9bpf947o862z1 2417247 2417246 2026-05-20T16:20:05Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1322-1324].eqiad.wmnet 2417247 wikitext text/x-wiki 5v2s2qprnntc78bb5yxb50cw4u4aaor 2417248 2417247 2026-05-20T16:20:45Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns1005.wikimedia.org 2417248 wikitext text/x-wiki jlz03qg4p5s3ccyq4um3r3det9b1xgv 2417249 2417248 2026-05-20T16:22:21Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1322-1324].eqiad.wmnet 2417249 wikitext text/x-wiki 6jg2rzu2md7sa1yep2ybhgintbjanjc 2417250 2417249 2026-05-20T16:22:54Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet 2417250 wikitext text/x-wiki tla1vab67jig019c43s11e7gqtsnonj 2417252 2417250 2026-05-20T16:29:32Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet 2417252 wikitext text/x-wiki kpzgn6vy09sw3p3utbkk7eyxlgk06p1 2417253 2417252 2026-05-20T16:29:34Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet 2417253 wikitext text/x-wiki mjcet915kb12y9i7dshl0a18eiwbf7r 2417254 2417253 2026-05-20T16:29:38Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad 2417254 wikitext text/x-wiki ne6r3yc0py3rk7ykujsylt8qs8tcye1 2417255 2417254 2026-05-20T16:29:40Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1322-1324].eqiad.wmnet 2417255 wikitext text/x-wiki lo7lcz0rhzyd1x16wpncezxnyijqb2r 2417256 2417255 2026-05-20T16:29:42Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1322-1324].eqiad.wmnet 2417256 wikitext text/x-wiki mskqkiyua048thnw64h64onn0mdxnid 2417257 2417256 2026-05-20T16:29:47Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1325-1327].eqiad.wmnet 2417257 wikitext text/x-wiki cgjel3tktxeuaqd5uz49zg4s6znfdxh 2417258 2417257 2026-05-20T16:31:30Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1325-1327].eqiad.wmnet 2417258 wikitext text/x-wiki g4nxvpot24mfzl6rhri5zm2j6oa5f60 2417260 2417258 2026-05-20T16:33:22Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:cassandra-dev 2417260 wikitext text/x-wiki 33oxvnx2cag6jnuh6f856fdcd36awun 2417261 2417260 2026-05-20T16:34:57Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns1005.wikimedia.org 2417261 wikitext text/x-wiki ttlj14y60olo01p6m4104hy42szayle 2417262 2417261 2026-05-20T16:36:13Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_ulsfo and not P{cp4037.ulsfo.wmnet} and not P{cp4038.ulsfo.wmnet} and A:cp 2417262 wikitext text/x-wiki o61ldtnvmshu2f4q8ctkitgizmfe6bt 2417263 2417262 2026-05-20T16:37:19Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_ulsfo and A:cp 2417263 wikitext text/x-wiki cu1n5wk0p5dz4ou5thtiozmtpi8luwf 2417264 2417263 2026-05-20T16:37:31Z Stashbot 7414 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply 2417264 wikitext text/x-wiki rextlmv2pilarxniwcz0w9usmca69lo 2417265 2417264 2026-05-20T16:38:08Z Stashbot 7414 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply 2417265 wikitext text/x-wiki 36mxwg067b756uf6gyye6mtnz5ndjfv 2417266 2417265 2026-05-20T16:39:01Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1325-1327].eqiad.wmnet 2417266 wikitext text/x-wiki ogw3q3pxm0qlm08p5vosdcq1hf6xpy1 2417267 2417266 2026-05-20T16:39:03Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1325-1327].eqiad.wmnet 2417267 wikitext text/x-wiki 7oauqa9aaem3w0rs3e48qyej71qoxk2 2417268 2417267 2026-05-20T16:39:05Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{wikikube-worker[1294-1327].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2417268 wikitext text/x-wiki rfe6118ffy7xlvuml0yp1ro1txipaqh 2417269 2417268 2026-05-20T16:43:55Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad 2417269 wikitext text/x-wiki 3a2c2gzc4685dv69uihv9nuorq4jw1l 2417270 2417269 2026-05-20T16:47:35Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4045.ulsfo.wmnet 2417270 wikitext text/x-wiki scpbx4ddmypkozdvr52det8mlfkrt15 2417271 2417270 2026-05-20T16:47:46Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4039.ulsfo.wmnet 2417271 wikitext text/x-wiki 5n7bnd4c58q39sb6y7kx0l09jgnziix 2417272 2417271 2026-05-20T16:49:57Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns1006.wikimedia.org 2417272 wikitext text/x-wiki 45aez661didntm40acy3whtjr5xwkqn 2417273 2417272 2026-05-20T16:50:36Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7002.magru.wmnet} and A:liberica 2417273 wikitext text/x-wiki ddiyehe3q3i9ypjvc7f7izwmftz8x96 2417274 2417273 2026-05-20T16:53:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw 2417274 wikitext text/x-wiki i8dtyperxv21p5kkzdzwwgyar4fo7bt 2417275 2417274 2026-05-20T16:53:28Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet 2417275 wikitext text/x-wiki 5fuxzjh0qhdvtw4w7g4zx58qvwx5bul 2417276 2417275 2026-05-20T16:53:53Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2007.codfw.wmnet 2417276 wikitext text/x-wiki 093de84jolmpxrvoqxb18z6iq1t75av 2417277 2417276 2026-05-20T16:54:20Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7002.magru.wmnet} and A:liberica 2417277 wikitext text/x-wiki h0vknnd5cl4olkgybgi08v3c74zv241 2417278 2417277 2026-05-20T16:58:14Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7001.magru.wmnet} and A:liberica 2417278 wikitext text/x-wiki siayeqyn4dpzc0pppm49guq859brp09 2417279 2417278 2026-05-20T16:58:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet 2417279 wikitext text/x-wiki id8x3zve9bx6g62m3h9jr3udl5bi997 2417280 2417279 2026-05-20T16:59:15Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2417280 wikitext text/x-wiki 8aij209gyipa9nm5yo50oniutocq4t9 2417281 2417280 2026-05-20T17:00:53Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2007.codfw.wmnet 2417281 wikitext text/x-wiki fhjb2ots412jo4itwkk51tniwmhy6e9 2417282 2417281 2026-05-20T17:00:57Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2008.codfw.wmnet 2417282 wikitext text/x-wiki 2jupg943vo76s3cidsbpdtle5cmwsbw 2417283 2417282 2026-05-20T17:02:04Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7001.magru.wmnet} and A:liberica 2417283 wikitext text/x-wiki 0wid4t5bjow0hk0vv4gs63wewdzmt5y 2417284 2417283 2026-05-20T17:04:03Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.admin rebooting P{lvs4010.ulsfo.wmnet} and A:liberica 2417284 wikitext text/x-wiki kzac78l5hp3gtkjf7q4ztwvb0gebbvf 2417285 2417284 2026-05-20T17:04:09Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns1006.wikimedia.org 2417285 wikitext text/x-wiki 3x52wlqfinn3aaj19shqyqgqigfc0z3 2417286 2417285 2026-05-20T17:05:26Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet 2417286 wikitext text/x-wiki qn2gbhi5oshzwjy8vyqw28ne9v55jo1 2417287 2417286 2026-05-20T17:05:28Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet 2417287 wikitext text/x-wiki 9n7azbzm0hsu8thtuy3kur2vagjgk91 2417288 2417287 2026-05-20T17:05:34Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet 2417288 wikitext text/x-wiki kns201ugnbzcjs1542fet3bziat5aye 2417289 2417288 2026-05-20T17:06:50Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2008.codfw.wmnet 2417289 wikitext text/x-wiki pmgjijyqt4qfbc144816q1apnyn7rsl 2417290 2417289 2026-05-20T17:06:54Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2010.codfw.wmnet 2417290 wikitext text/x-wiki 9f34tti9kt3dyswhlv86ju4t1lif74z 2417291 2417290 2026-05-20T17:07:19Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs4010.ulsfo.wmnet} and A:liberica 2417291 wikitext text/x-wiki r0h4vyadkomg47f49jgqxmak4959ced 2417292 2417291 2026-05-20T17:08:24Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426560 - bking@cumin2002 2417292 wikitext text/x-wiki 6o5o24ennwjyt3lsw0nlr6m5lc1r41x 2417293 2417292 2026-05-20T17:10:27Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt 2417293 wikitext text/x-wiki 5acmeb1tngsjaf7r0t0na83cc17sycr 2417294 2417293 2026-05-20T17:10:29Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt 2417294 wikitext text/x-wiki jpc4w4vwh8g9z2ug034wks3h7a7rgbz 2417295 2417294 2026-05-20T17:10:44Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet 2417295 wikitext text/x-wiki 1clghutqq91bbrevdz1ci8omzp2n55p 2417296 2417295 2026-05-20T17:13:25Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2010.codfw.wmnet 2417296 wikitext text/x-wiki c11l9pbc7cpw03dbwazifvfrfupju9b 2417297 2417296 2026-05-20T17:13:29Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2011.codfw.wmnet 2417297 wikitext text/x-wiki oh6v7eruf7vo1bbsx6w4frxwz81ytu9 2417298 2417297 2026-05-20T17:14:56Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mr1-ulsfo,mr1-ulsfo IPv6,mr1-ulsfo.oob,mr1-ulsfo.oob IPv6 with reason: switch refresh 2417298 wikitext text/x-wiki tmx7ibzgjbrud5t2orwce1v7s6qnrb5 2417299 2417298 2026-05-20T17:17:23Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Rebuild to pick up new production image 2417299 wikitext text/x-wiki lu83v5fpebe4lv1qq2vdn3fmti7y75l 2417300 2417299 2026-05-20T17:17:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet 2417300 wikitext text/x-wiki df6x1iap4jfrjsilsatao6m2o1l6ecn 2417301 2417300 2026-05-20T17:17:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet 2417301 wikitext text/x-wiki mxmqtv1n27dzt2ybbb3t10vqunrpgcw 2417302 2417301 2026-05-20T17:17:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet 2417302 wikitext text/x-wiki oprypj94yx2zqommxkckjtu0cc65rqc 2417303 2417302 2026-05-20T17:19:09Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns2004.wikimedia.org 2417303 wikitext text/x-wiki of8017lb7qv58t2ruivarfoyktq0z97 2417304 2417303 2026-05-20T17:19:51Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2011.codfw.wmnet 2417304 wikitext text/x-wiki pxgbkboyvcxdoln0q8tk7ibbxcv0k0v 2417305 2417304 2026-05-20T17:19:55Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2012.codfw.wmnet 2417305 wikitext text/x-wiki aln6caq4bal8sapvz8mj217l2ymvx37 2417306 2417305 2026-05-20T17:20:09Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad 2417306 wikitext text/x-wiki 2vro7zaef09s3xbgiklrc7l8ggzylq8 2417307 2417306 2026-05-20T17:22:55Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet 2417307 wikitext text/x-wiki izq6r71toktkvnhpga3scsqtcisumql 2417308 2417307 2026-05-20T17:26:02Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2012.codfw.wmnet 2417308 wikitext text/x-wiki aatkxcm91ilxua6h2gm8xtav7q0p8iq 2417309 2417308 2026-05-20T17:26:06Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2013.codfw.wmnet 2417309 wikitext text/x-wiki sffohalvporuxl4iqwqf9uf7j2bu3b0 2417310 2417309 2026-05-20T17:27:26Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4040.ulsfo.wmnet 2417310 wikitext text/x-wiki fk05wpb99gpg7z2jw3qv1ty73ibalky 2417311 2417310 2026-05-20T17:27:32Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4046.ulsfo.wmnet 2417311 wikitext text/x-wiki pz83w3en71t13fx0ojpdemxaojgpozd 2417312 2417311 2026-05-20T17:28:00Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns2004.wikimedia.org 2417312 wikitext text/x-wiki qnme8zbqliko719ymfbzwl1j8j11z3i 2417313 2417312 2026-05-20T17:28:25Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet 2417313 wikitext text/x-wiki 2hragcw7zjpiryv7mbied0b3oc8eif7 2417314 2417313 2026-05-20T17:28:27Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet 2417314 wikitext text/x-wiki bidenaqf57rqwyzyikwh8g805z0om63 2417315 2417314 2026-05-20T17:28:29Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw 2417315 wikitext text/x-wiki nv6j7y68nwotlsdahv3gzp7yg1gfimq 2417316 2417315 2026-05-20T17:28:31Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:aqs 2417316 wikitext text/x-wiki n9go7ftmc94fohywqumnhho7jm0bbsx 2417317 2417316 2026-05-20T17:32:37Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2013.codfw.wmnet 2417317 wikitext text/x-wiki 4t1p52dgs7dt7jt9u082p125siodb5h 2417318 2417317 2026-05-20T17:32:40Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2014.codfw.wmnet 2417318 wikitext text/x-wiki d7toawcfzyvif4aok12gs9ubo3s6sfy 2417319 2417318 2026-05-20T17:38:57Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2014.codfw.wmnet 2417319 wikitext text/x-wiki ki7ol39hks6d1gndf3v2d9hq4jadv2s 2417320 2417319 2026-05-20T17:39:01Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2015.codfw.wmnet 2417320 wikitext text/x-wiki s8loxildrhb1sl2w0wz1g6ol4rovr63 2417321 2417320 2026-05-20T17:43:01Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns2005.wikimedia.org 2417321 wikitext text/x-wiki 7vhagnumzpk7rdzq7v81gjks2sb0efg 2417322 2417321 2026-05-20T17:43:20Z Stashbot 7414 hnowlan: disabled puppet on grafana* to temporarily fix file ownership issue on /etc/grafana/provisioning/plugins/mahendrapaipuri-dashboardreporter-app.yaml 2417322 wikitext text/x-wiki qr3x6ja71ytu8jn4kjgms0fgpg32czj 2417323 2417322 2026-05-20T17:45:14Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2015.codfw.wmnet 2417323 wikitext text/x-wiki heaf3kr2g2lkdlmibvqdtxhdx84b3vl 2417324 2417323 2026-05-20T17:45:18Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2021.codfw.wmnet 2417324 wikitext text/x-wiki 8lruxg3iadd6oybndg8u20ucpxa8f3c 2417325 2417324 2026-05-20T17:45:55Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Rebuild to pick up new production image (duration: 28m 32s) 2417325 wikitext text/x-wiki 9083ywmlrth9w4zuwhny8jwsj7u634l 2417326 2417325 2026-05-20T17:51:35Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2021.codfw.wmnet 2417326 wikitext text/x-wiki 8dxwn8t02syjtm6558fe1em4tmn5r6v 2417327 2417326 2026-05-20T17:51:39Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2022.codfw.wmnet 2417327 wikitext text/x-wiki 0ttv6jl5gwax8bk44jfropp7yt38xvl 2417328 2417327 2026-05-20T17:56:31Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for mr1-ulsfo,mr1-ulsfo IPv6,mr1-ulsfo.oob,mr1-ulsfo.oob IPv6 2417328 wikitext text/x-wiki mofxpiy0aojchlvmrqu3swuemcspbze 2417329 2417328 2026-05-20T17:56:34Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mr1-ulsfo,mr1-ulsfo IPv6,mr1-ulsfo.oob,mr1-ulsfo.oob IPv6 2417329 wikitext text/x-wiki ozuqqzope4sne9txb9mogn0m3029wjn 2417330 2417329 2026-05-20T17:56:52Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns2005.wikimedia.org 2417330 wikitext text/x-wiki bj0htd8icjzehnya8w5hucbmn93b04n 2417331 2417330 2026-05-20T17:57:51Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2022.codfw.wmnet 2417331 wikitext text/x-wiki 7nq60ye7pfn1flbhdrw0on2az6avutv 2417332 2417331 2026-05-20T17:57:55Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet 2417332 wikitext text/x-wiki f0l82a9zlxez3wo6l7g9lb848odnrt5 2417333 2417332 2026-05-20T18:01:49Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool ulsfo [reason: no reason specified, T416562] 2417333 wikitext text/x-wiki hf2gpj3o3r8f7cu0ynarqxovihoky9o 2417334 2417333 2026-05-20T18:01:54Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool ulsfo [reason: no reason specified, T416562] 2417334 wikitext text/x-wiki ohblzmt6rk3lvi8lkzp1oz9z5za321a 2417335 2417334 2026-05-20T18:04:53Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet 2417335 wikitext text/x-wiki omxj9cat8xs4nbfzsq4p5k1vxj64n1k 2417336 2417335 2026-05-20T18:07:37Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4047.ulsfo.wmnet 2417336 wikitext text/x-wiki 5pspemsohsw5ayulkhi10pd162ebkuo 2417337 2417336 2026-05-20T18:09:25Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4041.ulsfo.wmnet 2417337 wikitext text/x-wiki 4i1r37c3n18nee8s7ibaqojnxs0n8yt 2417340 2417337 2026-05-20T18:11:52Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns2006.wikimedia.org 2417340 wikitext text/x-wiki qgnuxokp9tpqkx32wi8xk8fpw7p1ntq 2417341 2417340 2026-05-20T18:26:12Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns2006.wikimedia.org 2417341 wikitext text/x-wiki a01tl4e8xz7u9sab04xqk40leugblzv 2417342 2417341 2026-05-20T18:36:32Z Stashbot 7414 dwisehaupt@dns1004: START - running authdns-update 2417342 wikitext text/x-wiki hdpbewajfcodtweloa028mjbemuwk7g 2417343 2417342 2026-05-20T18:37:13Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417343 wikitext text/x-wiki kt68v5eh52h7f1gwcfy1nnbc83ngp6j 2417344 2417343 2026-05-20T18:38:26Z Stashbot 7414 dwisehaupt@dns1004: END - running authdns-update 2417344 wikitext text/x-wiki dcibkqdi89v5d42qnx5a9l2zjap0h0y 2417345 2417344 2026-05-20T18:41:13Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns3003.wikimedia.org 2417345 wikitext text/x-wiki d2zqoos5nvq1g6j2z7vsncoxgb8mr7j 2417346 2417345 2026-05-20T18:41:50Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1290037|Update symfony/* (T426861)]] 2417346 wikitext text/x-wiki j4z7oz4dexlt7249t4705ybgsnyhpbw 2417347 2417346 2026-05-20T18:43:47Z Stashbot 7414 reedy@deploy1003: reedy: Backport for [[gerrit:1290037|Update symfony/* (T426861)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417347 wikitext text/x-wiki gef4g24lvdbnzlyvh6ukmowl665f1a8 2417348 2417347 2026-05-20T18:45:04Z Stashbot 7414 reedy@deploy1003: reedy: Continuing with deployment 2417348 wikitext text/x-wiki srxy7xxoiddnklfgaligx3wyegalakl 2417350 2417348 2026-05-20T18:47:33Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4048.ulsfo.wmnet 2417350 wikitext text/x-wiki nm67u9tt1dloap86vkrcpx1peyro5oa 2417351 2417350 2026-05-20T18:49:18Z Stashbot 7414 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290037|Update symfony/* (T426861)]] (duration: 07m 28s) 2417351 wikitext text/x-wiki du2cx3s0hkrt9b47ebyxivjcu9ggq4b 2417352 2417351 2026-05-20T18:50:08Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4042.ulsfo.wmnet 2417352 wikitext text/x-wiki dsc0jo941teikz50g6mvkri7jkg30cm 2417357 2417352 2026-05-20T18:56:20Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns3003.wikimedia.org 2417357 wikitext text/x-wiki b7utq3co4jqb4v5x4u9hxbguuhgtthm 2417358 2417357 2026-05-20T19:02:55Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417358 wikitext text/x-wiki mayijmfhcts7hbceype9w9w6snimts2 2417359 2417358 2026-05-20T19:03:41Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417359 wikitext text/x-wiki kzv25o9bdbwr24wh2oxpocsaz04avhi 2417360 2417359 2026-05-20T19:09:03Z Stashbot 7414 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1290012|Restore mistakenly-deleted messages (T111677)]], [[gerrit:1290013|Restore translations of mistakenly deleted messages (T111677)]] 2417360 wikitext text/x-wiki 9vipn7yt87d3b9hns526ga67n5ihltx 2417361 2417360 2026-05-20T19:11:21Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns3004.wikimedia.org 2417361 wikitext text/x-wiki ncbwxa9enfce0a9llybeejo3igu2bhc 2417362 2417361 2026-05-20T19:26:25Z Stashbot 7414 ejegg@deploy1003: ejegg: Backport for [[gerrit:1290012|Restore mistakenly-deleted messages (T111677)]], [[gerrit:1290013|Restore translations of mistakenly deleted messages (T111677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417362 wikitext text/x-wiki c8q6kta8flkolcwac5b5qdm9mog27gv 2417363 2417362 2026-05-20T19:26:30Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns3004.wikimedia.org 2417363 wikitext text/x-wiki 45ilja2lzgie0491vw4x8vddil9hcrm 2417364 2417363 2026-05-20T19:27:20Z Stashbot 7414 ejegg@deploy1003: ejegg: Continuing with deployment 2417364 wikitext text/x-wiki omt3811tnolc4asxxuns636j5xijjf6 2417365 2417364 2026-05-20T19:27:27Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4049.ulsfo.wmnet 2417365 wikitext text/x-wiki f9f1q3i133dquqys2talpcmo4ppzpth 2417366 2417365 2026-05-20T19:31:54Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4043.ulsfo.wmnet 2417366 wikitext text/x-wiki 85eeshpkvj75t32afezlz87whph3a0q 2417367 2417366 2026-05-20T19:39:21Z Stashbot 7414 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290012|Restore mistakenly-deleted messages (T111677)]], [[gerrit:1290013|Restore translations of mistakenly deleted messages (T111677)]] (duration: 30m 19s) 2417367 wikitext text/x-wiki awcnrut9hz040e8qtg3k8mp1b40ixep 2417368 2417367 2026-05-20T19:39:47Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2012.codfw.wmnet 2417368 wikitext text/x-wiki qweb1ffq00dbyhbpt5k2lmgwwl83kyw 2417369 2417368 2026-05-20T19:41:30Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns4003.wikimedia.org 2417369 wikitext text/x-wiki ls695hrq27kuxlbarww8k0pfwt4xk82 2417370 2417369 2026-05-20T19:49:34Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2012.codfw.wmnet 2417370 wikitext text/x-wiki a7ubgd3rfqf24c59h2ewpjbvhsx79g8 2417371 2417370 2026-05-20T19:50:41Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns4003.wikimedia.org 2417371 wikitext text/x-wiki qimcuuob43go8j2njrd8a95n9nh3krt 2417372 2417371 2026-05-20T20:00:23Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1011.eqiad.wmnet 2417372 wikitext text/x-wiki d5lyukgd5lglwleauvlhuypk5eu3i4l 2417373 2417372 2026-05-20T20:01:08Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417373 wikitext text/x-wiki s2cacxsdg9l93mcqgacvpux14ugupcn 2417374 2417373 2026-05-20T20:02:58Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.provision for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2417374 wikitext text/x-wiki nbfh1ccyyjcknny3o1uyj8l28kd6ksa 2417376 2417374 2026-05-20T20:05:41Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns4004.wikimedia.org 2417376 wikitext text/x-wiki po93xq53j35kf86pyum8qxoyj06ekf9 2417377 2417376 2026-05-20T20:06:51Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1011.eqiad.wmnet 2417377 wikitext text/x-wiki af3zvjgycl7n1fubz3otzkokcjqbny6 2417378 2417377 2026-05-20T20:06:56Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1012.eqiad.wmnet 2417378 wikitext text/x-wiki cpa3p73m5id7m3g768q2c86f263zus9 2417379 2417378 2026-05-20T20:07:15Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4050.ulsfo.wmnet 2417379 wikitext text/x-wiki qyi2vxzgp6kfz5m8lc7z7282vxemhfk 2417380 2417379 2026-05-20T20:12:54Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2417380 wikitext text/x-wiki 17zj1sotmzsefoz4jrbenosr1r14q6h 2417381 2417380 2026-05-20T20:12:57Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2417381 wikitext text/x-wiki oqenav40h8gjnlkadmrufuvyfc9k3ic 2417382 2417381 2026-05-20T20:13:39Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4044.ulsfo.wmnet 2417382 wikitext text/x-wiki 4h9y3dc3i8gidol7ewia2tqqbfyx8gd 2417383 2417382 2026-05-20T20:13:42Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_ulsfo and not P{cp4037.ulsfo.wmnet} and not P{cp4038.ulsfo.wmnet} and A:cp 2417383 wikitext text/x-wiki ogsmgdiph6nfkhls0zy1zv5twww93ow 2417384 2417383 2026-05-20T20:13:49Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1012.eqiad.wmnet 2417384 wikitext text/x-wiki nrwp1r4c0g5sgi6txg2we6y2najx20e 2417385 2417384 2026-05-20T20:13:55Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1013.eqiad.wmnet 2417385 wikitext text/x-wiki s5rjj2yqkkc3ovjvzxhgfjd6bwhu3id 2417386 2417385 2026-05-20T20:15:04Z Stashbot 7414 dwisehaupt@dns1005: START - running authdns-update 2417386 wikitext text/x-wiki pxwvqvhx0ky6phjmc1q4krhku0plr8g 2417387 2417386 2026-05-20T20:15:42Z Stashbot 7414 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on lvs2012.codfw.wmnet with reason: Maintenance 2417387 wikitext text/x-wiki 1tl9weporfgwp1vr28dt88v2al6h5q7 2417388 2417387 2026-05-20T20:15:47Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.provision for host lvs2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2417388 wikitext text/x-wiki iz2izlenv2dhyd54gdldujugv4nv5xw 2417389 2417388 2026-05-20T20:16:25Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2417389 wikitext text/x-wiki 8ox2jvr2luca7m9koqx8e8mm6i5yb7d 2417390 2417389 2026-05-20T20:16:41Z Stashbot 7414 dwisehaupt@dns1005: END - running authdns-update 2417390 wikitext text/x-wiki 0923l2clzwlrzkduszoen7e412v6ctu 2417392 2417390 2026-05-20T20:18:26Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2417392 wikitext text/x-wiki ow3lm3a3xfctgtxjf5ujnxs7dsalinb 2417393 2417392 2026-05-20T20:19:10Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417393 wikitext text/x-wiki 6fcg7o8ehe5ji480q95j1f5stigxv7a 2417394 2417393 2026-05-20T20:20:34Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1013.eqiad.wmnet 2417394 wikitext text/x-wiki cn1kyq3z7zns20r9xhw11ltjeyke5zf 2417395 2417394 2026-05-20T20:20:39Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1014.eqiad.wmnet 2417395 wikitext text/x-wiki pkq6toxdayiid1ku4mhfeiexlp8id85 2417396 2417395 2026-05-20T20:21:45Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns4004.wikimedia.org 2417396 wikitext text/x-wiki ch3efzyw9y2rppyfqaly1o93jfk20yb 2417397 2417396 2026-05-20T20:26:55Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1014.eqiad.wmnet 2417397 wikitext text/x-wiki 4sdpf0jn2i02m2qmqrjvvxk4gwqm6po 2417398 2417397 2026-05-20T20:26:59Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1015.eqiad.wmnet 2417398 wikitext text/x-wiki eo0qsginxdm8l50lu44u1tn7h69lggq 2417399 2417398 2026-05-20T20:33:20Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1015.eqiad.wmnet 2417399 wikitext text/x-wiki r1xuhhu23nwauqd71olgsnry100x9sl 2417400 2417399 2026-05-20T20:33:25Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1016.eqiad.wmnet 2417400 wikitext text/x-wiki jd6q76hdnrx8b2637fi1xs6k8dcn3uv 2417401 2417400 2026-05-20T20:33:52Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lvs2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2417401 wikitext text/x-wiki mwlxsfkc48l5jmn3s1d2xb597tiypj2 2417402 2417401 2026-05-20T20:34:59Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs2012.codfw.wmnet with OS bullseye 2417402 wikitext text/x-wiki f6vwqiav4387wo18e9fxmfynay64k2a 2417403 2417402 2026-05-20T20:36:45Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns5003.wikimedia.org 2417403 wikitext text/x-wiki 0wqfruljso2c38lared6ckpmymalsdo 2417404 2417403 2026-05-20T20:37:28Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417404 wikitext text/x-wiki 17j7zeuwyh0zisqe388l4o948bvws08 2417405 2417404 2026-05-20T20:39:45Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1016.eqiad.wmnet 2417405 wikitext text/x-wiki h7xpkj3l4ousiq994uk2gsymz9u0oxw 2417406 2417405 2026-05-20T20:39:49Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1017.eqiad.wmnet 2417406 wikitext text/x-wiki kbmznx3ansrcllnhvmx6v6c8d8clyj5 2417407 2417406 2026-05-20T20:46:40Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1017.eqiad.wmnet 2417407 wikitext text/x-wiki 26t9z2mioc2os46v1wr1bkwdx49mm25 2417408 2417407 2026-05-20T20:46:44Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1018.eqiad.wmnet 2417408 wikitext text/x-wiki j8qauv7q3r9hzg7kqhojwbx5l7eloce 2417409 2417408 2026-05-20T20:47:01Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4051.ulsfo.wmnet 2417409 wikitext text/x-wiki 1098gaab41k4dthwmzy2i39obnl4p6f 2417410 2417409 2026-05-20T20:49:23Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:1287433|Restrict the changetags user right to bots and sysops on mediawiki.org (T355445)]] 2417410 wikitext text/x-wiki 4flowr5lpe6g7kliw7m2ns1sugboqi5 2417411 2417410 2026-05-20T20:52:11Z Stashbot 7414 dancy@deploy1003: codenamenoreste, dancy: Backport for [[gerrit:1287433|Restrict the changetags user right to bots and sysops on mediawiki.org (T355445)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417411 wikitext text/x-wiki ov3ndksh4ud8b9lr19d7mh0vr53zdza 2417412 2417411 2026-05-20T20:53:05Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns5003.wikimedia.org 2417412 wikitext text/x-wiki 4e8f6cohcc0cfqm2d76v3inyjppd7d3 2417413 2417412 2026-05-20T20:53:45Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs2012.codfw.wmnet with reason: host reimage 2417413 wikitext text/x-wiki iba8gyklsi4ti5w6f6v4aauclbl2yhy 2417414 2417413 2026-05-20T20:53:47Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1018.eqiad.wmnet 2417414 wikitext text/x-wiki ajfp422k7lc2pmf47byylduc9buli87 2417415 2417414 2026-05-20T20:53:52Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1019.eqiad.wmnet 2417415 wikitext text/x-wiki hz6ge0bpvbfepexpb8afrhiqno3hive 2417416 2417415 2026-05-20T20:54:06Z Stashbot 7414 dancy@deploy1003: codenamenoreste, dancy: Continuing with deployment 2417416 wikitext text/x-wiki pfeyyrf0dj2frhfg0equn7ceetyi7ah 2417417 2417416 2026-05-20T20:58:32Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2012.codfw.wmnet with reason: host reimage 2417417 wikitext text/x-wiki chsz7b9secwzzubu87pq836jyti2cwn 2417419 2417417 2026-05-20T21:00:08Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287433|Restrict the changetags user right to bots and sysops on mediawiki.org (T355445)]] (duration: 10m 45s) 2417419 wikitext text/x-wiki hmox3z5eeqx89uq5lnhb9ah99dqelrw 2417420 2417419 2026-05-20T21:00:46Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1019.eqiad.wmnet 2417420 wikitext text/x-wiki o68eubcbd6mvdbabevzcjgh9tk9bqk9 2417421 2417420 2026-05-20T21:00:52Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1020.eqiad.wmnet 2417421 wikitext text/x-wiki 4srjcwzupdr6trdgr53248hu4aj7byg 2417422 2417421 2026-05-20T21:07:45Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1020.eqiad.wmnet 2417422 wikitext text/x-wiki rcodnu3vbp0f8helnopg4udu78wlmp5 2417423 2417422 2026-05-20T21:07:50Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1021.eqiad.wmnet 2417423 wikitext text/x-wiki jphdpg0r5tmgiw4egcz3qlgkt9vyw65 2417424 2417423 2026-05-20T21:08:05Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns5004.wikimedia.org 2417424 wikitext text/x-wiki f2pux380u8v33bfg0is10osbw56pfje 2417425 2417424 2026-05-20T21:14:38Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1021.eqiad.wmnet 2417425 wikitext text/x-wiki 4m3xkpri7u8u93272r9stiv9vy6qghc 2417426 2417425 2026-05-20T21:14:42Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1022.eqiad.wmnet 2417426 wikitext text/x-wiki 0pogxgwm8moh0zp9v13x6uf6hki8ie7 2417427 2417426 2026-05-20T21:16:07Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs2012.codfw.wmnet with OS bullseye 2417427 wikitext text/x-wiki ivv7dm3xof68273is7wgc6ukg76bokm 2417428 2417427 2026-05-20T21:21:36Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1022.eqiad.wmnet 2417428 wikitext text/x-wiki 95ikyh2us1ifk7ekxubcyb4ixkylxu3 2417429 2417428 2026-05-20T21:22:30Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns5004.wikimedia.org 2417429 wikitext text/x-wiki swx37l4c7encucd2qvauu0ufz3ea52m 2417432 2417429 2026-05-20T21:26:46Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp4052.ulsfo.wmnet 2417432 wikitext text/x-wiki 60rzg8r4pijrxq7rm2kvea67ybgwg36 2417433 2417432 2026-05-20T21:26:49Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_ulsfo and A:cp 2417433 wikitext text/x-wiki ea7hcjrxbrhdg91a9zbv4uv7udnh75d 2417434 2417433 2026-05-20T21:27:03Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1025.eqiad.wmnet 2417434 wikitext text/x-wiki lemryhulunm4d5a5vf008xqku82ebei 2417435 2417434 2026-05-20T21:33:20Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1025.eqiad.wmnet 2417435 wikitext text/x-wiki 5zfm3109yn1xb5brztowroh0iy7vfek 2417436 2417435 2026-05-20T21:33:25Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1026.eqiad.wmnet 2417436 wikitext text/x-wiki 5hsbttrod5qt6iovtew359e4c7rck4r 2417437 2417436 2026-05-20T21:37:30Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot begin reboot of dns6001.wikimedia.org 2417437 wikitext text/x-wiki m2yfmvdhduc7blq3qi3b3zvpbd7wdl4 2417438 2417437 2026-05-20T21:40:19Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1026.eqiad.wmnet 2417438 wikitext text/x-wiki o2kttnuqd0yntfvbwwqfca86o2q3829 2417439 2417438 2026-05-20T21:46:00Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2018.codfw.wmnet 2417439 wikitext text/x-wiki d6dat2s2uzb3js49y4unuxj8vt0o0xu 2417440 2417439 2026-05-20T21:47:53Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp6009.drmrs.wmnet} and A:cp 2417440 wikitext text/x-wiki efes9hldxyytpaut50mia5rmxveht74 2417441 2417440 2026-05-20T21:48:07Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp6001.drmrs.wmnet} and A:cp 2417441 wikitext text/x-wiki h78jfry34xt2smobhjqggk09kuls77d 2417442 2417441 2026-05-20T21:51:38Z Stashbot 7414 brett@cumin2002: cookbooks.sre.dns.roll-reboot finished rebooting dns6001.wikimedia.org 2417442 wikitext text/x-wiki tjxos7rgkwvxxx36le81i3jrjl10prz 2417443 2417442 2026-05-20T21:51:40Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:dnsbox and not P{dns6002.wikimedia.org} and not A:magru and (A:dnsbox) 2417443 wikitext text/x-wiki qhloxlzo33b066nno4gnns20e89npst 2417444 2417443 2026-05-20T21:52:16Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2018.codfw.wmnet 2417444 wikitext text/x-wiki 099vjhwbzb1kzo213evdctjp0jnohuk 2417445 2417444 2026-05-20T21:52:20Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2019.codfw.wmnet 2417445 wikitext text/x-wiki kmy980zbnn9gsp8c6rvod64wsmj5hg2 2417446 2417445 2026-05-20T21:55:10Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:aqs 2417446 wikitext text/x-wiki fxpbzrqdb7ivkt4g62qh3tcnbt5auvi 2417447 2417446 2026-05-20T21:58:25Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2019.codfw.wmnet 2417447 wikitext text/x-wiki ghhz2gfiou8o7lvj17d95fhrj83rpa2 2417448 2417447 2026-05-20T21:58:29Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2020.codfw.wmnet 2417448 wikitext text/x-wiki 18z54llko3aysras2k97hzi19hpah20 2417449 2417448 2026-05-20T21:58:57Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6001.drmrs.wmnet 2417449 wikitext text/x-wiki 87ph6p9u7lhkniz8w517qrs7zk3sjey 2417450 2417449 2026-05-20T21:58:59Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp6001.drmrs.wmnet} and A:cp 2417450 wikitext text/x-wiki hus619mk9h3zsi0w29ourx0bjjfhfm1 2417451 2417450 2026-05-20T21:59:02Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6009.drmrs.wmnet 2417451 wikitext text/x-wiki jva0x33k03g1xwjuf82gpexlxwhy91s 2417452 2417451 2026-05-20T21:59:04Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp6009.drmrs.wmnet} and A:cp 2417452 wikitext text/x-wiki 9hweuliw6dyb3xmbz6q070050c8xugy 2417453 2417452 2026-05-20T22:04:58Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2020.codfw.wmnet 2417453 wikitext text/x-wiki 517v93ybz7fnsx1hsh6i8n5ba32d4nw 2417455 2417453 2026-05-20T22:23:16Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2017.codfw.wmnet 2417455 wikitext text/x-wiki oywzg4fukeu7oxj9w9w6b6s1kgpnwn3 2417456 2417455 2026-05-20T22:29:47Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2017.codfw.wmnet 2417456 wikitext text/x-wiki mgb1rtesus0xk90y59nlk1f64etfkd1 2417457 2417456 2026-05-20T22:29:50Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2026.codfw.wmnet 2417457 wikitext text/x-wiki 6zeu16zktqni7s2lwdi28l3arvvsh7r 2417458 2417457 2026-05-20T22:33:54Z Stashbot 7414 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2417458 wikitext text/x-wiki ky3r4uzt0mgluxb4gzg7nzdb86gfvtb 2417459 2417458 2026-05-20T22:36:35Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2026.codfw.wmnet 2417459 wikitext text/x-wiki t5687ytyfz9h9qop8cftlu3uaq1bhwk 2417460 2417459 2026-05-20T22:36:39Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2027.codfw.wmnet 2417460 wikitext text/x-wiki 6nitcj2mvfmf9ealkzik3i6izn2wq4z 2417461 2417460 2026-05-20T22:43:21Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2027.codfw.wmnet 2417461 wikitext text/x-wiki hwesvwutv8qi1vsdlxghw4gvxay3bhn 2417462 2417461 2026-05-20T22:59:55Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1290086|errorpage: Fix unclosed bold tag in 404.php (T129433)]] 2417462 wikitext text/x-wiki awnczjggoe1z3ty8cicaghkp2cxrf0u 2417463 2417462 2026-05-20T23:02:01Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup, krinkle: Backport for [[gerrit:1290086|errorpage: Fix unclosed bold tag in 404.php (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417463 wikitext text/x-wiki 75g7h92bji63jkl1i1lsnl99onyulul 2417464 2417463 2026-05-20T23:02:50Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup, krinkle: Continuing with deployment 2417464 wikitext text/x-wiki 21emuv1hc7szly1d8bk6px4ajug2e7c 2417465 2417464 2026-05-20T23:07:04Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290086|errorpage: Fix unclosed bold tag in 404.php (T129433)]] (duration: 07m 09s) 2417465 wikitext text/x-wiki 287pvffumdojy6oavsyk3glcfq30dp7 2417466 2417465 2026-05-20T23:07:33Z Stashbot 7414 Amir1: wikiadmin2023@10.64.48.159(svwiki)> delete from user_properties where up_value = '2' and up_property = 'thumbsize'; Query OK, 215 rows affected (0.018 sec) (T426880 and T376152) 2417466 wikitext text/x-wiki 4n3ompa52rus8gbd96741pphabsjm8k 2417467 2417466 2026-05-20T23:10:29Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1290075|Migrate Swedish to same preference values as other wikis (1/2) (T426880)]] 2417467 wikitext text/x-wiki k7ti1nwqdqju7yyk8qo8ohydlb4hwvr 2417468 2417467 2026-05-20T23:12:26Z Stashbot 7414 ladsgroup@deploy1003: jdlrobson, ladsgroup: Backport for [[gerrit:1290075|Migrate Swedish to same preference values as other wikis (1/2) (T426880)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417468 wikitext text/x-wiki t5h2putc10sfwerhiytju24l495hj1k 2417469 2417468 2026-05-20T23:14:57Z Stashbot 7414 ladsgroup@deploy1003: jdlrobson, ladsgroup: Continuing with deployment 2417469 wikitext text/x-wiki 8iqne97an6xluypneaxd0k5f1pyv86s 2417470 2417469 2026-05-20T23:19:06Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290075|Migrate Swedish to same preference values as other wikis (1/2) (T426880)]] (duration: 08m 35s) 2417470 wikitext text/x-wiki 6q01pzwy2ffh7rrtfoyqntelnfmofyx 2417471 2417470 2026-05-20T23:26:04Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1290076|Migrate Swedish to same preference values as other wikis (2/2) (T426880)]] 2417471 wikitext text/x-wiki nu5bivnsurzcv50kote6sfmxpvoxfns 2417472 2417471 2026-05-20T23:27:55Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup, jdlrobson: Backport for [[gerrit:1290076|Migrate Swedish to same preference values as other wikis (2/2) (T426880)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417472 wikitext text/x-wiki 2x8eysvjlus7okpc2wjtzj4imnoqbp9 2417473 2417472 2026-05-20T23:28:34Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup, jdlrobson: Continuing with deployment 2417473 wikitext text/x-wiki jq8dysb30latjfb44a8sc9chruw2zfm 2417474 2417473 2026-05-20T23:32:41Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290076|Migrate Swedish to same preference values as other wikis (2/2) (T426880)]] (duration: 06m 37s) 2417474 wikitext text/x-wiki 9yjv40e5y58quem5wvexo350qzfogp7 2417484 2417474 2026-05-21T00:55:06Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2417484 wikitext text/x-wiki 5fy6rkkobtc30x1znt0wucoqo1sm641 2417485 2417484 2026-05-21T01:22:54Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet 2417485 wikitext text/x-wiki gn808scewhpokg8hqisjuxjlay7h8eu 2417486 2417485 2026-05-21T01:29:55Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet 2417486 wikitext text/x-wiki c6p3b6sv7c3j21n50af4boec8zl7f4w 2417487 2417486 2026-05-21T02:01:13Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2417487 wikitext text/x-wiki dqzmrp2y2lswqcya7lup8kq8ryxzj6b 2417488 2417487 2026-05-21T02:07:42Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) 2417488 wikitext text/x-wiki kvefkngubgfxpz39yljranlfyz2v7r6 2417489 2417488 2026-05-21T02:11:44Z Stashbot 7414 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2417489 wikitext text/x-wiki 5hje3f6ad6xlwzyjg7l539g3mkiwy5z 2417490 2417489 2026-05-21T02:41:24Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on planet1003.eqiad.wmnet with reason: debug wip 2417490 wikitext text/x-wiki i5nqlngt48ubcmtofm9jgrd9idohk1f 2417492 2417490 2026-05-21T05:21:43Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1012.eqiad.wmnet with reason: Cloning 2417492 wikitext text/x-wiki cfztsgmnhi1csnjagcvrhdqp3xawuvo 2417493 2417492 2026-05-21T05:29:08Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1022 to pc2 master T418973', diff saved to https://phabricator.wikimedia.org/P92726 and previous config saved to /var/cache/conftool/dbconfig/20260521-052905-marostegui.json 2417493 wikitext text/x-wiki jzdms9u0n0ww1zrjpguju71ogupry94 2417494 2417493 2026-05-21T05:30:03Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc2 T418973', diff saved to https://phabricator.wikimedia.org/P92727 and previous config saved to /var/cache/conftool/dbconfig/20260521-053000-marostegui.json 2417494 wikitext text/x-wiki shjv23606l8yuv4o5e6zyoyehjqpv6e 2417496 2417494 2026-05-21T05:39:02Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1012 from dbctl T426930', diff saved to https://phabricator.wikimedia.org/P92728 and previous config saved to /var/cache/conftool/dbconfig/20260521-053858-marostegui.json 2417496 wikitext text/x-wiki lgu3u5gyx33avni50jzibh2l41fyj8p 2417497 2417496 2026-05-21T06:14:00Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2417497 wikitext text/x-wiki nqn4s0w43m8cy4pg5rbqvtkw15wyvk0 2417498 2417497 2026-05-21T06:14:15Z Stashbot 7414 marostegui: Failover m2-master T426633 2417498 wikitext text/x-wiki ajwi828uthq50s7u55zfqbglsdiqslb 2417499 2417498 2026-05-21T06:15:43Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2417499 wikitext text/x-wiki 2neip50jkmhq4ufhivz6ukxb3d8fohq 2417500 2417499 2026-05-21T06:22:11Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit2003 2417500 wikitext text/x-wiki 0rtfwo6iw940kzpt0pl8hs66e1isfwf 2417501 2417500 2026-05-21T06:23:25Z Stashbot 7414 arnaudb@cumin1003: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit2003 2417501 wikitext text/x-wiki 7npziwl76anjh718sor9amv364zd4rg 2417502 2417501 2026-05-21T06:24:56Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd 2417502 wikitext text/x-wiki 0ydlx1rbl8p8mc4g8zndk8tddxjpzb4 2417503 2417502 2026-05-21T06:33:22Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet 2417503 wikitext text/x-wiki h1wfk0kzrwligh84jdhtdw8tlydxyfr 2417504 2417503 2026-05-21T06:34:05Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org 2417504 wikitext text/x-wiki 7kq8j643v4wu2j0g30qwmhwodo1spd4 2417505 2417504 2026-05-21T06:34:46Z Stashbot 7414 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org 2417505 wikitext text/x-wiki 0jt01wen5wqccrb4y8cv7xj7ygl9m3a 2417506 2417505 2026-05-21T06:40:00Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet 2417506 wikitext text/x-wiki 6cd73o31wydh94yn9pndj9s5v42zlal 2417507 2417506 2026-05-21T06:40:45Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org 2417507 wikitext text/x-wiki epklq00i4b6vwdm96cp75sr6g297lem 2417508 2417507 2026-05-21T06:42:18Z Stashbot 7414 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org 2417508 wikitext text/x-wiki 1r0cehob8j1j7mz3n9jzbanzhnnrl1l 2417509 2417508 2026-05-21T06:49:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd 2417509 wikitext text/x-wiki 11dqrf8bcpppd3bkb8t9qd58ol4isxc 2417510 2417509 2026-05-21T06:52:19Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain 2417510 wikitext text/x-wiki lrnl9lj85yeequkq4tqabna8di1s6m8 2417511 2417510 2026-05-21T06:53:09Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain 2417511 wikitext text/x-wiki 0v8hw3lcyjicalgykw1p7yvqtpr11nw 2417512 2417511 2026-05-21T06:54:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd 2417512 wikitext text/x-wiki 6wfyewhlg9ml40iplpcemyqeoaxyogg 2417513 2417512 2026-05-21T07:04:48Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd 2417513 wikitext text/x-wiki g175o0xme8glamm6lg4l7zln18886mn 2417514 2417513 2026-05-21T07:17:52Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbproxy1025.eqiad.wmnet with reason: Rebooting 2417514 wikitext text/x-wiki r82wu84bbsfdf4wglgbybwtb7nymji3 2417515 2417514 2026-05-21T07:18:10Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain 2417515 wikitext text/x-wiki d8mr185wxk8htba2yw94wmqipa2pwa2 2417516 2417515 2026-05-21T07:20:37Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain 2417516 wikitext text/x-wiki qj7qdg6qneju37dp1zpgm2p0htqaqtb 2417517 2417516 2026-05-21T07:21:35Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2417517 wikitext text/x-wiki 6qkv0a4w5o264z57bcrg5bz2bvdrfy1 2417518 2417517 2026-05-21T07:21:57Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1256: Upgrading db1256.eqiad.wmnet 2417518 wikitext text/x-wiki cmwpgehohedaag4u6ljzd6svo6xaxev 2417519 2417518 2026-05-21T07:22:26Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1256: Upgrading db1256.eqiad.wmnet 2417519 wikitext text/x-wiki 07aum7ce1551frul3v9igrb8iul6bvs 2417520 2417519 2026-05-21T07:24:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1256.eqiad.wmnet with OS trixie 2417520 wikitext text/x-wiki 50wamy2g7wjpbyllnzu84ntn8929mvv 2417521 2417520 2026-05-21T07:24:23Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp6002.drmrs.wmnet} and A:cp 2417521 wikitext text/x-wiki eqsloj39w1whm7ig4o7lux1ftm6au3g 2417522 2417521 2026-05-21T07:28:00Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd 2417522 wikitext text/x-wiki kun6isnbx43sg8a9y5jmg1nbp14q7qc 2417523 2417522 2026-05-21T07:35:18Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet 2417523 wikitext text/x-wiki 1ol5dqo1a3oogjf77y5duw3lofwvkuz 2417524 2417523 2026-05-21T07:35:21Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp6002.drmrs.wmnet} and A:cp 2417524 wikitext text/x-wiki 36n8ql68gnuerc0hnipp7qdpfh9awv7 2417525 2417524 2026-05-21T07:35:49Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp6010.drmrs.wmnet} and A:cp 2417525 wikitext text/x-wiki g7sp01338ltvo2co02q6bsaqju020kp 2417526 2417525 2026-05-21T07:38:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage 2417526 wikitext text/x-wiki shzahf60uodzuxdtk2bgsluio4yhopm 2417527 2417526 2026-05-21T07:42:23Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd 2417527 wikitext text/x-wiki phttj7j7pbp3z7v84xjwjguhg7k638r 2417528 2417527 2026-05-21T07:43:59Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage 2417528 wikitext text/x-wiki 02m9tk00og5592yz2h6mqivzil76wzf 2417529 2417528 2026-05-21T07:44:23Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to plain 2417529 wikitext text/x-wiki 3f2roytboi3h0zo3oz5w36mt6sui4oq 2417530 2417529 2026-05-21T07:45:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to plain 2417530 wikitext text/x-wiki 5kyfbkl5bax3j3hn8zodxfkm9305bj9 2417531 2417530 2026-05-21T07:46:16Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet 2417531 wikitext text/x-wiki 1fch186lckkfd6cpywaxzayqaye3qre 2417532 2417531 2026-05-21T07:46:17Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp6010.drmrs.wmnet} and A:cp 2417532 wikitext text/x-wiki l627omkmjcbuitp4l800e8qedpjvw1j 2417533 2417532 2026-05-21T07:47:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1023.eqiad.wmnet 2417533 wikitext text/x-wiki efp03f8r074l6bd0qorydzx47sqsbo0 2417534 2417533 2026-05-21T07:48:23Z Stashbot 7414 marostegui: Failover m3-master T426633 2417534 wikitext text/x-wiki kollbbl313z2z5z3w7oc73trghsxccg 2417535 2417534 2026-05-21T07:50:18Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2417535 wikitext text/x-wiki 2bczvp3em0wit2fylckpybtufh6oizx 2417536 2417535 2026-05-21T07:51:55Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2417536 wikitext text/x-wiki olsl7wrafou3mwe1y70ewmvf65m6b9m 2417537 2417536 2026-05-21T07:52:35Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp600[3-4].drmrs.wmnet} and A:cp 2417537 wikitext text/x-wiki llgilbmaly4fkpk0n5yejaz0p802w61 2417538 2417537 2026-05-21T08:00:18Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1256.eqiad.wmnet with OS trixie 2417538 wikitext text/x-wiki kpaypzieny9a33n9gqijsm1bsyusvq9 2417539 2417538 2026-05-21T08:01:33Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet 2417539 wikitext text/x-wiki j9bly8ah7gklryibjuissaqjch2n3ft 2417540 2417539 2026-05-21T08:02:30Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1256: Migration of db1256.eqiad.wmnet completed 2417540 wikitext text/x-wiki k3txz1i7ewvcfek6tv314xjg892gh5z 2417542 2417540 2026-05-21T08:16:37Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance 2417542 wikitext text/x-wiki ay2jxtunf4r0tyy360d2trbris192fg 2417543 2417542 2026-05-21T08:16:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1036 (T426633)', diff saved to https://phabricator.wikimedia.org/P92731 and previous config saved to /var/cache/conftool/dbconfig/20260521-081642-fceratto.json 2417543 wikitext text/x-wiki 0ddx4n2iytgk3u98yjpcpdk37gfap9y 2417545 2417543 2026-05-21T08:29:05Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.3 refs T423912 2417545 wikitext text/x-wiki o5n4srk3qa1z4sifg5safjtuq9r0ma2 2417546 2417545 2026-05-21T08:29:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1036 (T426633)', diff saved to https://phabricator.wikimedia.org/P92733 and previous config saved to /var/cache/conftool/dbconfig/20260521-082951-fceratto.json 2417546 wikitext text/x-wiki jrw3yhnjghhorxt8vf57tapxll3l39p 2417547 2417546 2026-05-21T08:37:37Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Repooling 2417547 wikitext text/x-wiki ov4ppzq2zalhpr1lj1v43qee5cv8sby 2417548 2417547 2026-05-21T08:42:21Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet 2417548 wikitext text/x-wiki 4pzs7cl99qgfa4eopvxxp8fqjra5lf6 2417549 2417548 2026-05-21T08:42:23Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp600[3-4].drmrs.wmnet} and A:cp 2417549 wikitext text/x-wiki 0dxumhyvos990rkf6q5mf9w3d58vd24 2417550 2417549 2026-05-21T08:44:41Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp601[1-2].drmrs.wmnet} and A:cp 2417550 wikitext text/x-wiki 38vo9pzd4kq446f1l0wpuert3dkxmnz 2417551 2417550 2026-05-21T08:47:58Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1256: Migration of db1256.eqiad.wmnet completed 2417551 wikitext text/x-wiki tgndov3jmsg3dquui4usk1flzuex8mk 2417552 2417551 2026-05-21T08:48:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2417552 wikitext text/x-wiki biic41vbh77hezzswcw7tn9dn4ldrka 2417553 2417552 2026-05-21T08:49:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1023.eqiad.wmnet 2417553 wikitext text/x-wiki cj37asi3d2oh8hqhc0jjfy89uwq9as7 2417554 2417553 2026-05-21T08:55:15Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet 2417554 wikitext text/x-wiki mqcu4mhmprsduc646sez8k0c6jiyrh5 2417555 2417554 2026-05-21T09:01:14Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.provision for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2417555 wikitext text/x-wiki dc0ijaqzvddr4q793x7jz4im649k5bi 2417556 2417555 2026-05-21T09:02:12Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1375.eqiad.wmnet 2417556 wikitext text/x-wiki 8dwisin9c5y1biwt7hv5wsxkawyv2i4 2417561 2417556 2026-05-21T09:06:04Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1037.eqiad.wmnet with reason: Maintenance 2417561 wikitext text/x-wiki 2vkwlv3ishgtc6bolbmc1b5atid5ys6 2417562 2417561 2026-05-21T09:06:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1037 (T426633)', diff saved to https://phabricator.wikimedia.org/P92738 and previous config saved to /var/cache/conftool/dbconfig/20260521-090609-fceratto.json 2417562 wikitext text/x-wiki f7dnj07pzvjp5vgnbf36acevp9di5um 2417563 2417562 2026-05-21T09:07:30Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1375.eqiad.wmnet 2417563 wikitext text/x-wiki jifw5y6395lt0eizmqii1lid2tbrib2 2417564 2417563 2026-05-21T09:07:34Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1376.eqiad.wmnet 2417564 wikitext text/x-wiki 8yvww6vpb8bvio2gzkc428j9bgvna3a 2417565 2417564 2026-05-21T09:08:00Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1036: Repooling 2417565 wikitext text/x-wiki 1q4wo3r2fko7tvlpjvwxh2t238056vm 2417568 2417565 2026-05-21T09:12:51Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1376.eqiad.wmnet 2417568 wikitext text/x-wiki fzfor64pou66ckbpqhtrw3i9xmtjy63 2417569 2417568 2026-05-21T09:12:55Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1377.eqiad.wmnet 2417569 wikitext text/x-wiki h690r3fug3mek42jikvd6f4yv9m7gn1 2417571 2417569 2026-05-21T09:16:45Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1377.eqiad.wmnet 2417571 wikitext text/x-wiki kvngxbaxl6ft584gysocbalrg9uslym 2417572 2417571 2026-05-21T09:16:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1378.eqiad.wmnet 2417572 wikitext text/x-wiki rat949g8y8m58uai8tu1zlz6qn7avzn 2417573 2417572 2026-05-21T09:17:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 (T426633)', diff saved to https://phabricator.wikimedia.org/P92740 and previous config saved to /var/cache/conftool/dbconfig/20260521-091738-fceratto.json 2417573 wikitext text/x-wiki 5pg971ot21o0mb2zdgnlmns3ybwb1ba 2417574 2417573 2026-05-21T09:18:54Z Stashbot 7414 moritzm: remove ganeti1023 foom eqiad Ganeti cluster T424680 2417574 wikitext text/x-wiki htxjzdendvyr701oun9glg9r2dsee9u 2417575 2417574 2026-05-21T09:19:00Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2417575 wikitext text/x-wiki nd7bfwu52zmgdt0a9ecforsusqy0lg7 2417576 2417575 2026-05-21T09:20:53Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417576 wikitext text/x-wiki 63dl46rgnudubr8j1zmrxi83ol33zvd 2417577 2417576 2026-05-21T09:21:46Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A 2417577 wikitext text/x-wiki gk9y7awnzehmugziqknlmdfatvwkcex 2417578 2417577 2026-05-21T09:21:52Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw 2417578 wikitext text/x-wiki rseg4pjzpv3mzt00365bl0dxypb64uz 2417579 2417578 2026-05-21T09:21:56Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet 2417579 wikitext text/x-wiki 48wm7hvuuiu7kjymip0m2yit3txmfd7 2417580 2417579 2026-05-21T09:21:58Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet 2417580 wikitext text/x-wiki 6v3o04i58z2f7uazxrlhz1gekbxytjk 2417581 2417580 2026-05-21T09:22:15Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1378.eqiad.wmnet 2417581 wikitext text/x-wiki sacm6sd0t69h7xlsj3t4hcppklyepsv 2417582 2417581 2026-05-21T09:22:19Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1379.eqiad.wmnet 2417582 wikitext text/x-wiki 7oyn8yxlgnjowz7b5jlkgq7aen3yohq 2417583 2417582 2026-05-21T09:22:44Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad 2417583 wikitext text/x-wiki h4csgt0tk1sphjljof6pzu1v52f4o8v 2417584 2417583 2026-05-21T09:22:47Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet 2417584 wikitext text/x-wiki ldvyxdnwcu2gsaqfv5yius7ll3wbww9 2417585 2417584 2026-05-21T09:22:49Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet 2417585 wikitext text/x-wiki 6sgziuaos6xfhdocqbin1t850e8md5p 2417586 2417585 2026-05-21T09:23:33Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet 2417586 wikitext text/x-wiki qdm8lqo8ka5ll7zuz4jtkjljbum50qv 2417587 2417586 2026-05-21T09:23:57Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A 2417587 wikitext text/x-wiki nqwec1a8an8nxc20d8ozi78qsj0s0wf 2417588 2417587 2026-05-21T09:24:09Z Stashbot 7414 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw 2417588 wikitext text/x-wiki 7fsg828tbpnubnerns4h3sv34pykc2n 2417589 2417588 2026-05-21T09:25:05Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet 2417589 wikitext text/x-wiki qced4i9hwbnz81cmqzlrrdr66bdcvkh 2417590 2417589 2026-05-21T09:26:37Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet 2417590 wikitext text/x-wiki 816dvnky8nu4hf9pepn0v4tvqvrqf8c 2417591 2417590 2026-05-21T09:27:20Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet 2417591 wikitext text/x-wiki sjpy0ivebyx5ftw1e4yzb1yl45ehr85 2417592 2417591 2026-05-21T09:27:44Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1379.eqiad.wmnet 2417592 wikitext text/x-wiki asmeqwos7gi5bf9v82jan9tx8ni7j1y 2417593 2417592 2026-05-21T09:27:48Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1380.eqiad.wmnet 2417593 wikitext text/x-wiki nf34ssegdhow2xbjaudetp09bnv1geq 2417594 2417593 2026-05-21T09:27:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92741 and previous config saved to /var/cache/conftool/dbconfig/20260521-092746-fceratto.json 2417594 wikitext text/x-wiki nc08vfgf78zra3nidcyxatdrx98hxdx 2417595 2417594 2026-05-21T09:28:10Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet 2417595 wikitext text/x-wiki bo3xm0jf5t1xswikzsg1agrolx4sm0l 2417597 2417595 2026-05-21T09:29:09Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet 2417597 wikitext text/x-wiki ndnsqdns4opgw06fs4l98szens3fjiv 2417598 2417597 2026-05-21T09:29:27Z Stashbot 7414 jayme@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw 2417598 wikitext text/x-wiki aibgoabiuz41ppai36xcac0wfokt95z 2417599 2417598 2026-05-21T09:29:32Z Stashbot 7414 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad 2417599 wikitext text/x-wiki p2ve5qatnpvzugl9dsqr2xdct8gcncu 2417600 2417599 2026-05-21T09:29:49Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet 2417600 wikitext text/x-wiki gls1b9sykm12b4brj58e5bm5qdukxff 2417601 2417600 2026-05-21T09:30:03Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet 2417601 wikitext text/x-wiki dorjnv5m1ub581nvzawrgl9d39l4ct2 2417602 2417601 2026-05-21T09:30:05Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet 2417602 wikitext text/x-wiki fmu8enn8wp097kylxxs5s62gzxw38yu 2417603 2417602 2026-05-21T09:30:10Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet 2417603 wikitext text/x-wiki 3ivcbt4ydyvfji36o5hm8usavqzxl30 2417604 2417603 2026-05-21T09:30:12Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet 2417604 wikitext text/x-wiki myq1sicc924aarmb8a15ox62z0pppqz 2417605 2417604 2026-05-21T09:31:37Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet 2417605 wikitext text/x-wiki ilc5htzoayi4cv8l0ndxt8cuo1hpmbj 2417606 2417605 2026-05-21T09:31:39Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet 2417606 wikitext text/x-wiki f0ls5hoye5zmdxy60ztu5ovu3r2iihb 2417607 2417606 2026-05-21T09:31:45Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet 2417607 wikitext text/x-wiki c2o9z10dvmc9mprej3mqxwxk0ok1y8s 2417608 2417607 2026-05-21T09:31:47Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet 2417608 wikitext text/x-wiki 78rgh08xsf4jwifcaf3akjnpkc5jhh3 2417609 2417608 2026-05-21T09:32:00Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet 2417609 wikitext text/x-wiki nch08ztm49vz0ot4xijk9guaxwgyul4 2417610 2417609 2026-05-21T09:32:40Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet 2417610 wikitext text/x-wiki 3950lijij7acaur5g2m3ownt3bfgve7 2417611 2417610 2026-05-21T09:33:15Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1380.eqiad.wmnet 2417611 wikitext text/x-wiki pflanvxvcar0qdzdi7eddn4w18frh6s 2417613 2417611 2026-05-21T09:33:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1381.eqiad.wmnet 2417613 wikitext text/x-wiki czfld1lzpqzijp15r0j13jp2zviiy02 2417614 2417613 2026-05-21T09:33:48Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet 2417614 wikitext text/x-wiki 7duteksqw0y51amsqzmfgtm57uudyqa 2417615 2417614 2026-05-21T09:34:52Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet 2417615 wikitext text/x-wiki 66428dwls0q020qfibkit2s1xeskm6e 2417616 2417615 2026-05-21T09:35:50Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet 2417616 wikitext text/x-wiki 9t4ek4j9xel5mm32jdt2d7366sm82ua 2417617 2417616 2026-05-21T09:35:52Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp601[1-2].drmrs.wmnet} and A:cp 2417617 wikitext text/x-wiki ffp4j5ui70u78sqb6xpmnl9ji6mzad0 2417618 2417617 2026-05-21T09:36:14Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417618 wikitext text/x-wiki b4lgzzadpguhvvj8q95xzv4cfnpcoff 2417619 2417618 2026-05-21T09:36:46Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet 2417619 wikitext text/x-wiki gq43ev6upwe4l748lz2q0dl2asnshe2 2417620 2417619 2026-05-21T09:37:28Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet 2417620 wikitext text/x-wiki 8z5zvlvuczt2122wmrrgop4vdeg5wmx 2417621 2417620 2026-05-21T09:37:30Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet 2417621 wikitext text/x-wiki lm1dehf7k0m9vh8kl2enkzxgmk0dlw4 2417622 2417621 2026-05-21T09:37:36Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet 2417622 wikitext text/x-wiki 6hcuqnl6omr6x63rx81jate1ps2vf16 2417623 2417622 2026-05-21T09:37:39Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet 2417623 wikitext text/x-wiki eymrazvk4wrdqz9bfl5v8txlu7afrq2 2417624 2417623 2026-05-21T09:37:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417624 wikitext text/x-wiki mzwg714cnncy1y0j4jrx4ts9d5fytoc 2417625 2417624 2026-05-21T09:37:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92742 and previous config saved to /var/cache/conftool/dbconfig/20260521-093754-fceratto.json 2417625 wikitext text/x-wiki 37bk3rhnhujvcvo9a2tf92nia8ortcg 2417626 2417625 2026-05-21T09:38:22Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet 2417626 wikitext text/x-wiki eqynv7b3finmw8gxzem45t9pd2duxju 2417627 2417626 2026-05-21T09:38:25Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet 2417627 wikitext text/x-wiki p4gyed7gk2uefdrvywc5e7otszrpys1 2417628 2417627 2026-05-21T09:38:29Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet 2417628 wikitext text/x-wiki 2l0d0ysld6k1w180fgbtvrwexgxbmuy 2417629 2417628 2026-05-21T09:38:31Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet 2417629 wikitext text/x-wiki q3i3dnpcxz4t3rjrumf3wxb9xgsxufv 2417630 2417629 2026-05-21T09:38:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet 2417630 wikitext text/x-wiki alx0dxbib0pf9icvq6fbdh26fu3hn5s 2417631 2417630 2026-05-21T09:38:38Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1381.eqiad.wmnet 2417631 wikitext text/x-wiki rbu2auotmyb8yl2evfjl650npxx3buk 2417632 2417631 2026-05-21T09:38:41Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1382.eqiad.wmnet 2417632 wikitext text/x-wiki q6c5hw07jat2wfpjyc2l38r1u232ft4 2417633 2417632 2026-05-21T09:39:21Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet 2417633 wikitext text/x-wiki cagebnlftn6z9y2qmcfq95d1xthuixm 2417634 2417633 2026-05-21T09:40:24Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet 2417634 wikitext text/x-wiki m35cur7dw5f5x5hucocje5hpuny3f4j 2417635 2417634 2026-05-21T09:42:52Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet 2417635 wikitext text/x-wiki n9xb3ro9ewbst37os9nmq6xyuw8781x 2417636 2417635 2026-05-21T09:44:05Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1382.eqiad.wmnet 2417636 wikitext text/x-wiki a7x16kpfkhgpkkp2fcy9gdsf24tbtny 2417637 2417636 2026-05-21T09:44:08Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2417637 wikitext text/x-wiki 9konpinyvp08twv6zrgnx3ss4ojz7un 2417638 2417637 2026-05-21T09:44:10Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1383.eqiad.wmnet 2417638 wikitext text/x-wiki kvn1cxk4vo29wev7l9vj1dv1qbfnf7k 2417639 2417638 2026-05-21T09:45:02Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet 2417639 wikitext text/x-wiki ssh1wczp1epf8vsk0k8sp3mfmr4t71k 2417640 2417639 2026-05-21T09:45:04Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet 2417640 wikitext text/x-wiki f070da6ba24ymu4lklya8qs7rree3ig 2417641 2417640 2026-05-21T09:45:07Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad 2417641 wikitext text/x-wiki 92o4z70glpz401pl0kqvvolb0224pqi 2417642 2417641 2026-05-21T09:45:38Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet 2417642 wikitext text/x-wiki 8lh8sayfmp1ppg07m42gzj42qok7g2h 2417643 2417642 2026-05-21T09:45:40Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet 2417643 wikitext text/x-wiki 4n9ynfe64ftqhxdzkqjttzxodn61tz2 2417644 2417643 2026-05-21T09:45:46Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet 2417644 wikitext text/x-wiki r1al28le7wwy3wp70k5lcgw22mtb7sb 2417645 2417644 2026-05-21T09:45:48Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet 2417645 wikitext text/x-wiki exys20r0essxiuszgmxr1tolsxw7wzh 2417646 2417645 2026-05-21T09:47:32Z Stashbot 7414 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 T426563 2417646 wikitext text/x-wiki aajdmxiuan7gtgql3vg539k54h3pki0 2417647 2417646 2026-05-21T09:47:40Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet 2417647 wikitext text/x-wiki hci05ar62q9k9anwbgft9szfhi620tk 2417648 2417647 2026-05-21T09:48:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 (T426633)', diff saved to https://phabricator.wikimedia.org/P92743 and previous config saved to /var/cache/conftool/dbconfig/20260521-094801-fceratto.json 2417648 wikitext text/x-wiki 2m1a604dhi7jv9lu0v3f6la922lm1rj 2417650 2417648 2026-05-21T09:48:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance 2417650 wikitext text/x-wiki kv98qy6faev9y7a839fb9n1ohif2uqt 2417651 2417650 2026-05-21T09:48:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet 2417651 wikitext text/x-wiki mi3yrxn3ovfe43db5fot6e5bissnz2p 2417652 2417651 2026-05-21T09:48:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1047 (T426633)', diff saved to https://phabricator.wikimedia.org/P92744 and previous config saved to /var/cache/conftool/dbconfig/20260521-094829-fceratto.json 2417652 wikitext text/x-wiki 4cmgbk908pmdnk9qrdmstu2ed50f30r 2417653 2417652 2026-05-21T09:48:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet 2417653 wikitext text/x-wiki 1d8264okgjo2oxgxrymt3j1de8ryoor 2417655 2417653 2026-05-21T09:48:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet 2417655 wikitext text/x-wiki mtcbdst2ef46csd3hkdcai6zaijr95j 2417656 2417655 2026-05-21T09:49:22Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1383.eqiad.wmnet 2417656 wikitext text/x-wiki pyhxvoejapw7oz4r30jrmlr3x41qe29 2417657 2417656 2026-05-21T09:49:24Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2417657 wikitext text/x-wiki i7m0w3rda1h0awcxiatgq290q9ppwmx 2417658 2417657 2026-05-21T09:49:26Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1384.eqiad.wmnet 2417658 wikitext text/x-wiki rffvpiukoujqkgc6qapyzylgs6cq70f 2417659 2417658 2026-05-21T09:50:50Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet 2417659 wikitext text/x-wiki lgsyz4nz6haq73bcnc3ag2brq3fh0y6 2417660 2417659 2026-05-21T09:51:47Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2417660 wikitext text/x-wiki pewe4xvs53izmwnzococw9whqslmpao 2417661 2417660 2026-05-21T09:52:12Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet 2417661 wikitext text/x-wiki hbbwqlirvt3sa3evxyb6auc59ey4o3v 2417662 2417661 2026-05-21T09:52:14Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet 2417662 wikitext text/x-wiki 1vxa08q9en4e7p0t6q7cj1ycg7yju1k 2417663 2417662 2026-05-21T09:52:17Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2417663 wikitext text/x-wiki b3786nr6bvpet9ql5z8zn6orb1y7rij 2417664 2417663 2026-05-21T09:52:19Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet 2417664 wikitext text/x-wiki 4tga14ya53ylrfsb7v0e44dor84mvgi 2417665 2417664 2026-05-21T09:52:21Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet 2417665 wikitext text/x-wiki gadqw0w4ll22lbiucr6lougo9lst2qj 2417666 2417665 2026-05-21T09:52:36Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417666 wikitext text/x-wiki 1veyu2u9kew9a5wigozyg5oyxv7ordm 2417667 2417666 2026-05-21T09:53:13Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417667 wikitext text/x-wiki fuprcmgzj217h4kjav4l3leno5d7waz 2417668 2417667 2026-05-21T09:54:20Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417668 wikitext text/x-wiki fue67c4ef5d45fy5sqn8idrs9rh2fu3 2417669 2417668 2026-05-21T09:54:25Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417669 wikitext text/x-wiki l221p3uqkfrot8bcoblfigv1bs915rb 2417670 2417669 2026-05-21T09:54:35Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org 2417670 wikitext text/x-wiki d51cugg2jphtyfi9l0r3dfyd2xddb5d 2417671 2417670 2026-05-21T09:54:44Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1384.eqiad.wmnet 2417671 wikitext text/x-wiki qdmex65kif8h8ssqjn18h0rm1b7jze3 2417672 2417671 2026-05-21T09:55:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 (T426633)', diff saved to https://phabricator.wikimedia.org/P92745 and previous config saved to /var/cache/conftool/dbconfig/20260521-095536-fceratto.json 2417672 wikitext text/x-wiki bvavrwoxwriu2zr4f5zcpdpcitlbo69 2417673 2417672 2026-05-21T09:55:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2417673 wikitext text/x-wiki mgrljjchqljb6xquju6d381rz6coyze 2417674 2417673 2026-05-21T09:55:52Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417674 wikitext text/x-wiki cetf82h8flikw96pwzv7jm95piilrak 2417675 2417674 2026-05-21T09:56:03Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417675 wikitext text/x-wiki k54y6dtu023qsoz0gojf18x00u4i1s2 2417676 2417675 2026-05-21T09:56:13Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2417676 wikitext text/x-wiki r73cm4xzi7f4740vh2tzmby7dhrng1n 2417677 2417676 2026-05-21T09:56:15Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet 2417677 wikitext text/x-wiki e4op47jrzl24zzq3h1b1ykrdrj5sala 2417678 2417677 2026-05-21T09:56:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet 2417678 wikitext text/x-wiki 0k6pzj0vi22wnhkfq13z43l2pkbdc4r 2417679 2417678 2026-05-21T09:56:28Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet 2417679 wikitext text/x-wiki tm9ibvqbcxklzpnnaz0eqr1fqopk62u 2417680 2417679 2026-05-21T09:58:55Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet 2417680 wikitext text/x-wiki j56bnhw1uz6afqoxcrxpwi57a68xhxo 2417681 2417680 2026-05-21T09:58:58Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet 2417681 wikitext text/x-wiki d8vkv00kftxefc5skacyss3j36fupqm 2417682 2417681 2026-05-21T09:59:00Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw 2417682 wikitext text/x-wiki liytu34bw631jqfpn9jzd71znrxegux 2417683 2417682 2026-05-21T09:59:40Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet 2417683 wikitext text/x-wiki 7yw7b0rabq62dov8w70uav5ivcjn1rz 2417684 2417683 2026-05-21T10:00:20Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org 2417684 wikitext text/x-wiki 9wgii8lh4el1p7hlx3n36senyvmox81 2417685 2417684 2026-05-21T10:00:33Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417685 wikitext text/x-wiki 84ijevabsfhed8oku31ngaqutqeu7jp 2417686 2417685 2026-05-21T10:01:01Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417686 wikitext text/x-wiki 292hwa6irkr22dwcej1zp37lifgvi3q 2417688 2417686 2026-05-21T10:01:05Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet 2417688 wikitext text/x-wiki nhmt4wcub3h65yjkibsiwausrgtrs8h 2417689 2417688 2026-05-21T10:01:08Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2417689 wikitext text/x-wiki m57e627cm3ihjku05gvhqquv8hzm0ig 2417691 2417689 2026-05-21T10:02:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org 2417691 wikitext text/x-wiki 0n7k89spzdbgrbqqv4axhaetow3dyi1 2417692 2417691 2026-05-21T10:03:38Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet 2417692 wikitext text/x-wiki 8xlqxle1tmlbit7mm2774yomw9nn7rj 2417693 2417692 2026-05-21T10:04:10Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet 2417693 wikitext text/x-wiki 5ppu4k7t4gzqwwxksyjzsa1anmlh5wy 2417694 2417693 2026-05-21T10:04:21Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet 2417694 wikitext text/x-wiki sy7dozr7q0be810flcdwx485iokhu7r 2417695 2417694 2026-05-21T10:04:31Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet 2417695 wikitext text/x-wiki 3cmqae3ts2q5ljo412daxvitthuezzg 2417696 2417695 2026-05-21T10:05:47Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet 2417696 wikitext text/x-wiki b29qdg0afxr8m7x66i6kf5lp9xv0x6l 2417697 2417696 2026-05-21T10:05:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92746 and previous config saved to /var/cache/conftool/dbconfig/20260521-100545-fceratto.json 2417697 wikitext text/x-wiki 8tz59o6siu7vf9q5w10hn3hgvd03jnz 2417698 2417697 2026-05-21T10:07:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet 2417698 wikitext text/x-wiki pyuxks7vqor0y5jqyv8bj9i1n35vvst 2417699 2417698 2026-05-21T10:07:58Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet 2417699 wikitext text/x-wiki 3fxtzhwmuq3d2gpi9jazhbui2060a1x 2417700 2417699 2026-05-21T10:08:04Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1013.eqiad.wmnet 2417700 wikitext text/x-wiki 3d5h3gl6vuuq4f1gbo488jyjg28a3io 2417701 2417700 2026-05-21T10:08:06Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1013.eqiad.wmnet 2417701 wikitext text/x-wiki 2wpmrxwvujh9x6lahecks1pe92hjom3 2417702 2417701 2026-05-21T10:09:09Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2417702 wikitext text/x-wiki db1ms6e5afv4ffor1xsmk3b59gt8hb9 2417703 2417702 2026-05-21T10:09:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org 2417703 wikitext text/x-wiki 95hs1q41jw5e0dggqvzaogwt6v0zfi6 2417704 2417703 2026-05-21T10:10:22Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet 2417704 wikitext text/x-wiki jw2bngw6opg05lrzjldt7zh8cxk04dp 2417706 2417704 2026-05-21T10:12:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet 2417706 wikitext text/x-wiki qj9kxvaai7s65c00nxlnblgu0yzax5i 2417707 2417706 2026-05-21T10:12:58Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp600[5-6].drmrs.wmnet} and A:cp 2417707 wikitext text/x-wiki tpk4y813bcsgswe5y5jyogrfuikf0fv 2417708 2417707 2026-05-21T10:13:00Z Stashbot 7414 moritzm: installing postgresql security updates 2417708 wikitext text/x-wiki 91bgz2q0sv9ursl1lygc4ie3azdegpw 2417709 2417708 2026-05-21T10:13:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet 2417709 wikitext text/x-wiki l3z8tdpoemtyv3lkjkzaed47jjqk0hw 2417710 2417709 2026-05-21T10:13:15Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet 2417710 wikitext text/x-wiki 0hxg3v5voepb3g585e4fth1paohr93z 2417711 2417710 2026-05-21T10:14:13Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . 2417711 wikitext text/x-wiki 3ikipsb8xvvenmnukiwi403p7fzfub0 2417712 2417711 2026-05-21T10:15:10Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417712 wikitext text/x-wiki 8j0cueg3twct47ybpgtgavrbgwowkmp 2417713 2417712 2026-05-21T10:15:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92747 and previous config saved to /var/cache/conftool/dbconfig/20260521-101552-fceratto.json 2417713 wikitext text/x-wiki dfpt9qqvmjaxfwfnzo4xuh6j275bqpw 2417714 2417713 2026-05-21T10:17:03Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet 2417714 wikitext text/x-wiki t4bk2j1i4heeomuyv7kmav7h55li4w6 2417715 2417714 2026-05-21T10:17:20Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet 2417715 wikitext text/x-wiki d4yyb6zhgrufqfuv77sh5mjk0quiwoo 2417716 2417715 2026-05-21T10:17:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417716 wikitext text/x-wiki j2xq9c42jkh2hdnsdpsepm4dmymymdc 2417720 2417716 2026-05-21T10:22:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet 2417720 wikitext text/x-wiki lto1021mar1at2pi31ivh0bl76vx7m9 2417721 2417720 2026-05-21T10:22:24Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet 2417721 wikitext text/x-wiki bjliv596osi1eogs43qme3art9d8zaw 2417722 2417721 2026-05-21T10:24:02Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet 2417722 wikitext text/x-wiki g45jclt89jcun66iniq4eh8hx428a85 2417723 2417722 2026-05-21T10:24:34Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet 2417723 wikitext text/x-wiki tibfz9gdybnniwz52k26fi8cu7ciet7 2417724 2417723 2026-05-21T10:26:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 (T426633)', diff saved to https://phabricator.wikimedia.org/P92748 and previous config saved to /var/cache/conftool/dbconfig/20260521-102601-fceratto.json 2417724 wikitext text/x-wiki hbume98o9n6v0is1ac27r5jrbv2u6gc 2417725 2417724 2026-05-21T10:26:06Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet 2417725 wikitext text/x-wiki mne3wltfc9yet2k5hjpye26ipz8vmuk 2417726 2417725 2026-05-21T10:26:24Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance 2417726 wikitext text/x-wiki 4o82lmhmnu4bkvz1r19rn76rudxr59x 2417727 2417726 2026-05-21T10:26:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2036 (T426633)', diff saved to https://phabricator.wikimedia.org/P92749 and previous config saved to /var/cache/conftool/dbconfig/20260521-102630-fceratto.json 2417727 wikitext text/x-wiki 97rcl7gjdi48vr9njx56dy9za8oocgl 2417729 2417727 2026-05-21T10:27:18Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . 2417729 wikitext text/x-wiki pdwdgrjibiuravqu86knm21rngljqot 2417730 2417729 2026-05-21T10:27:54Z Stashbot 7414 dcausse: T423993: reindexing all archive indices 2417730 wikitext text/x-wiki i0y98p6d8zebk57qmn5vhu2qoyacqx1 2417731 2417730 2026-05-21T10:29:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet 2417731 wikitext text/x-wiki pt8jg81oxbwv9ezovk22d1facqugcoz 2417732 2417731 2026-05-21T10:29:55Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2417732 wikitext text/x-wiki n2gthlnlshoatmzuejhabazu0563k6g 2417734 2417732 2026-05-21T10:34:29Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2417734 wikitext text/x-wiki 0z3yc52g5hyzwnyc4sv494sh4hi1icn 2417735 2417734 2026-05-21T10:35:12Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet 2417735 wikitext text/x-wiki 9fdbuxhd5synej3qbis9pkkeipxfzy9 2417736 2417735 2026-05-21T10:35:34Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet 2417736 wikitext text/x-wiki ax1ls8s5n246fgef4xukshxwf36k5ug 2417737 2417736 2026-05-21T10:35:37Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet 2417737 wikitext text/x-wiki rw2pcda7ek8h6wl8fqh10wlfuoiugct 2417738 2417737 2026-05-21T10:36:10Z Stashbot 7414 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1290709|ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] 2417738 wikitext text/x-wiki 9t50ygtom8pox9sb9kr4c0sr41al60i 2417739 2417738 2026-05-21T10:37:56Z Stashbot 7414 jiji@deploy1003: jiji: Backport for [[gerrit:1290709|ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417739 wikitext text/x-wiki 2ik54v28tujjt36qgnhvzp56oo80pbl 2417740 2417739 2026-05-21T10:38:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 (T426633)', diff saved to https://phabricator.wikimedia.org/P92751 and previous config saved to /var/cache/conftool/dbconfig/20260521-103759-fceratto.json 2417740 wikitext text/x-wiki haiu6hmxh929vqde1b1bycch5z5dgzp 2417741 2417740 2026-05-21T10:39:59Z Stashbot 7414 jiji@deploy1003: jiji: Continuing with deployment 2417741 wikitext text/x-wiki ml2bj7115rke5homqq46llu5spf1qj7 2417742 2417741 2026-05-21T10:40:10Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417742 wikitext text/x-wiki a7gkiqzexlr4ilrzsn6477bqpl4ltba 2417743 2417742 2026-05-21T10:40:13Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet 2417743 wikitext text/x-wiki e88fvhcj8afqvroo1jvsrmr94bphhaz 2417744 2417743 2026-05-21T10:41:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417744 wikitext text/x-wiki ocbohqohd1o1dt6dphpu6d9mrid7n8u 2417745 2417744 2026-05-21T10:43:36Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2417745 wikitext text/x-wiki q1pynhcgt0z0vz73xd1pamaqpuu3ddh 2417746 2417745 2026-05-21T10:44:12Z Stashbot 7414 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290709|ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] (duration: 08m 02s) 2417746 wikitext text/x-wiki 269ent7wz3x2vf7mikd8lv6ajeiiysc 2417747 2417746 2026-05-21T10:46:24Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet 2417747 wikitext text/x-wiki rjk1dhohnt9y2aog0lm7eirce8n2ex5 2417748 2417747 2026-05-21T10:47:02Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet 2417748 wikitext text/x-wiki 164xrdyo3zkjn3osdyjwbk4g56bxvac 2417749 2417748 2026-05-21T10:48:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92752 and previous config saved to /var/cache/conftool/dbconfig/20260521-104807-fceratto.json 2417749 wikitext text/x-wiki rqswidddk6fepgcwsa5ofz68r1mzia2 2417751 2417749 2026-05-21T10:50:06Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417751 wikitext text/x-wiki 4aq57j374hseyivvp1mj010ywcy8ymg 2417752 2417751 2026-05-21T10:50:11Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417752 wikitext text/x-wiki suuucbadbm4ini6jdt041owf6yf3kmf 2417754 2417752 2026-05-21T10:50:34Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417754 wikitext text/x-wiki qxxaihiquqe0zvs3sv9oroe6691gjan 2417755 2417754 2026-05-21T10:50:45Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417755 wikitext text/x-wiki bqtnib21fj7tpekcxcc7662zfywvkjl 2417756 2417755 2026-05-21T10:50:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet 2417756 wikitext text/x-wiki q825u4dqnugxyyxzvouzt57fb6hdyzb 2417757 2417756 2026-05-21T10:51:10Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417757 wikitext text/x-wiki rvxocl93c68bly7tyn31ogyhqfjt87s 2417758 2417757 2026-05-21T10:51:18Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2417758 wikitext text/x-wiki e774aup7cgz1nrxcwu00ut4uaknqmke 2417759 2417758 2026-05-21T10:51:39Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2417759 wikitext text/x-wiki j6x09mrnsvu6dkk1681kl05o19u3n1u 2417760 2417759 2026-05-21T10:53:28Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet 2417760 wikitext text/x-wiki t85j6gvqxd60l462w2y5w7kqvwicekr 2417761 2417760 2026-05-21T10:54:03Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-codfw 2417761 wikitext text/x-wiki m4c51imfpvz2xz7y3xbgfff47xkjkxi 2417764 2417761 2026-05-21T10:55:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage 2417764 wikitext text/x-wiki 0rca253gb2bnagtt634jzrzk22vp4ku 2417767 2417764 2026-05-21T10:56:29Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet 2417767 wikitext text/x-wiki fpp6kwqtfu13fm3ot5q7tw7zdc1zn1j 2417768 2417767 2026-05-21T10:56:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet 2417768 wikitext text/x-wiki 4x8tztik5fhhhsnfdu7pgioc5jvh6m3 2417770 2417768 2026-05-21T10:57:27Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet 2417770 wikitext text/x-wiki koe42cjkzriy8egi5ztmqde5v2t7uga 2417772 2417770 2026-05-21T10:58:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92753 and previous config saved to /var/cache/conftool/dbconfig/20260521-105815-fceratto.json 2417772 wikitext text/x-wiki 1my30z4d2o5ymljhm9zpm5omekhn5ba 2417773 2417772 2026-05-21T10:59:04Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage 2417773 wikitext text/x-wiki m8erhfw1cfis91ihpayirmho2r8visz 2417775 2417773 2026-05-21T11:00:04Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet 2417775 wikitext text/x-wiki a272lna8sdu8oko2yzoydt1em16wh27 2417776 2417775 2026-05-21T11:02:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-codfw 2417776 wikitext text/x-wiki jdki1wzruduf0t7k1bz26wo9jpni1ml 2417777 2417776 2026-05-21T11:04:50Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet 2417777 wikitext text/x-wiki dyzbbf4pginyq6a8k1jrnqisdczaqce 2417778 2417777 2026-05-21T11:04:53Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp600[5-6].drmrs.wmnet} and A:cp 2417778 wikitext text/x-wiki 3dvqhtbn6zhlfce8m8s2m1t4zx7g05f 2417779 2417778 2026-05-21T11:05:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet 2417779 wikitext text/x-wiki ahbhcyun1stsmjxzms933rh12lnhyw3 2417780 2417779 2026-05-21T11:05:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-eqiad 2417780 wikitext text/x-wiki pf6ykg9i962hqy1s4trugmcdaefx60s 2417781 2417780 2026-05-21T11:06:07Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet 2417781 wikitext text/x-wiki hdwqjhm30p92fvqc8i12r0ujy1rtsy7 2417782 2417781 2026-05-21T11:06:14Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet 2417782 wikitext text/x-wiki g7qvpvzdp04lmxzwvblk08vb72pz6ke 2417783 2417782 2026-05-21T11:08:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 (T426633)', diff saved to https://phabricator.wikimedia.org/P92756 and previous config saved to /var/cache/conftool/dbconfig/20260521-110822-fceratto.json 2417783 wikitext text/x-wiki tfuk4a3zwextk0pc4bqqs7szagh2yai 2417784 2417783 2026-05-21T11:08:46Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2037.codfw.wmnet with reason: Maintenance 2417784 wikitext text/x-wiki pm8n8sdpneshd4ax2bptp7ojggmldsx 2417785 2417784 2026-05-21T11:08:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2037 (T426633)', diff saved to https://phabricator.wikimedia.org/P92757 and previous config saved to /var/cache/conftool/dbconfig/20260521-110851-fceratto.json 2417785 wikitext text/x-wiki skh0bl8a25gav595gjuv8tqanx376tj 2417786 2417785 2026-05-21T11:09:39Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp601[3-4].drmrs.wmnet} and A:cp 2417786 wikitext text/x-wiki s78vqelqjq4gesux9ikpwkhw6d0pti0 2417787 2417786 2026-05-21T11:11:09Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet 2417787 wikitext text/x-wiki iggv6k2af6j294a5nlxvs47w2pfaw9n 2417788 2417787 2026-05-21T11:12:20Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet 2417788 wikitext text/x-wiki hywo2owf8x5i7izcpfss7on384fkohi 2417789 2417788 2026-05-21T11:13:59Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet with OS trixie 2417789 wikitext text/x-wiki tngkurmaa5zit1ae5zvubf5qxyfxhzi 2417790 2417789 2026-05-21T11:14:05Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-eqiad 2417790 wikitext text/x-wiki rrchz3qieerdetlriuq4rfmg3izq385 2417791 2417790 2026-05-21T11:18:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet 2417791 wikitext text/x-wiki dlgykt9vhssln4uidt1rrhvea6ormyo 2417792 2417791 2026-05-21T11:20:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 (T426633)', diff saved to https://phabricator.wikimedia.org/P92758 and previous config saved to /var/cache/conftool/dbconfig/20260521-112021-fceratto.json 2417792 wikitext text/x-wiki 983keiu23fmxcmt8ymr6iha40p19i0t 2417793 2417792 2026-05-21T11:20:45Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet 2417793 wikitext text/x-wiki ckupmn63dysrgvrjmuo88n9saeswhdp 2417794 2417793 2026-05-21T11:20:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet 2417794 wikitext text/x-wiki 0a34jsqrnopa7wy2fedca4hd965u8rq 2417795 2417794 2026-05-21T11:24:17Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet 2417795 wikitext text/x-wiki ky6qlgu9wgeiqfmfzpjl20vbcjrgzgo 2417796 2417795 2026-05-21T11:24:50Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet 2417796 wikitext text/x-wiki tw0mx6bqhe6kd6b5btmowr1cdthfex7 2417797 2417796 2026-05-21T11:24:56Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet 2417797 wikitext text/x-wiki b9t1hnqb1nvvpfwdk1wje1oecb7n0wv 2417798 2417797 2026-05-21T11:26:51Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet 2417798 wikitext text/x-wiki 509qam8w3q3e7xfy90pranwzvaltvyy 2417799 2417798 2026-05-21T11:26:57Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet 2417799 wikitext text/x-wiki nfgtth20u665ptxaekcihq3ops3k8sy 2417800 2417799 2026-05-21T11:27:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet 2417800 wikitext text/x-wiki 1zbwyasdum8gkr1fcrf8jqa2vvcncou 2417801 2417800 2026-05-21T11:27:24Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet 2417801 wikitext text/x-wiki or8j28y0qobl8r4pglfrybnrx2dmrck 2417802 2417801 2026-05-21T11:30:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92759 and previous config saved to /var/cache/conftool/dbconfig/20260521-113028-fceratto.json 2417802 wikitext text/x-wiki 98xvngwkax17tf13fyzn48hvb3nclp3 2417803 2417802 2026-05-21T11:30:58Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet 2417803 wikitext text/x-wiki 5w54rpw20auq9abjde5ebsqrsiykfkn 2417804 2417803 2026-05-21T11:31:39Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet 2417804 wikitext text/x-wiki 1xzlrscs8v8okb5vd02wg0qt6efzngm 2417805 2417804 2026-05-21T11:32:33Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet 2417805 wikitext text/x-wiki 5p1oqmuwe4cst6acwy4pp6hnqrlkn7l 2417806 2417805 2026-05-21T11:33:01Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet 2417806 wikitext text/x-wiki 5d1h5ykac86yd5z915ovlks61hhkukm 2417807 2417806 2026-05-21T11:34:47Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet 2417807 wikitext text/x-wiki 5o79v0cetegql0jw0wa73vr1k21y6pm 2417808 2417807 2026-05-21T11:35:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet 2417808 wikitext text/x-wiki 5gudemdw8d6cveig3vlemu9e55yfge5 2417809 2417808 2026-05-21T11:35:06Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet 2417809 wikitext text/x-wiki ec5yr3j6qzboah698z8woqk8gs9hcjr 2417810 2417809 2026-05-21T11:35:15Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker 2417810 wikitext text/x-wiki rwiuqpwhx03or2ldfh48u41m1q7o4fv 2417811 2417810 2026-05-21T11:35:17Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet 2417811 wikitext text/x-wiki sy753waf8bv3zxu9qb96t6sxje9pcxp 2417812 2417811 2026-05-21T11:35:34Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet 2417812 wikitext text/x-wiki o1qsczh20r02aclcwyhghigo0vjevrq 2417813 2417812 2026-05-21T11:36:55Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet 2417813 wikitext text/x-wiki oscwmv8wlk01bb292jkh1jb1wgju8pe 2417814 2417813 2026-05-21T11:36:59Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet 2417814 wikitext text/x-wiki ft1v91og2usaq1td5jfxqyle6pwg9w0 2417815 2417814 2026-05-21T11:37:21Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet 2417815 wikitext text/x-wiki 5a2vcm326suqsupyn3e6t6f6xtjpkez 2417816 2417815 2026-05-21T11:38:41Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet 2417816 wikitext text/x-wiki t87au7c91p4cifu39jdr6t2yu5jd6qz 2417817 2417816 2026-05-21T11:39:06Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet 2417817 wikitext text/x-wiki c0mqqc9ecu4vbt4yb999vk7f7eg1i9h 2417818 2417817 2026-05-21T11:39:12Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet 2417818 wikitext text/x-wiki c4wu7y70hxcwbytlpcb3vhlec4s1f6w 2417819 2417818 2026-05-21T11:40:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92760 and previous config saved to /var/cache/conftool/dbconfig/20260521-114036-fceratto.json 2417819 wikitext text/x-wiki siyyl19py9yoep97gb380lw7up7smoq 2417820 2417819 2026-05-21T11:40:49Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet 2417820 wikitext text/x-wiki 0mcmgc0ljmba7q192l0rbuwb9rk9ihz 2417821 2417820 2026-05-21T11:42:16Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet 2417821 wikitext text/x-wiki mgztbn96u8bqxfauk0yrdgct8r7dv6q 2417822 2417821 2026-05-21T11:43:58Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org 2417822 wikitext text/x-wiki gx91qphrtrv2gxnp7lrjhiph4to1dze 2417823 2417822 2026-05-21T11:44:01Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet 2417823 wikitext text/x-wiki 282h67bz5yv526qewsjpic7fs6v6q4d 2417824 2417823 2026-05-21T11:44:38Z Stashbot 7414 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2417824 wikitext text/x-wiki 8knf0qos6035eh30227nyno43dnp9mg 2417825 2417824 2026-05-21T11:45:14Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet 2417825 wikitext text/x-wiki 2x88k89p4mafaimni2gb0h8dc2xae1e 2417826 2417825 2026-05-21T11:45:22Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet 2417826 wikitext text/x-wiki k2nt7f22igq0dco179kxffnc85rcd0r 2417827 2417826 2026-05-21T11:49:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet 2417827 wikitext text/x-wiki 3zvrh5m8mymd1io22d4fk0gcmm0z45t 2417828 2417827 2026-05-21T11:49:51Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org 2417828 wikitext text/x-wiki gfz5a6ib9332ckse1g7aj4bdmvyafhc 2417829 2417828 2026-05-21T11:49:58Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet 2417829 wikitext text/x-wiki e1bzxhj0pap9pen2gw57wix29m7281y 2417830 2417829 2026-05-21T11:50:40Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet 2417830 wikitext text/x-wiki j1jpsxdd45zins8yqh3zgz0pqdrsebj 2417831 2417830 2026-05-21T11:50:42Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet 2417831 wikitext text/x-wiki czjy0onpehanl5ui5t7lu9grwm1836m 2417832 2417831 2026-05-21T11:50:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 (T426633)', diff saved to https://phabricator.wikimedia.org/P92761 and previous config saved to /var/cache/conftool/dbconfig/20260521-115043-fceratto.json 2417832 wikitext text/x-wiki m8e4umel3730vo907h5fak0cg8hryh2 2417833 2417832 2026-05-21T11:50:48Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet 2417833 wikitext text/x-wiki 433rjkoesj5uag2cab93h4fgatu8gcq 2417834 2417833 2026-05-21T11:50:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet 2417834 wikitext text/x-wiki l4tb2flu8gcf39m8n5bvzr56cr5ub0z 2417835 2417834 2026-05-21T11:51:07Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2047.codfw.wmnet with reason: Maintenance 2417835 wikitext text/x-wiki tj28dc1y2h6dyy4wocqp3m331cpdhkx 2417836 2417835 2026-05-21T11:51:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2047 (T426633)', diff saved to https://phabricator.wikimedia.org/P92762 and previous config saved to /var/cache/conftool/dbconfig/20260521-115112-fceratto.json 2417836 wikitext text/x-wiki bdqrzvzf9imng5lvzvlnofps07g2zkl 2417837 2417836 2026-05-21T11:51:29Z Stashbot 7414 taavi: disabling puppet on C:bird to roll out 1289919 2417837 wikitext text/x-wiki f31d6eq5h17jng287wfdwy4349hj64q 2417838 2417837 2026-05-21T11:53:05Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org 2417838 wikitext text/x-wiki q4d79hqwelbga9d93cjedbfmq4ah6u0 2417840 2417838 2026-05-21T11:57:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet 2417840 wikitext text/x-wiki ledizh1u8t3bt13s5ubptc80h88ga82 2417841 2417840 2026-05-21T11:58:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 (T426633)', diff saved to https://phabricator.wikimedia.org/P92763 and previous config saved to /var/cache/conftool/dbconfig/20260521-115817-fceratto.json 2417841 wikitext text/x-wiki ite09kkdeu10h8jsfxzn98rdb6u88uq 2417843 2417841 2026-05-21T11:58:46Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org 2417843 wikitext text/x-wiki 7dke6h4x2ci3kemagv1n9cl2i2k2cya 498 30942 2416792 2416726 2026-05-20T12:41:17Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416792 wikitext text/x-wiki 9ku06qwcq826oxtyz2caewxyxg0b3j5 2416793 2416792 2026-05-20T12:41:21Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416793 wikitext text/x-wiki j8ln1ydzxmv189e1higpo3azbvw2hgj 2416796 2416793 2026-05-20T12:42:35Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416796 wikitext text/x-wiki qgiarr4xgutnv083ahonivu8l4o1246 2416802 2416796 2026-05-20T12:43:12Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416802 wikitext text/x-wiki sms4omp9jg4epnl6wotwonybi30grah 2416925 2416802 2026-05-20T13:23:55Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.88}' 2416925 wikitext text/x-wiki rn060g3gfqkqtthrv84wfmwsrp47w4i 2417839 2416925 2026-05-21T11:56:41Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'D{cloudlb2002-dev.codfw.wmnet}' 2417839 wikitext text/x-wiki dm724k7o48bqrwejsb9olbghtlnmpnl 0 43014 2417259 2334955 2026-05-20T16:32:20Z TChin (WMF) 25137 2417259 wikitext text/x-wiki fvay2g98e47jivtcph0r2mvd9ms3zdh 498 443946 2417349 2416121 2026-05-20T18:46:31Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed a42e65b596 (configure Gunicorn --forwarded-allow-ips) 2417349 wikitext text/x-wiki i6tqngjjgt28dczqf6i8vudkv7vvhf3 2417353 2417349 2026-05-20T18:50:55Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 5b92917005 (upgrade dependencies) 2417353 wikitext text/x-wiki qt7qlo6iq93ynpaelf2xwm2objjimos 2417356 2417353 2026-05-20T18:55:26Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed dfc101f3e0 (Python 3.14, aka 𝜋thon) 2417356 wikitext text/x-wiki df5xasymiw835nd176fbuifrwtbe4ow 0 444245 2416787 2369670 2026-05-20T12:38:42Z AYounsi (WMF) 10040 2416787 wikitext text/x-wiki pk5ag9hxhv0q96w6yyqaeibwcmo92sm 2416805 2416787 2026-05-20T12:44:13Z AYounsi (WMF) 10040 /* Upgrade */ 2416805 wikitext text/x-wiki 6ok66av2zmcwan0nxygrq234aw9krph 2416929 2416805 2026-05-20T13:25:34Z AYounsi (WMF) 10040 /* Preparation */ 2416929 wikitext text/x-wiki srg9qigrrui66gqhp83hwq5flr405ao 0 446120 2417338 2301296 2026-05-20T18:09:46Z CMooney (WMF) 24288 /* Cleanup */ 2417338 wikitext text/x-wiki kb9bcxlaqbx3x8fvq1xmab4inaib5of 2417339 2417338 2026-05-20T18:10:11Z CMooney (WMF) 24288 /* Cleanup */ 2417339 wikitext text/x-wiki 04wolx6tcjl480qv8ritpda8xhwd02o 0 446977 2417243 2401888 2026-05-20T16:12:55Z APaskulin (WMF) 14880 /* Long-term access */ Add database dump link 2417243 wikitext text/x-wiki nvgkwz7iaodex604q927bu3jxw5t2ux 0 447017 2417391 2409217 2026-05-20T20:17:16Z RLazarus (WMF) 15215 /* Published SLOs */ 2417391 wikitext text/x-wiki cka9c2z3qpgjjuuqwcj2vujybwj6k0o 0 449160 2417480 2416312 2026-05-21T00:01:53Z Dexbot 30554 Bot: Updating the report 2417480 wikitext text/x-wiki 0vbcgsgspzrmvfgbqbvvca5a45c0ij3 2417495 2417480 2026-05-21T05:32:34Z Dexbot 30554 Bot: Updating the report 2417495 wikitext text/x-wiki c3ghsr6tux5nasdjsq1iscijcfhajjv 2417544 2417495 2026-05-21T08:17:55Z Dexbot 30554 Bot: Updating the report 2417544 wikitext text/x-wiki rddduzoisv4xc136kh0pb2ywhkfaugw 2417728 2417544 2026-05-21T10:27:16Z Dexbot 30554 Bot: Updating the report 2417728 wikitext text/x-wiki djxuyjszsur2x7cp69xaudflw90g2k6 0 451075 2417232 2412626 2026-05-20T16:09:11Z DAdedoyin (WMF) 39150 /* 2026-04-21 - 2026-05-18 */ 2417232 wikitext text/x-wiki ef5s0byr3q1dqlt017ouu1ag8tl2j43 2417233 2417232 2026-05-20T16:09:32Z DAdedoyin (WMF) 39150 /* Current */ 2417233 wikitext text/x-wiki fznb7cnthnbsf56j28ry48ct9qfxkuu 0 452392 2416754 2178858 2026-05-20T12:16:25Z ABran-WMF 37196 2416754 wikitext text/x-wiki efs2dwcoexv0irxepjlpm0ycl680vq8 2 457956 2417717 2329161 2026-05-21T10:21:15Z GModena (WMF) 20194 2417717 wikitext text/x-wiki fzi1tm04f0tw03nsd2d4torb8iizx0q 2417718 2417717 2026-05-21T10:21:34Z GModena (WMF) 20194 2417718 wikitext text/x-wiki 94ip1lpfan8jpnkoistggfleti6lfqg 2417719 2417718 2026-05-21T10:21:56Z GModena (WMF) 20194 2417719 wikitext text/x-wiki jm255o6sopm3lp9c4bwlqjf1a3agc1z 0 458670 2416869 2328741 2026-05-20T13:07:19Z CMooney (WMF) 24288 /* Additional Information */ 2416869 wikitext text/x-wiki g5vjfcogxwcz2jgmmr8wy21sgf0wl4r 0 459043 2417150 2377414 2026-05-20T15:33:51Z AStein-WMF 39485 2417150 wikitext text/x-wiki l34eeobvg128zi00ne30fhrmpdye1os 0 459138 2417149 2397068 2026-05-20T15:32:54Z WBrown (WMF) 37696 /* Client Side */ add client side info 2417149 wikitext text/x-wiki dk0e25q25eogc4agi7umb7wj5b5af8l 2417194 2417149 2026-05-20T15:49:57Z WBrown (WMF) 37696 /* Mediawiki */ update 2417194 wikitext text/x-wiki eugoaxy2ovrb3qegtybzjvun2msdzwv 2417207 2417194 2026-05-20T15:53:45Z WBrown (WMF) 37696 update status 2417207 wikitext text/x-wiki 07gism4zwocwizg1fg0duly1ufw92wh 2417221 2417207 2026-05-20T16:03:55Z WBrown (WMF) 37696 /* mediawiki-config */ + 2417221 wikitext text/x-wiki edqk5r6xukvkjysyannpclnm2adyd2y 2417222 2417221 2026-05-20T16:04:23Z WBrown (WMF) 37696 /* mediawiki-config */ + 2417222 wikitext text/x-wiki kzfadx7kodegyov41pvrvd0lokvoriu 2417226 2417222 2026-05-20T16:06:55Z WBrown (WMF) 37696 /* Completely disable hCaptcha */ update 2417226 wikitext text/x-wiki 11z8vns5antmaje4kwbra5v1te9ws3n 2417227 2417226 2026-05-20T16:07:10Z WBrown (WMF) 37696 /* Contact information */ + 2417227 wikitext text/x-wiki ob9imd7wne7gaubkmp17p41bkszqpqd 0 459160 2417162 2336322 2026-05-20T15:39:58Z DAdedoyin (WMF) 39150 First draft of the Donor Portal documentation 2417162 wikitext text/x-wiki nkr2xjdtv0qwgdus10epl5hnm890h8u 2417170 2417162 2026-05-20T15:44:24Z DAdedoyin (WMF) 39150 /* Donor Portal — Internal Reference Guide */ Remove unnecessary starting paragraph. 2417170 wikitext text/x-wiki h9rimhdve69uw5khmz1dystrixiy2hg 0 459290 2417541 2414716 2026-05-21T08:06:08Z KCVelaga 26729 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ +change 1288444 2417541 wikitext text/x-wiki g2d3psjlagvlwsz57kiiajaofyko3ko 0 459699 2417454 2416301 2026-05-20T22:22:30Z Stashbot 7414 eileen: SmashPig upgraded from df441f78 to 252b6cfa 2417454 wikitext text/x-wiki 92pv3z1jt3gyvjetu4rqyp1nk2ez1ft 2417482 2417454 2026-05-21T00:09:17Z Stashbot 7414 cstone: civicrm upgraded from fcbbf763 to cae2ab0e 2417482 wikitext text/x-wiki 5a584lmka271eijd9quom7u6ddab7vd 2417491 2417482 2026-05-21T04:13:14Z Stashbot 7414 eileen: civicrm upgraded from cae2ab0e to dbafc0b4 2417491 wikitext text/x-wiki 7tams3byzvft6vfa34thyzu08zpx6gt 2 460097 2417733 2416654 2026-05-21T10:30:45Z Effie Mouzeli (WMF) 12880 Replaced content with "moved" 2417733 wikitext text/x-wiki azmbrr0lpsx25wmujpcwi6uqyuq9udo 2 460100 2416847 2415700 2026-05-20T13:00:04Z Effie Mouzeli (WMF) 12880 2416847 wikitext text/x-wiki jc1hffngk13cemssoxajn9rnkzyl8if 2 460176 2416845 2415715 2026-05-20T12:59:50Z Effie Mouzeli (WMF) 12880 2416845 wikitext text/x-wiki ffb014mghc07p43afzlklfqlxppod1f 0 460177 2417430 2414256 2026-05-20T21:24:38Z Effie Mouzeli (WMF) 12880 Effie Mouzeli (WMF) moved page [[SCROLL/Ducks]] to [[SCROLL/Another Duck]] 2414256 wikitext text/x-wiki kv7t8rszizt6di5uh4a3ywpqiqwdyrl 0 460178 2416789 2415702 2026-05-20T12:39:14Z Effie Mouzeli (WMF) 12880 2416789 wikitext text/x-wiki 5l8vf814twnvxxaq12fbe1ddfvgg33o 14 460180 2416804 2414279 2026-05-20T12:43:47Z Effie Mouzeli (WMF) 12880 2416804 wikitext text/x-wiki salxv4h617shn97ohr1lhgi1l25eeu4 2416855 2416804 2026-05-20T13:01:53Z Effie Mouzeli (WMF) 12880 2416855 wikitext text/x-wiki 5qdpzubmd7ycsekbhokxa5hmi0ab9ff 2416867 2416855 2026-05-20T13:06:14Z Effie Mouzeli (WMF) 12880 2416867 wikitext text/x-wiki 9yoa02xgiutqsip3eemwrkcd4pcp6j3 2417001 2416867 2026-05-20T14:08:57Z Effie Mouzeli (WMF) 12880 Replaced content with "nothing" 2417001 wikitext text/x-wiki 1uyr6jbis31k9f4l753jmdkvcmcof81 14 460181 2417004 2414312 2026-05-20T14:10:18Z Effie Mouzeli (WMF) 12880 2417004 wikitext text/x-wiki 1uyr6jbis31k9f4l753jmdkvcmcof81 2417011 2417004 2026-05-20T14:12:51Z Effie Mouzeli (WMF) 12880 2417011 wikitext text/x-wiki cwjz8ypmuw96etnfftmgoi8lk4o4i4x 0 460191 2416844 2416693 2026-05-20T12:59:27Z Effie Mouzeli (WMF) 12880 2416844 wikitext text/x-wiki ibkspsa6g8t9u7dkvmv6huo5e2e2l2a 2417033 2416844 2026-05-20T14:21:52Z Effie Mouzeli (WMF) 12880 2417033 wikitext text/x-wiki 0vuq0knigf2wucl1c3m6trkyvara3xf 2417055 2417033 2026-05-20T14:38:14Z Effie Mouzeli (WMF) 12880 2417055 wikitext text/x-wiki p9dg6aw2dcvriyhund80rup6t0qn7dv 2417074 2417055 2026-05-20T14:46:16Z Effie Mouzeli (WMF) 12880 2417074 wikitext text/x-wiki g1d4v1x4y4ezww2hvpro4qa0w1vspba 2417084 2417074 2026-05-20T14:50:14Z Effie Mouzeli (WMF) 12880 2417084 wikitext text/x-wiki gp6appy7tsqt7okcicq6pcry1fj24vl 2417120 2417084 2026-05-20T15:13:59Z Effie Mouzeli (WMF) 12880 2417120 wikitext text/x-wiki 4lqqxwuiti64g2fzm8gwcgbw7x2c27g 2417121 2417120 2026-05-20T15:14:24Z Effie Mouzeli (WMF) 12880 2417121 wikitext text/x-wiki 04b7pou030f1txnqiwbo4im0upi3xep 2417164 2417121 2026-05-20T15:41:48Z Effie Mouzeli (WMF) 12880 2417164 wikitext text/x-wiki o3imtlfazrwftgekp12dw05w90c3mdp 2417206 2417164 2026-05-20T15:53:07Z Effie Mouzeli (WMF) 12880 2417206 wikitext text/x-wiki 6hocp26x78pipzg26w6b52tx4x3yuro 2417208 2417206 2026-05-20T15:53:49Z Effie Mouzeli (WMF) 12880 2417208 wikitext text/x-wiki 6vrlg7sft3uqt6gc7q2p73ot8nd4drz 0 460192 2416860 2416668 2026-05-20T13:03:41Z Effie Mouzeli (WMF) 12880 2416860 wikitext text/x-wiki e1627c5b457bm3qtaigb6x6ikzhlu2v 2416863 2416860 2026-05-20T13:04:20Z Effie Mouzeli (WMF) 12880 2416863 wikitext text/x-wiki qy07mfcb2bqo7f7b9hhgz6jf76ks730 2416865 2416863 2026-05-20T13:05:07Z Effie Mouzeli (WMF) 12880 2416865 wikitext text/x-wiki mjhggc24kau4oruv6jmboazunpf9mst 2417075 2416865 2026-05-20T14:46:44Z Effie Mouzeli (WMF) 12880 2417075 wikitext text/x-wiki sgtfto26ld3a44g8oc0jjxv2fb7ng1l 0 460194 2416810 2416679 2026-05-20T12:46:34Z Effie Mouzeli (WMF) 12880 2416810 wikitext text/x-wiki gbhzvvrllcbf2by5iffdl24rn777kqv 2416835 2416810 2026-05-20T12:57:59Z Effie Mouzeli (WMF) 12880 2416835 wikitext text/x-wiki 97hfrn3zp4399vei3n1t94oabypqe4p 2417023 2416835 2026-05-20T14:18:21Z Effie Mouzeli (WMF) 12880 Effie Mouzeli (WMF) moved page [[SCROLL/Guide]] to [[SCROLL/Docs/Guide]] without leaving a redirect: Poorly named 2416835 wikitext text/x-wiki 97hfrn3zp4399vei3n1t94oabypqe4p 2417029 2417023 2026-05-20T14:20:02Z Effie Mouzeli (WMF) 12880 Effie Mouzeli (WMF) moved page [[SCROLL/Docs/Guide]] to [[SCROLL/Docs//Guide]] without leaving a redirect 2416835 wikitext text/x-wiki 97hfrn3zp4399vei3n1t94oabypqe4p 2417030 2417029 2026-05-20T14:20:19Z Effie Mouzeli (WMF) 12880 Effie Mouzeli (WMF) moved page [[SCROLL/Docs//Guide]] to [[SCROLL/Docs/Guide]] without leaving a redirect 2416835 wikitext text/x-wiki 97hfrn3zp4399vei3n1t94oabypqe4p 2417037 2417030 2026-05-20T14:24:03Z Effie Mouzeli (WMF) 12880 2417037 wikitext text/x-wiki n8278z0z25ox2i48xqb0lgm9e5tblht 2417160 2417037 2026-05-20T15:39:30Z Effie Mouzeli (WMF) 12880 2417160 wikitext text/x-wiki te9lraha2xawqnlwwitg7gw96vzj4sx 0 460195 2416750 2026-05-20T12:14:42Z AYounsi (WMF) 10040 Initial page creation 2416750 wikitext text/x-wiki 6qe1jvk41mykx98p1qfp2jbjcqxertt 2416756 2416750 2026-05-20T12:16:58Z AYounsi (WMF) 10040 /* Cleanup */ 2416756 wikitext text/x-wiki kpom36xxqc5lbodq0evh8eddu9z6otl 2416788 2416756 2026-05-20T12:39:06Z AYounsi (WMF) 10040 /* Maintenance */ 2416788 wikitext text/x-wiki ia1b0b0uqu8o7hl0zfu7xv7lr9g10z4 2416912 2416788 2026-05-20T13:21:03Z CMooney (WMF) 24288 /* Maintenance */ 2416912 wikitext text/x-wiki inb8ngfjbyzu206ddn3yen891t76mjm 0 460196 2416801 2026-05-20T12:43:12Z CMooney (WMF) 24288 Created page with "== Download the Nokia SR-Linux Image == This can be done through the [https://customer.nokia.com/support/s/ Nokia Customer Portal], or if you have SFTP access set up with Nokia via SFTP. There is a single version for all variants of the 7220-IXR platform, the 'HARDWARE' zip file is the one that contains the OS image. == Verify Image == Unzip the file and verify the MD5 hash for it cmooney@wikilap:~$ unzip Nokia-7220_IXR_SR_Linux-HARDWARE-26.3.2.zip Archive:..." 2416801 wikitext text/x-wiki t4w8odb4hrjwbw69ncljqciclxxzy2u 2416811 2416801 2026-05-20T12:47:23Z CMooney (WMF) 24288 /* Verify Image */ 2416811 wikitext text/x-wiki jfqwb63y5ci8s6pz79aut78i6j04y6s 2416825 2416811 2026-05-20T12:55:38Z CMooney (WMF) 24288 /* Place it on our apt servers */ 2416825 wikitext text/x-wiki p0talp6gkpj7x292s1zx8x6o516tw7b 2416831 2416825 2026-05-20T12:57:19Z CMooney (WMF) 24288 /* Download the Nokia SR-Linux Image */ 2416831 wikitext text/x-wiki 8kxr78ha0jp6t7rahbjpj2wl4b4j454 2416834 2416831 2026-05-20T12:57:56Z CMooney (WMF) 24288 /* Download the Nokia SR-Linux Image */ 2416834 wikitext text/x-wiki 3scsa26enrmgv8alqnb0aeasy6u44ci 2416838 2416834 2026-05-20T12:58:12Z CMooney (WMF) 24288 /* Check routing protocol status before commencing updgrade */ 2416838 wikitext text/x-wiki 1mm4qq0ht6ktwehtu0a99qykcsevtek 2416843 2416838 2026-05-20T12:59:06Z CMooney (WMF) 24288 /* Upgrade the switch */ 2416843 wikitext text/x-wiki 079q3a57skss3vull5vt26h9e8p012e 2416849 2416843 2026-05-20T13:01:23Z CMooney (WMF) 24288 /* Download the Nokia SR-Linux Image */ 2416849 wikitext text/x-wiki 9ok7be5pst4dd9gx9mzv5bk1m2forym 2416862 2416849 2026-05-20T13:04:18Z CMooney (WMF) 24288 /* Download the Nokia SR-Linux Image */ 2416862 wikitext text/x-wiki 7up30xg3nu4v0l0pkemcoliglbkhxyl 2416870 2416862 2026-05-20T13:07:40Z CMooney (WMF) 24288 /* Upgrade the switch */ 2416870 wikitext text/x-wiki 4ij8dclz6a4i99c7347brq63rqhgm0w 2416877 2416870 2026-05-20T13:13:08Z CMooney (WMF) 24288 /* Versions */ 2416877 wikitext text/x-wiki bceml21bqnhm0gtbznt44jand8go5c1 2416878 2416877 2026-05-20T13:13:33Z CMooney (WMF) 24288 /* Versions */ 2416878 wikitext text/x-wiki ghjryeqtdcujgmmxeeb7ufa5irb7ucj 2416879 2416878 2026-05-20T13:14:03Z CMooney (WMF) 24288 /* Versions */ 2416879 wikitext text/x-wiki prio8ndx7luef0j3o6cybzacr91fkc7 2416880 2416879 2026-05-20T13:14:26Z CMooney (WMF) 24288 /* Versions */ 2416880 wikitext text/x-wiki 7ptlflih48wjc19mklgswfucv74g62m 2416886 2416880 2026-05-20T13:16:14Z CMooney (WMF) 24288 /* Versions */ 2416886 wikitext text/x-wiki o3q2rehhqlvc45wcssw4nw3q11oaiui 2416893 2416886 2026-05-20T13:18:01Z CMooney (WMF) 24288 /* Check routing protocol status before commencing updgrade */ 2416893 wikitext text/x-wiki bxwnl1dins6s7wle9n7z6enzhvbyiax 2416896 2416893 2026-05-20T13:18:24Z CMooney (WMF) 24288 /* Check routing protocols after upgrade */ 2416896 wikitext text/x-wiki tk2jwggew59jkgmq571fi1r4dy1tzwr 2416898 2416896 2026-05-20T13:18:33Z CMooney (WMF) 24288 /* Upgrade the switch */ 2416898 wikitext text/x-wiki a2ugb17xjf7jmpnwfnnb7l8mkybc3fd 2416909 2416898 2026-05-20T13:20:28Z CMooney (WMF) 24288 CMooney (WMF) moved page [[Nokia switch upgrade]] to [[Nokia Networking/Nokia switch upgrade]] 2416898 wikitext text/x-wiki a2ugb17xjf7jmpnwfnnb7l8mkybc3fd 2416974 2416909 2026-05-20T13:52:03Z CMooney (WMF) 24288 /* Download the Nokia SR-Linux Image */ 2416974 wikitext text/x-wiki i6qx5xhpk9q8w7cfh034486l8p9l5ku 2416977 2416974 2026-05-20T13:53:50Z CMooney (WMF) 24288 /* Check port & routing protocol status before commencing updgrade */ 2416977 wikitext text/x-wiki fbvthbbo9dq13bb8io5yqb5yu4c9893 0 460197 2416813 2026-05-20T12:48:06Z Effie Mouzeli (WMF) 12880 Created page with "{| class="wikitable" style="float: right; clear: right; width: 320px; margin: 0 0 1.2em 1.5em; font-size: 88%; border: 1px solid #a2a9b1;" |- ! colspan="2" style="background: #d5e8df; text-align: center; padding: 10px; font-size: 115%;" | Duck |- | style="background: #f8f9fa; width: 40%; padding: 4px 8px;" | '''Status''' | style="padding: 4px 8px;" | Production |- | style="background: #f8f9fa; padding: 4px 8px;" | '''Type''' | style="padding: 4px 8px;" | Service |- | sty..." 2416813 wikitext text/x-wiki eamcw70bi815kc77iutvk70ks6lm2zz 2416818 2416813 2026-05-20T12:50:42Z Effie Mouzeli (WMF) 12880 2416818 wikitext text/x-wiki lralwi4j0ksmyffeowpqk2yl3rjomo0 2416819 2416818 2026-05-20T12:52:10Z Effie Mouzeli (WMF) 12880 2416819 wikitext text/x-wiki 41eajnh8p9bsuqq8ukuxo4quabb94vb 2417099 2416819 2026-05-20T14:59:28Z Effie Mouzeli (WMF) 12880 2417099 wikitext text/x-wiki 9xgbol83d1rf1lsft7pldwj9yfyynq9 2417100 2417099 2026-05-20T15:00:02Z Effie Mouzeli (WMF) 12880 2417100 wikitext text/x-wiki amar2kmpnhcshyxbmrrde7zk0m53x4x 2417167 2417100 2026-05-20T15:43:14Z Effie Mouzeli (WMF) 12880 2417167 wikitext text/x-wiki f7exwl9jjbnnrtbmkfm5ow241yqyl70 2417176 2417167 2026-05-20T15:45:04Z Effie Mouzeli (WMF) 12880 Undid revision [[Special:Diff/2417167|2417167]] by [[Special:Contributions/Effie Mouzeli (WMF)|Effie Mouzeli (WMF)]] ([[User talk:Effie Mouzeli (WMF)|talk]]) 2417176 wikitext text/x-wiki amar2kmpnhcshyxbmrrde7zk0m53x4x 2417188 2417176 2026-05-20T15:47:18Z Effie Mouzeli (WMF) 12880 2417188 wikitext text/x-wiki l7parxs1r3gsyzr4ee4zaoar617i4ar 14 460198 2416848 2026-05-20T13:01:11Z Effie Mouzeli (WMF) 12880 Created page with "Scroll Templating and Support Docs" 2416848 wikitext text/x-wiki p0x9mctjka5i88647rzwwavb49wdkcp 2416852 2416848 2026-05-20T13:01:38Z Effie Mouzeli (WMF) 12880 2416852 wikitext text/x-wiki 2sxnb87p9r53dkgwp170qhui8g9c9mw 0 460199 2416910 2026-05-20T13:20:28Z CMooney (WMF) 24288 CMooney (WMF) moved page [[Nokia switch upgrade]] to [[Nokia Networking/Nokia switch upgrade]] 2416910 wikitext text/x-wiki 0faszm51n7sojrz5wvcpphx854pk7ay 14 460200 2417012 2026-05-20T14:13:44Z Effie Mouzeli (WMF) 12880 Created page with "Category:SCROLLs" 2417012 wikitext text/x-wiki ode5a7xfh45hg2ep0iwxmcc4dm7peub 2417013 2417012 2026-05-20T14:13:53Z Effie Mouzeli (WMF) 12880 2417013 wikitext text/x-wiki cwjz8ypmuw96etnfftmgoi8lk4o4i4x 14 460201 2417018 2026-05-20T14:15:51Z Effie Mouzeli (WMF) 12880 Created page with "[[Category:SCROLLs]]" 2417018 wikitext text/x-wiki cwjz8ypmuw96etnfftmgoi8lk4o4i4x 14 460202 2417019 2026-05-20T14:16:20Z Effie Mouzeli (WMF) 12880 Created page with "#REDIRECT [[[[Category:SCROLLs]]]]" 2417019 wikitext text/x-wiki 6pw7l7dboilxgwv6flosjbjrl9vj10z 2417021 2417019 2026-05-20T14:16:30Z Effie Mouzeli (WMF) 12880 Redirected page to [[Category:SCROLLs]] 2417021 wikitext text/x-wiki m632tff1hib8yfyanqf6iif4kn5smyq 3 460203 2417024 2026-05-20T14:18:55Z StrikerBot 8475 Welcome to Toolforge! 2417024 wikitext text/x-wiki hxdh4hnhhb0egr378ndnxukt5fwlcg2 0 460205 2417042 2026-05-20T14:27:45Z Effie Mouzeli (WMF) 12880 Created page with "{{Special:PrefixIndex/SCROLL/Docs}}" 2417042 wikitext text/x-wiki c2f1fcwho1gn0s30nlmtdpsl91emdru 0 460206 2417045 2026-05-20T14:29:33Z Effie Mouzeli (WMF) 12880 Created page with "{{Draft}} {{DISPLAYTITLE:SCROLL: {{SUBPAGENAME}}}} <templatestyles src="User:Effie_Mouzeli_(WMF)/SCROLL/styles.css" /> ''Service Checklist for Readiness, Operations, Launch and Lifecycle'' {| style="width: 100%; border: none; background: transparent;" | style="vertical-align: top; width: 38%; padding-right: 2%;" | {| class="wikitable" style="font-size: 90%; width: 100%; border: 1px solid #a2a9b1;" |- ! colspan="2" class="scroll-card-header" | 📜 Service/Component ide..." 2417045 wikitext text/x-wiki gbh73k3skb7dyae4katdyoc4zk52i78 0 460207 2417048 2026-05-20T14:30:27Z Effie Mouzeli (WMF) 12880 Created page with "{{Special:PrefixIndex/SCROLL/requests}}" 2417048 wikitext text/x-wiki hfn0qdzap3a50dipwgi3hhm9ze5gv4v 14 460208 2417076 2026-05-20T14:47:08Z Effie Mouzeli (WMF) 12880 Created page with "[[Category:SCROLLs]]" 2417076 wikitext text/x-wiki cwjz8ypmuw96etnfftmgoi8lk4o4i4x 0 460209 2417126 2026-05-20T15:19:47Z Effie Mouzeli (WMF) 12880 Created page with "[[File:Books_and_Scroll_Ornament_with_Open_Book.png|right|frameless|150px|alt=A scroll]] <div style="background: #f3f9f6; border: 1px solid #a2a9b1; border-left: 6px solid #4b8063; border-radius: 6px; padding: 16px 20px; margin: 0 0 1.2em 0;"> <div style="font-size: 165%; font-weight: bold;">📜 Welcome to SCROLL!</div> <div style="font-size: 110%; color: #555; margin-top: 4px;">🔦 The production readiness checklist for Wikimedia services!</div> </div> <h2> 🚀R..." 2417126 wikitext text/x-wiki j7jnakb1mawti9vgy6who5is1g2u3py 2417127 2417126 2026-05-20T15:20:33Z Effie Mouzeli (WMF) 12880 2417127 wikitext text/x-wiki 208rplhvvr6jy0inoysu5z03bkzynui 2417143 2417127 2026-05-20T15:30:08Z Effie Mouzeli (WMF) 12880 2417143 wikitext text/x-wiki f20o4pjzxp75ri32ktsvgqxp6y0yzxh 2417557 2417143 2026-05-21T09:02:48Z Effie Mouzeli (WMF) 12880 2417557 wikitext text/x-wiki 1xulijrjgjub76dlewyddyc3c458jmm 2417558 2417557 2026-05-21T09:03:04Z Effie Mouzeli (WMF) 12880 2417558 wikitext text/x-wiki o4g9ozat4rltefgixxhi9spiqr6th45 2417559 2417558 2026-05-21T09:03:24Z Effie Mouzeli (WMF) 12880 2417559 wikitext text/x-wiki 1mkdeopex54ji0ov8dvb05sarsbsoz3 2417560 2417559 2026-05-21T09:04:17Z Effie Mouzeli (WMF) 12880 2417560 wikitext text/x-wiki 9nwk3c5pmqwbzqnyovtyvdvejz7ge5y 0 460210 2417139 2026-05-20T15:28:26Z Effie Mouzeli (WMF) 12880 Created page with "= SCROLL: worked examples = This page collects worked examples to help teams understand what a finished SCROLL and its companion Wikitech component page look like in practice. == Duck == Duck is a fictional HTTP service used as the reference example throughout the SCROLL process. It illustrates a typical service going through both Chapter 1 (soft launch) and Chapter 2 (full launch). {| class="wikitable" style="width: 100%;" |- ! Resource !! Description |- | [https://..." 2417139 wikitext text/x-wiki rjou4dosppg0wcryxt5wf3akah05dsf 2417140 2417139 2026-05-20T15:28:55Z Effie Mouzeli (WMF) 12880 2417140 wikitext text/x-wiki 0970nx6u4jl7ulcaa9m7i1fr9ucg4b0 0 460211 2417147 2026-05-20T15:32:17Z Effie Mouzeli (WMF) 12880 Redirected page to [[SCROLL]] 2417147 wikitext text/x-wiki fsq82bhs5gum5319tsigfosiq3l1taf 0 460212 2417431 2026-05-20T21:24:38Z Effie Mouzeli (WMF) 12880 Effie Mouzeli (WMF) moved page [[SCROLL/Ducks]] to [[SCROLL/Another Duck]] 2417431 wikitext text/x-wiki ru68u186vw5q8i122cyu9s95223g7uf