Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.11 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 0 4108 2436990 2436622 2026-07-15T13:52:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1311054]] to Thursday, July 16 UTC afternoon backport window 2436990 wikitext text/x-wiki 7fvckzix8oujkt4oe6a957qb2eghjg2 0 7919 2436881 2436880 2026-07-15T12:02:09Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw 2436881 wikitext text/x-wiki 6czk21ij5qo7wnl2i1w550adn76q4tj 2436882 2436881 2026-07-15T12:07:13Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2133-2142].codfw.wmnet 2436882 wikitext text/x-wiki 0740thu6hjz50ttknwlixjitp94c6x3 2436883 2436882 2026-07-15T12:07:20Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2133-2142].codfw.wmnet 2436883 wikitext text/x-wiki ao1sqb9s9otw1zu7887wxxi9qqok88f 2436884 2436883 2026-07-15T12:07:44Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2143-2152].codfw.wmnet 2436884 wikitext text/x-wiki 5zmzcb8weotecza4ax48xxwls5wys8j 2436885 2436884 2026-07-15T12:10:24Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw 2436885 wikitext text/x-wiki rzxz8vuhsn61hdxlz5x3qfj8scaun3c 2436886 2436885 2026-07-15T12:11:29Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=ms-fe2010.codfw.wmnet 2436886 wikitext text/x-wiki kkbgwh3afkut3ofm9qmqzlnpemxdmpc 2436887 2436886 2026-07-15T12:12:37Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=ms-fe2010.codfw.wmnet 2436887 wikitext text/x-wiki mcydv4bbyi7l37qq2fbbpzyo4tn5cps 2436888 2436887 2026-07-15T12:13:53Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2436888 wikitext text/x-wiki 76s1vfaigl294if6916nxi66xiy7em3 2436889 2436888 2026-07-15T12:14:07Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2143-2152].codfw.wmnet 2436889 wikitext text/x-wiki n9cnf8aah86bjyxf84d6zk3wo1156re 2436890 2436889 2026-07-15T12:14:39Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2436890 wikitext text/x-wiki b0f71b3ppogos7caj4vbg7ycfrnwc01 2436891 2436890 2026-07-15T12:15:03Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2436891 wikitext text/x-wiki ea3mk6avia02h9j3axs73h8v1qzmiz6 2436892 2436891 2026-07-15T12:15:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2009.codfw.wmnet with OS trixie 2436892 wikitext text/x-wiki cnz82aaacof93e9uvflxhfyq8m69xco 2436893 2436892 2026-07-15T12:15:23Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2436893 wikitext text/x-wiki j7kiixqvn85m9iy8jpkawt6u12omjlj 2436894 2436893 2026-07-15T12:15:37Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.move-vlan for host ms-fe2009 2436894 wikitext text/x-wiki rqasdkuu2g566dqlx0eo7vjaavxe149 2436895 2436894 2026-07-15T12:15:40Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2436895 wikitext text/x-wiki lqypqi90kii6mp4un1572brc20jzt1s 2436896 2436895 2026-07-15T12:15:57Z Stashbot 7414 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2436896 wikitext text/x-wiki omozh0l8idi97lykx3eh9y5edr0q2sf 2436897 2436896 2026-07-15T12:16:00Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.netbox 2436897 wikitext text/x-wiki g9fqnyprsoxer933d52sbqrx2smhm4q 2436898 2436897 2026-07-15T12:22:02Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2009 - mvernon@cumin2003" 2436898 wikitext text/x-wiki 9txkosjn8loe8w8x28qknbhu1v1x17h 2436899 2436898 2026-07-15T12:22:07Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2009 - mvernon@cumin2003" 2436899 wikitext text/x-wiki 73drf7lbeubg0gywn1vljssrma4qaln 2436900 2436899 2026-07-15T12:22:08Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436900 wikitext text/x-wiki guv1ggq11mjxai6r5fbgf9wwkonz60n 2436901 2436900 2026-07-15T12:22:09Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.dns.wipe-cache ms-fe2009.codfw.wmnet 139.0.192.10.in-addr.arpa 9.3.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436901 wikitext text/x-wiki hvapnl6fqjoolzhqxo8hl3fr62pfxdn 2436902 2436901 2026-07-15T12:22:11Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2009.codfw.wmnet 139.0.192.10.in-addr.arpa 9.3.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436902 wikitext text/x-wiki bi8azyn6dwklynz95anfvbmnc3gkzs6 2436903 2436902 2026-07-15T12:22:12Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2009 2436903 wikitext text/x-wiki nkk549tgod0ovp5nr46ceubn11dy9z9 2436904 2436903 2026-07-15T12:22:51Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2009 2436904 wikitext text/x-wiki 7ft4xmchanog15z1mvj38hat2u4so0v 2436905 2436904 2026-07-15T12:22:52Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2009 2436905 wikitext text/x-wiki l4vast1rngra6vpe3pbw1bva2dqimfq 2436906 2436905 2026-07-15T12:24:59Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2143-2152].codfw.wmnet 2436906 wikitext text/x-wiki qu5de8a4k27nbk4qqo4v4i1g45zykgp 2436907 2436906 2026-07-15T12:25:06Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2143-2152].codfw.wmnet 2436907 wikitext text/x-wiki o42eupypvi8i21ax4ce91gnsn49a6np 2436908 2436907 2026-07-15T12:25:25Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2153-2162].codfw.wmnet 2436908 wikitext text/x-wiki 9sxnbyjl5a5f5x7jbv08bns3my71y06 2436909 2436908 2026-07-15T12:29:36Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw 2436909 wikitext text/x-wiki hm5d8816vd0ibb7ceo4i9alwgmkdtmd 2436910 2436909 2026-07-15T12:31:06Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2153-2162].codfw.wmnet 2436910 wikitext text/x-wiki 1n6shardm4jk8axhzt15zidzz08842k 2436911 2436910 2026-07-15T12:40:10Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2009.codfw.wmnet with reason: host reimage 2436911 wikitext text/x-wiki iwt4en3biynxoai467pz519o1hcprd2 2436912 2436911 2026-07-15T12:40:57Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2153-2162].codfw.wmnet 2436912 wikitext text/x-wiki bkikmwtasp75n5cy84y792kkrq0d0sa 2436913 2436912 2026-07-15T12:41:04Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2153-2162].codfw.wmnet 2436913 wikitext text/x-wiki 27mctgk30vjgqwquiq356ozvbbl9l05 2436914 2436913 2026-07-15T12:41:32Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2163-2172].codfw.wmnet 2436914 wikitext text/x-wiki gfo5sqq7qmky5jm4z5qwqsiboirw09j 2436915 2436914 2026-07-15T12:43:49Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2009.codfw.wmnet with reason: host reimage 2436915 wikitext text/x-wiki dvg4ld735rk6p4b4o2sr87qdg0cy1dy 2436917 2436915 2026-07-15T12:47:15Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2163-2172].codfw.wmnet 2436917 wikitext text/x-wiki 3l3f1lk9gvxby9gbq2yebsy9gn4uh7o 2436918 2436917 2026-07-15T12:54:28Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2163-2172].codfw.wmnet 2436918 wikitext text/x-wiki h3otoisq6ynj9osf2kahng5h1kl4g4n 2436919 2436918 2026-07-15T12:54:34Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2163-2172].codfw.wmnet 2436919 wikitext text/x-wiki e52ptdfier6h0142b24mmnqw3vcutin 2436920 2436919 2026-07-15T12:54:58Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2173-2179,2184-2186].codfw.wmnet 2436920 wikitext text/x-wiki 9g881swq4c5hvrpxnnzu67zhqk84iht 2436921 2436920 2026-07-15T12:57:26Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid analytics cluster: Reboot Druid nodes 2436921 wikitext text/x-wiki 0uo34ptt6mvfjr32pnb9bw6hkgl2rcw 2436922 2436921 2026-07-15T13:00:44Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2173-2179,2184-2186].codfw.wmnet 2436922 wikitext text/x-wiki 9z4u31v3nr7cus1hmaryxjssdizjjes 2436923 2436922 2026-07-15T13:01:02Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 2436923 wikitext text/x-wiki 2ycqr00biugj43w9cj5qrxvvbmz4qad 2436924 2436923 2026-07-15T13:01:10Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 (duration: 00m 15s) 2436924 wikitext text/x-wiki o4cz47d5akh6w06n5xsr2xz55xvl1fs 2436925 2436924 2026-07-15T13:03:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430879, restore data on newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1002.eqiad.wmnet, repooling source-only afterwards 2436925 wikitext text/x-wiki 66r11vlgfq8mi8mk2h2eigxnw9r4462 2436926 2436925 2026-07-15T13:03:28Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2436926 wikitext text/x-wiki gmu5dg73kwxo13i8zx53l2iszs0rj0a 2436927 2436926 2026-07-15T13:03:37Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2009.codfw.wmnet with OS trixie 2436927 wikitext text/x-wiki rxce7mr84hmn6b4ny0tdg93n666f8di 2436928 2436927 2026-07-15T13:04:14Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2436928 wikitext text/x-wiki bvhfm6bk1mu1mbhw8ap85rk63r35pbt 2436929 2436928 2026-07-15T13:04:33Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2436929 wikitext text/x-wiki 9z4qef8l9x7th1f87kk8h1auswrjv2l 2436930 2436929 2026-07-15T13:04:34Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2436930 wikitext text/x-wiki l5mqtq1zpqjl7ha4zpblc3j9yser2g9 2436931 2436930 2026-07-15T13:05:18Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2436931 wikitext text/x-wiki b2k1r0n6pcn9kwefgade7v644a6zvgj 2436932 2436931 2026-07-15T13:05:46Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2001 2436932 wikitext text/x-wiki rdw1rjruj5hpa1hfiu7bq0w8oqt0emv 2436933 2436932 2026-07-15T13:05:49Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430879, restore data on newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1002.eqiad.wmnet, repooling source-only afterwards 2436933 wikitext text/x-wiki 1r4ok7o1658syr1ctsf6alzgottuxtu 2436934 2436933 2026-07-15T13:06:30Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw 2436934 wikitext text/x-wiki cdv2c263gej14ur5uv4fu66vm9ao4dn 2436939 2436934 2026-07-15T13:11:12Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2173-2179,2184-2186].codfw.wmnet 2436939 wikitext text/x-wiki jinkh60l5ys5zyr78ar2z64slbw2tmt 2436940 2436939 2026-07-15T13:11:19Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2173-2179,2184-2186].codfw.wmnet 2436940 wikitext text/x-wiki 57otd74r7y5xeyqwhd2rvy5h327wrvc 2436941 2436940 2026-07-15T13:11:38Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2187-2196].codfw.wmnet 2436941 wikitext text/x-wiki kprpylwldxiheurz3ticlx47o28jl4x 2436942 2436941 2026-07-15T13:12:16Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_ulsfo 2436942 wikitext text/x-wiki 72vk9gvi8q9kmqo3q12u6wlgbg3zfec 2436943 2436942 2026-07-15T13:12:34Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436943 wikitext text/x-wiki 4fjdaey9cv0cjwf8b1hsn9d2qezry9a 2436944 2436943 2026-07-15T13:12:37Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_ulsfo 2436944 wikitext text/x-wiki rf5ogs6n6l3db4xwme6j4vvkb6ww3cd 2436945 2436944 2026-07-15T13:12:43Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436945 wikitext text/x-wiki dy8lrutrptrp47276ma0jcj6ydedx9j 2436946 2436945 2026-07-15T13:13:01Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436946 wikitext text/x-wiki fxw4sucpd124a4x5qjds12kbjtmx35w 2436947 2436946 2026-07-15T13:13:07Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox and not (A:ulsfo or A:magru) and (A:dnsbox) 2436947 wikitext text/x-wiki s73ed6g4w5kionnbnthqhzqs0ttdir1 2436948 2436947 2026-07-15T13:13:09Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns1004.wikimedia.org 2436948 wikitext text/x-wiki lo960js2umsa98lrewqs0kwuan09qi4 2436949 2436948 2026-07-15T13:13:10Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad 2436949 wikitext text/x-wiki mqj3jq0imwk5sara8u65zrbcyvahf65 2436950 2436949 2026-07-15T13:13:14Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-jumbo-eqiad 2436950 wikitext text/x-wiki bv8s8zs28e0oprkwt7o0no0vzimsid8 2436951 2436950 2026-07-15T13:13:15Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436951 wikitext text/x-wiki 1unhdha8qsbfp2jxhalo95xxj6q9lfj 2436952 2436951 2026-07-15T13:13:21Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436952 wikitext text/x-wiki 70omvxygp9k0gkkr975o73dwosfu5vd 2436953 2436952 2026-07-15T13:13:40Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436953 wikitext text/x-wiki rc4jxh2fjgaexv28hvtuj3rfghdsmuh 2436954 2436953 2026-07-15T13:13:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2436954 wikitext text/x-wiki b04wur34tnvgzsottdl1snkcedjq8cn 2436955 2436954 2026-07-15T13:15:15Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw 2436955 wikitext text/x-wiki qzx1q2lkwlyouj1snro8ev03vnhmx1w 2436956 2436955 2026-07-15T13:17:11Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2436956 wikitext text/x-wiki 2f84csmh9ohyztv7ac8xv7ytzpvgqdk 2436957 2436956 2026-07-15T13:17:17Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 15 hosts with reason: switch upgrade and line card install 2436957 wikitext text/x-wiki imuenkmzvyiaqiimhm7zz8bwi5877xh 2436958 2436957 2026-07-15T13:17:20Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_drmrs 2436958 wikitext text/x-wiki qu44a1hudjyubwcaeuv4xiatc55voaj 2436959 2436958 2026-07-15T13:17:22Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2187-2196].codfw.wmnet 2436959 wikitext text/x-wiki knqudgp5zaylze435g7xrljd4lc9aqt 2436960 2436959 2026-07-15T13:17:28Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_drmrs 2436960 wikitext text/x-wiki a6mk6t6656lw71drv4gq1c0f8bfb2zd 2436961 2436960 2026-07-15T13:17:43Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=inactive; selector: name=ms-fe2009.codfw.wmnet 2436961 wikitext text/x-wiki qvkksdqobz4l0cvfeyml0jko52g1v8i 2436962 2436961 2026-07-15T13:17:52Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2001 - bking@cumin2003" 2436962 wikitext text/x-wiki bmbj2qo6a8rdcdvhiv1a3tm8l0l8vmo 2436963 2436962 2026-07-15T13:17:56Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2001 - bking@cumin2003" 2436963 wikitext text/x-wiki ilu8w5bg7gomihz92e1b43el01srcxk 2436964 2436963 2026-07-15T13:17:58Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436964 wikitext text/x-wiki iwiz6qeqzhgk2v0uhelzkgps0ldjwie 2436965 2436964 2026-07-15T13:17:59Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2001.codfw.wmnet 26.16.192.10.in-addr.arpa 6.2.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436965 wikitext text/x-wiki c4s5s1l3cmtp165ca6j5csp39hs1h6f 2436966 2436965 2026-07-15T13:18:02Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2001.codfw.wmnet 26.16.192.10.in-addr.arpa 6.2.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436966 wikitext text/x-wiki lwycdn8yy0u321j6pzdvlgbrz2qv6ld 2436967 2436966 2026-07-15T13:18:04Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2001 2436967 wikitext text/x-wiki gdk2pokpv5rlocd65fu8uzhjehqnyk9 2436968 2436967 2026-07-15T13:18:51Z Stashbot 7414 mvernon@cumin2003: conftool action : set/pooled=yes; selector: name=ms-fe2009.codfw.wmnet 2436968 wikitext text/x-wiki 9ebx1y0arb1y8f53rod1qwevi76c31m 2436969 2436968 2026-07-15T13:19:13Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on lvs[1018-1020].eqiad.wmnet with reason: switch upgrade and line card install 2436969 wikitext text/x-wiki dfb5iq7c2dk76tycsoiscdopadxqr4w 2436970 2436969 2026-07-15T13:21:55Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns1004.wikimedia.org 2436970 wikitext text/x-wiki k2mg5gd7wjny8cftdxmv9082w6rz96n 2436971 2436970 2026-07-15T13:22:23Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4045.ulsfo.wmnet 2436971 wikitext text/x-wiki 87pk44ttu7hdryg27q6ekchc4iu4og9 2436972 2436971 2026-07-15T13:22:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2001 2436972 wikitext text/x-wiki s41qessgt2nwun1wy1wv9c3zjcmwi9a 2436973 2436972 2026-07-15T13:22:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2001 2436973 wikitext text/x-wiki 3hl6so5dks829qxfa7mdpoq69hxrkn6 2436974 2436973 2026-07-15T13:23:48Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4037.ulsfo.wmnet 2436974 wikitext text/x-wiki jbmt4buleskkdn2ey54190w3s7lks5m 2436975 2436974 2026-07-15T13:27:29Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm 2436975 wikitext text/x-wiki kd7adge8kpots410dr87330hivaycir 2436976 2436975 2026-07-15T13:28:02Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2187-2196].codfw.wmnet 2436976 wikitext text/x-wiki qo4y5g8qzh0pxnt0537lfivr1gbf2rq 2436977 2436976 2026-07-15T13:28:08Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2187-2196].codfw.wmnet 2436977 wikitext text/x-wiki c2408ne9il7b7qy36hvhkx8yhhyycjo 2436978 2436977 2026-07-15T13:28:09Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6009.drmrs.wmnet 2436978 wikitext text/x-wiki ovjxplkrnx3vyu3zr3n00md7txof5u5 2436979 2436978 2026-07-15T13:28:29Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6001.drmrs.wmnet 2436979 wikitext text/x-wiki 6kcskysod70t65z0gvewxjzjlouerld 2436980 2436979 2026-07-15T13:28:33Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2197-2206].codfw.wmnet 2436980 wikitext text/x-wiki 68p8y0f48x39f9dpqhdtm5foyakh22s 2436981 2436980 2026-07-15T13:30:54Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2436981 wikitext text/x-wiki 9ddxl7j7zvxwjw9qzn9rnt977bczt9n 2436982 2436981 2026-07-15T13:35:21Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2197-2206].codfw.wmnet 2436982 wikitext text/x-wiki ewpx9t23gnciga1090s3z35muufd04t 2436983 2436982 2026-07-15T13:36:55Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns1005.wikimedia.org 2436983 wikitext text/x-wiki tog7bsk2z448uz4vyktv16yxog8m6a3 2436984 2436983 2026-07-15T13:44:00Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2197-2206].codfw.wmnet 2436984 wikitext text/x-wiki t96hndslm78cy3wnqkgr11mo3ag0nnr 2436985 2436984 2026-07-15T13:44:06Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2197-2206].codfw.wmnet 2436985 wikitext text/x-wiki 37m7c91lm2x3ob8imk4xnnxg83qbo9d 2436986 2436985 2026-07-15T13:44:25Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2207-2215,2242].codfw.wmnet 2436986 wikitext text/x-wiki a4d2ssfl43f92c0gyeti8tg3f094qkw 2436987 2436986 2026-07-15T13:49:29Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad 2436987 wikitext text/x-wiki daxu238kiy0wm6uzm3yiua2anm1q8g9 2436988 2436987 2026-07-15T13:50:37Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2207-2215,2242].codfw.wmnet 2436988 wikitext text/x-wiki m5c64rz703qha8f2ye5yc72b16ujjfu 2436989 2436988 2026-07-15T13:51:05Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns1005.wikimedia.org 2436989 wikitext text/x-wiki 4ks1k4df1g5uvzdilqrv7pd8o5oj0rq 2436991 2436989 2026-07-15T13:53:24Z Stashbot 7414 topranks: switch routing-engine on cr2-eqiad resetting all interfaces T417873 2436991 wikitext text/x-wiki 4tr1mk9wjgz309s41uii185gk5yte54 2436992 2436991 2026-07-15T13:55:44Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes 2436992 wikitext text/x-wiki amlyexm0sb4j8gqt16spshk5ic06ike 2436993 2436992 2026-07-15T13:55:55Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad 2436993 wikitext text/x-wiki r2gwcyg65xjzbh6o8iqvx7pdaej3a0f 2436994 2436993 2026-07-15T13:57:30Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436994 wikitext text/x-wiki 8gg0eee9a0oe3sgs3504853q1vrpfrt 2436995 2436994 2026-07-15T13:57:39Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436995 wikitext text/x-wiki ody9rs7txn4lnb6cv523j273274gf8n 2436996 2436995 2026-07-15T13:59:55Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2207-2215,2242].codfw.wmnet 2436996 wikitext text/x-wiki duq1ry579hctkhr6b6wyw0d0e9zep16 2436997 2436996 2026-07-15T14:00:01Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2207-2215,2242].codfw.wmnet 2436997 wikitext text/x-wiki j8lbrarvpu08pq0t3sgjtzr3asvivg2 2436998 2436997 2026-07-15T14:00:12Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cr1-eqiad with reason: switch upgrade and line card install 2436998 wikitext text/x-wiki q3v8xkkqgomm91hb3w14tlp46rrl28w 2436999 2436998 2026-07-15T14:00:23Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2243,2248-2256].codfw.wmnet 2436999 wikitext text/x-wiki c01opcr7ekhztf6p8tsw23yro97r2qu 2437000 2436999 2026-07-15T14:02:07Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4046.ulsfo.wmnet 2437000 wikitext text/x-wiki 115o50czto6fteck8sr7yge4wqkyan3 2437001 2437000 2026-07-15T14:05:28Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4038.ulsfo.wmnet 2437001 wikitext text/x-wiki 0euts70xlbsy9mel1fv9m3uhatyhuoz 2437002 2437001 2026-07-15T14:05:57Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm 2437002 wikitext text/x-wiki sr8s546aq60x883bfz0efq0bzsvb6ng 2437003 2437002 2026-07-15T14:06:04Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2243,2248-2256].codfw.wmnet 2437003 wikitext text/x-wiki 6zg6clfbkamfgdnpax5ydcquh4y3aoi 2437004 2437003 2026-07-15T14:06:06Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns1006.wikimedia.org 2437004 wikitext text/x-wiki 249m2r3qxz9qrxuzql7ywsi041wdloq 2437005 2437004 2026-07-15T14:09:12Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet 2437005 wikitext text/x-wiki etxmys1n6avv9f6tpc1ehbe547tmxt3 2437006 2437005 2026-07-15T14:09:26Z Stashbot 7414 jforrester@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2437006 wikitext text/x-wiki cuuqphlr6455zkm7fxshvkisnpe779g 2437007 2437006 2026-07-15T14:09:32Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet 2437007 wikitext text/x-wiki o0zqmw8lywwh0b7uvj8mtitgcibc1n1 2437008 2437007 2026-07-15T14:10:01Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS bookworm 2437008 wikitext text/x-wiki rjoviqivz0rwlja3ouutf4ld4guzhyy 2437009 2437008 2026-07-15T14:10:18Z Stashbot 7414 jforrester@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2437009 wikitext text/x-wiki asimd5nik29scpozv96u5i21q6gfevy 2437010 2437009 2026-07-15T14:10:34Z Stashbot 7414 jforrester@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2437010 wikitext text/x-wiki 9vr3qsx4ig5q4uj54ry1ebygrfb2yhw 2437011 2437010 2026-07-15T14:11:00Z Stashbot 7414 jforrester@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2437011 wikitext text/x-wiki oc6equk36p9jtzyb7xw1isdm5oil6cz 2437012 2437011 2026-07-15T14:11:05Z Stashbot 7414 jforrester@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2437012 wikitext text/x-wiki 3cbym7nph38nz0gowt8doqo7tqw90mx 2437013 2437012 2026-07-15T14:11:30Z Stashbot 7414 jforrester@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2437013 wikitext text/x-wiki 96016jwz7u38jz1bk9wphnikjc8acrq 2437014 2437013 2026-07-15T14:14:05Z Stashbot 7414 topranks: switch routing-engine on cr2-eqiad resetting all interfaces T417873 2437014 wikitext text/x-wiki p2pz4dhoe5620nxrbp7f1qcp0ilwt9n 2437015 2437014 2026-07-15T14:15:35Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema-eqiad 2437015 wikitext text/x-wiki i3bjs3wui7j8igggpdi6cpau3u8ym4p 2437016 2437015 2026-07-15T14:15:36Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns1006.wikimedia.org 2437016 wikitext text/x-wiki tir883i6hgk8e8h1udxxg41doe9yb64 2437017 2437016 2026-07-15T14:15:52Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1064.eqiad.wmnet with OS trixie 2437017 wikitext text/x-wiki efbtp4e3wdw4l4cjb1l24jn5t7s7thi 2437018 2437017 2026-07-15T14:16:24Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS trixie 2437018 wikitext text/x-wiki kfvelpphkqln1fua5uydaompfis1yuw 2437019 2437018 2026-07-15T14:16:29Z Stashbot 7414 jforrester@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2437019 wikitext text/x-wiki h3oll5ue8vdr924qhcofbnexkcrwqs8 2437020 2437019 2026-07-15T14:16:30Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2243,2248-2256].codfw.wmnet 2437020 wikitext text/x-wiki qosnpbyzeipbsnk9qfibjk6it8trez3 2437021 2437020 2026-07-15T14:16:37Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2243,2248-2256].codfw.wmnet 2437021 wikitext text/x-wiki suw3ozd7jhw0un20qimn8i2i2r6hm02 2437022 2437021 2026-07-15T14:16:54Z Stashbot 7414 jforrester@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2437022 wikitext text/x-wiki skcps5v2hmibmgwtt2o0kcwbsjeql4b 2437023 2437022 2026-07-15T14:16:56Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2257-2266].codfw.wmnet 2437023 wikitext text/x-wiki 9i0wgys7ttyvka7eay3bl42n92ncfqa 2437024 2437023 2026-07-15T14:17:08Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437024 wikitext text/x-wiki jaeewgkbnv5hto22ur34isa5vvkj2ov 2437025 2437024 2026-07-15T14:17:12Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437025 wikitext text/x-wiki nngegjgzfz0arm4tg3biufsk4viejii 2437026 2437025 2026-07-15T14:17:33Z Stashbot 7414 jforrester@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2437026 wikitext text/x-wiki bt4gvdz166452lo0ga3yuaom40rhswd 2437027 2437026 2026-07-15T14:18:16Z Stashbot 7414 jforrester@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2437027 wikitext text/x-wiki 3bwsfbtrdn8gp6tif8h7g4okozihjq9 2437028 2437027 2026-07-15T14:18:25Z Stashbot 7414 jforrester@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2437028 wikitext text/x-wiki 9tz3tgj9m71m3h3t3jxpug1e4g2un5p 2437029 2437028 2026-07-15T14:19:05Z Stashbot 7414 jforrester@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2437029 wikitext text/x-wiki 8g8dvukawdgtamsjv6jzu1da3vtpor9 2437030 2437029 2026-07-15T14:20:09Z Stashbot 7414 jforrester@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2437030 wikitext text/x-wiki meaibxun43skeou8uvgihlnm61mfnbs 2437031 2437030 2026-07-15T14:20:13Z Stashbot 7414 jforrester@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2437031 wikitext text/x-wiki m1mmarvlvkhkb0qmib3bb6chwn4ltbf 2437032 2437031 2026-07-15T14:22:39Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2257-2266].codfw.wmnet 2437032 wikitext text/x-wiki n8jc8igpfez4um5kbpjcqzowetywd6u 2437033 2437032 2026-07-15T14:24:54Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema-eqiad 2437033 wikitext text/x-wiki 5qe64nonfgfvjywsqv8wot5lx6nt2wc 2437034 2437033 2026-07-15T14:29:33Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2257-2266].codfw.wmnet 2437034 wikitext text/x-wiki 8uxotnroxtbpjqcjy3j19m8wnymqw5i 2437035 2437034 2026-07-15T14:29:39Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2257-2266].codfw.wmnet 2437035 wikitext text/x-wiki 50qie4ge14czh62n3sdr4oqtg05sqkh 2437036 2437035 2026-07-15T14:30:00Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2267-2276].codfw.wmnet 2437036 wikitext text/x-wiki alx92iuq6stikvls5v7xt0ooqwsnh4m 2437037 2437036 2026-07-15T14:30:01Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns2004.wikimedia.org 2437037 wikitext text/x-wiki hy06jknbxjnlak7evvlmyyy5he4755z 2437038 2437037 2026-07-15T14:30:34Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437038 wikitext text/x-wiki 37g3y8jkssdhaal33ljmdq6xpedn2kz 2437039 2437038 2026-07-15T14:32:36Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437039 wikitext text/x-wiki bbws9eiwavgr6kurgd095e85sz9fsz2 2437040 2437039 2026-07-15T14:33:11Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437040 wikitext text/x-wiki tqyhzsvx4bfvng6kyngzrc5u7821hkx 2437041 2437040 2026-07-15T14:33:23Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437041 wikitext text/x-wiki a89iylxhdsvuqbp8ik955m9f75pbdm6 2437042 2437041 2026-07-15T14:34:12Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage 2437042 wikitext text/x-wiki rvrreylgesar9yc42639n4bzf2sxjmm 2437043 2437042 2026-07-15T14:35:26Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2437043 wikitext text/x-wiki q7dqgrbquigk5kglqd8aypbhjl8cyrx 2437044 2437043 2026-07-15T14:35:27Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve1001.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2437044 wikitext text/x-wiki 49q0gsr6xmbryucqzl1nif0868e7ff1 2437045 2437044 2026-07-15T14:35:30Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet 2437045 wikitext text/x-wiki b5a0680d5jc956lnn22ll0jxg3wa4s6 2437046 2437045 2026-07-15T14:35:33Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm 2437046 wikitext text/x-wiki 64dkemd8y6dus4n09b269j8gc02prt2 2437047 2437046 2026-07-15T14:36:13Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2267-2276].codfw.wmnet 2437047 wikitext text/x-wiki mpsmwmhbqhqv8x4vrcb24t8956f86mn 2437048 2437047 2026-07-15T14:36:38Z Stashbot 7414 topranks: disconnect power on cr2-eqiad to shut down device for switch fabric replacement T426343 2437048 wikitext text/x-wiki humkwlr8f5k7uq1o7wmltblinizypg1 2437049 2437048 2026-07-15T14:36:39Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage 2437049 wikitext text/x-wiki 7110ywuzmh1du9umy83m8g9vnglvafc 2437050 2437049 2026-07-15T14:38:32Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage 2437050 wikitext text/x-wiki qyl4kox9hxj756lnsithb5sgjnbbpdu 2437051 2437050 2026-07-15T14:40:32Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet 2437051 wikitext text/x-wiki bizidde467q5geq0bmupvdn0m15oyvu 2437052 2437051 2026-07-15T14:41:53Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4047.ulsfo.wmnet 2437052 wikitext text/x-wiki 76vy5p3p6c6hu0q9w5wzmecpnv3xhoe 2437053 2437052 2026-07-15T14:43:08Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage 2437053 wikitext text/x-wiki a0whr36gffejj8hrnlfv8nco7r22ebw 2437054 2437053 2026-07-15T14:43:36Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2267-2276].codfw.wmnet 2437054 wikitext text/x-wiki ta0d6xdusvmjd9d276enbqnyn2i83qh 2437055 2437054 2026-07-15T14:43:43Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2267-2276].codfw.wmnet 2437055 wikitext text/x-wiki dw74zg121q508lxvpcq9r32h56gbdf0 2437056 2437055 2026-07-15T14:44:03Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2277-2286].codfw.wmnet 2437056 wikitext text/x-wiki 3db9i5stpg9kctk2884vdtushb6pje7 2437057 2437056 2026-07-15T14:44:14Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns2004.wikimedia.org 2437057 wikitext text/x-wiki hk7l3bbbgpu96taqr5uhavclgjhvov0 2437058 2437057 2026-07-15T14:44:17Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema-codfw 2437058 wikitext text/x-wiki p0wrw7bublq2i32wjjr5h2pfbwf82nh 2437059 2437058 2026-07-15T14:44:35Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host relforge1009.eqiad.wmnet 2437059 wikitext text/x-wiki b35vpe5xibhnpwrmgskt4jbeu6vdu2o 2437060 2437059 2026-07-15T14:45:09Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4039.ulsfo.wmnet 2437060 wikitext text/x-wiki lzn4ozacr8derayvsnd8k6iu096xghp 2437061 2437060 2026-07-15T14:45:49Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet 2437061 wikitext text/x-wiki dqdu4nw2oan1g7gvsb9nv5zx20em1my 2437062 2437061 2026-07-15T14:46:16Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437062 wikitext text/x-wiki ek8y19v4wh62p2d6qywka3rks4mzljk 2437063 2437062 2026-07-15T14:46:38Z Stashbot 7414 klausman@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host ml-serve1001.eqiad.wmnet 2437063 wikitext text/x-wiki 4yw2fbc6extcv87iyfd7pplo4mgcpya 2437064 2437063 2026-07-15T14:46:39Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve1001.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2437064 wikitext text/x-wiki grm2g5hb1ei8xmp66h4kqff54p8cg0e 2437065 2437064 2026-07-15T14:47:43Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437065 wikitext text/x-wiki f66297fi92osqp6o8copips9np013xi 2437066 2437065 2026-07-15T14:49:44Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet 2437066 wikitext text/x-wiki 2bwxl30bkg7ok86mm5bjbn27trvwya5 2437067 2437066 2026-07-15T14:49:47Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2277-2286].codfw.wmnet 2437067 wikitext text/x-wiki 4kvzp7eclqc13mc3j9fx41k28pbywmc 2437068 2437067 2026-07-15T14:50:06Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host relforge1009.eqiad.wmnet 2437068 wikitext text/x-wiki czl9h1o4ajwgcz4j5848jy384nbhge8 2437069 2437068 2026-07-15T14:50:07Z Stashbot 7414 jforrester@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2437069 wikitext text/x-wiki 1yfm6v6pabo24izmd8a4imn07vfu7tx 2437070 2437069 2026-07-15T14:50:11Z Stashbot 7414 jforrester@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2437070 wikitext text/x-wiki 4apva0fu1rpcqblu3sdt79onkucxyys 2437071 2437070 2026-07-15T14:50:27Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet 2437071 wikitext text/x-wiki tj7rjeq908pmc3p8hflzqux4ahve3pp 2437072 2437071 2026-07-15T14:52:39Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema-codfw 2437072 wikitext text/x-wiki n0waaeusulu1rzzsu2fjxbq4zrxhe7v 2437073 2437072 2026-07-15T14:57:44Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-jumbo-eqiad 2437073 wikitext text/x-wiki r3bw7ekz9p7aah8aq163r0yn48rmogy 2437075 2437073 2026-07-15T14:59:15Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns2005.wikimedia.org 2437075 wikitext text/x-wiki 25z2b5hq7t3rd8zr3el7r85sjsm3bh5 2437076 2437075 2026-07-15T15:00:16Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2277-2286].codfw.wmnet 2437076 wikitext text/x-wiki 78nxt0s5jlvrrwgksj806r8esl7xc4w 2437077 2437076 2026-07-15T15:00:22Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2277-2286].codfw.wmnet 2437077 wikitext text/x-wiki q9jcyqedbbuavleah33wsi637xg1ojg 2437078 2437077 2026-07-15T15:00:41Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2287-2296].codfw.wmnet 2437078 wikitext text/x-wiki i9nhslu1hb7xbdvaiueemvrl981skrk 2437079 2437078 2026-07-15T15:05:39Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS trixie 2437079 wikitext text/x-wiki 8dbsby2y0gmvl95tc1jqp4rho6svy0j 2437080 2437079 2026-07-15T15:06:38Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1064.eqiad.wmnet with OS trixie 2437080 wikitext text/x-wiki kz5bvm6o8pkr3x2svpseyenyuv0ybj6 2437082 2437080 2026-07-15T15:11:36Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad 2437082 wikitext text/x-wiki g84a36eza7weg8je5rq4k55egy4rk0l 2437083 2437082 2026-07-15T15:11:57Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2287-2296].codfw.wmnet 2437083 wikitext text/x-wiki 8foiug94tzebjdcl3ypm7f0kmzwz7d0 2437084 2437083 2026-07-15T15:13:01Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns2005.wikimedia.org 2437084 wikitext text/x-wiki n8boduwyco7h1a32dv1sof5w5gueps6 2437085 2437084 2026-07-15T15:14:23Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2063.codfw.wmnet with OS trixie 2437085 wikitext text/x-wiki 81ehbxf1eisiksdcja4l06o1fvmc5s4 2437086 2437085 2026-07-15T15:17:26Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS bookworm 2437086 wikitext text/x-wiki bw55m9wrb1vww2swr4yiolj3utjixhm 2437087 2437086 2026-07-15T15:18:00Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm 2437087 wikitext text/x-wiki 3thlxpn3ft6ikzi40mgpddfvuy2tpir 2437089 2437087 2026-07-15T15:20:27Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid public cluster: Reboot Druid nodes 2437089 wikitext text/x-wiki 13y1bzb7pjkaqusut58h6nce1l19q8q 2437090 2437089 2026-07-15T15:20:48Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2287-2296].codfw.wmnet 2437090 wikitext text/x-wiki nl5idb9g5yodigwn7k0cga4wmjq31qk 2437091 2437090 2026-07-15T15:20:54Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2287-2296].codfw.wmnet 2437091 wikitext text/x-wiki h2uwoi5xd8jfynwaril5io7lr9ammi3 2437092 2437091 2026-07-15T15:21:14Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2297-2306].codfw.wmnet 2437092 wikitext text/x-wiki 30kifxkcuozth6umb9tjjw8ch8bigz2 2437093 2437092 2026-07-15T15:21:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. 2437093 wikitext text/x-wiki hhv60rermigyt8baqz2tvivjw7rn1p4 2437094 2437093 2026-07-15T15:23:21Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4048.ulsfo.wmnet 2437094 wikitext text/x-wiki 24h8au0ewzcmdsrfgx11267upbie4c8 2437095 2437094 2026-07-15T15:24:08Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1065.eqiad.wmnet with OS trixie 2437095 wikitext text/x-wiki eqaf8wzyeyem73upa22mxbzwrsdzc1n 2437096 2437095 2026-07-15T15:27:11Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4040.ulsfo.wmnet 2437096 wikitext text/x-wiki qcs8x239npdrsumhrj5yj0t10g6y99b 2437097 2437096 2026-07-15T15:27:35Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2297-2306].codfw.wmnet 2437097 wikitext text/x-wiki se5l4j4i6jh1c6i1reldk7mmbtsgm1p 2437098 2437097 2026-07-15T15:27:44Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. 2437098 wikitext text/x-wiki tl8ceaj72wpu61432udfk9qa70have7 2437099 2437098 2026-07-15T15:28:01Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns2006.wikimedia.org 2437099 wikitext text/x-wiki 6530u6fxvomhde0arkslv82kgcqhm6r 2437100 2437099 2026-07-15T15:30:46Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet 2437100 wikitext text/x-wiki 32md1uxx10bvh3qas9e7wk85ur694vf 2437101 2437100 2026-07-15T15:31:20Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw 2437101 wikitext text/x-wiki kvr36n2xt1xfg2y72f5s0vot12ia17z 2437102 2437101 2026-07-15T15:31:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet 2437102 wikitext text/x-wiki 5sv12y04yib5nrv5hdxd0mn3upzo2sk 2437103 2437102 2026-07-15T15:31:25Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet 2437103 wikitext text/x-wiki 2n02h26pvi2a1h467z6w14nmw3fhkaj 2437104 2437103 2026-07-15T15:31:28Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet 2437104 wikitext text/x-wiki ry2sbtqcc43mndg87c3cko1xr0ifmd4 2437105 2437104 2026-07-15T15:33:53Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage 2437105 wikitext text/x-wiki 64vi98h1sy5b6cw9h2usnyvmljwairx 2437108 2437105 2026-07-15T15:36:46Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet 2437108 wikitext text/x-wiki 5ui9mwnoh0eziki6llj89zcagqmc5wr 2437109 2437108 2026-07-15T15:36:47Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet 2437109 wikitext text/x-wiki 3ne7x5c2y4080pv6ko36alxi4ki88h2 2437110 2437109 2026-07-15T15:36:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet 2437110 wikitext text/x-wiki tcldg4rziebkhy98latkwffpymuoe3a 2437111 2437110 2026-07-15T15:36:55Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet 2437111 wikitext text/x-wiki 09nmxozukevbbho0rqpst6mkmzwxtsd 2437112 2437111 2026-07-15T15:37:13Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2297-2306].codfw.wmnet 2437112 wikitext text/x-wiki tpjeef5ncxnadefabsqmt6bpo1rstil 2437113 2437112 2026-07-15T15:37:20Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2297-2306].codfw.wmnet 2437113 wikitext text/x-wiki iktvbix0hrdlokuxhh3eh9oa6j8lmpw 2437114 2437113 2026-07-15T15:37:44Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2307-2316].codfw.wmnet 2437114 wikitext text/x-wiki 9nw97rs41prw8pwn1ir4dbxok9tcg8v 2437115 2437114 2026-07-15T15:38:30Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage 2437115 wikitext text/x-wiki oh1d05zfuu1m8aachrb40tx0zbb5x5d 2437116 2437115 2026-07-15T15:42:09Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns2006.wikimedia.org 2437116 wikitext text/x-wiki naqqqnviq2okbte5xrhlevtss5cdc4u 2437117 2437116 2026-07-15T15:42:16Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet 2437117 wikitext text/x-wiki lfgdne94mq5ygmezzhgnt4i3sk2yfub 2437118 2437117 2026-07-15T15:42:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet 2437118 wikitext text/x-wiki 2q28avf1omyk7ktyz490upfmp8gtpcz 2437119 2437118 2026-07-15T15:42:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet 2437119 wikitext text/x-wiki jxrkw8l69cntfz7dfg5e6a5akf6mjuu 2437120 2437119 2026-07-15T15:42:54Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1065.eqiad.wmnet with reason: host reimage 2437120 wikitext text/x-wiki 26sjhwirofzcl75febo6eafmir11o3y 2437121 2437120 2026-07-15T15:42:55Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet 2437121 wikitext text/x-wiki qvul20j2uk0o1y1z51cx5e9xymp2x9a 2437122 2437121 2026-07-15T15:43:27Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2307-2316].codfw.wmnet 2437122 wikitext text/x-wiki nc8onwcxtfyqwbuxgaco2b6x23v2le5 2437123 2437122 2026-07-15T15:47:59Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1065.eqiad.wmnet with reason: host reimage 2437123 wikitext text/x-wiki a9slu8yqmygjarg1nzufvbnl4f07zgy 2437124 2437123 2026-07-15T15:48:28Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet 2437124 wikitext text/x-wiki 4xst4h9v6cmttqzbifrwrr50duwa4jb 2437125 2437124 2026-07-15T15:48:30Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet 2437125 wikitext text/x-wiki lls5jeifntdxv2t8wjyi3wchkvybrlr 2437126 2437125 2026-07-15T15:48:35Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet 2437126 wikitext text/x-wiki dg97mxwvy9wal6w9nuu8fx7feeqyqd8 2437127 2437126 2026-07-15T15:49:10Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet 2437127 wikitext text/x-wiki 3n9a2uckjphml9kl0nz31pt976rl4ug 2437128 2437127 2026-07-15T15:51:25Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2307-2316].codfw.wmnet 2437128 wikitext text/x-wiki e4hfg2oei5mrd02a1p7fe6tsejlbh2g 2437129 2437128 2026-07-15T15:51:31Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2307-2316].codfw.wmnet 2437129 wikitext text/x-wiki 1lis6fszzouebnz2gzxeyd8dqxd6iyv 2437130 2437129 2026-07-15T15:51:50Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2317-2326].codfw.wmnet 2437130 wikitext text/x-wiki 0w9n0rjteavc6vfer90evh5vtgprwe3 2437131 2437130 2026-07-15T15:54:36Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet 2437131 wikitext text/x-wiki fbl382l9mokv3hg6ukh3344bpqsmv7n 2437132 2437131 2026-07-15T15:54:37Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet 2437132 wikitext text/x-wiki kn7dsok31mdmg4uja1r0fjl6175ka9f 2437133 2437132 2026-07-15T15:54:43Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet 2437133 wikitext text/x-wiki h6xdgl1im3h4qqmizd266mfo90r7q5d 2437134 2437133 2026-07-15T15:54:44Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet 2437134 wikitext text/x-wiki 1vm811tyg56lx47z92ngjxc8jzl5bie 2437135 2437134 2026-07-15T15:57:09Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns3003.wikimedia.org 2437135 wikitext text/x-wiki pin3ft8fumy1y0gtfo3o1bud7tjeopv 2437136 2437135 2026-07-15T15:57:32Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2317-2326].codfw.wmnet 2437136 wikitext text/x-wiki kdy212gxmsw8muc7p129lbypf6tn886 2437137 2437136 2026-07-15T16:00:34Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2063.codfw.wmnet with OS trixie 2437137 wikitext text/x-wiki gkukucfyc26qfdccplwozf2xve4nyvk 2437138 2437137 2026-07-15T16:00:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet 2437138 wikitext text/x-wiki mjyl8st56kvkwc45tzfoe2c841e6qkh 2437139 2437138 2026-07-15T16:00:47Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet 2437139 wikitext text/x-wiki r8wbeaxdi3jhhef97bsx8k5h7hqnc7v 2437140 2437139 2026-07-15T16:00:51Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet 2437140 wikitext text/x-wiki b1w0j4wni0kzpp8ggikalyj72k97qho 2437142 2437140 2026-07-15T16:03:08Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4049.ulsfo.wmnet 2437142 wikitext text/x-wiki oswjcudiadggt5po5bhbtnwj2jc4dur 2437143 2437142 2026-07-15T16:05:15Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-analytics-product: apply 2437143 wikitext text/x-wiki fc01in4zhsv0l4qmy3kac9a4bhxg918 2437144 2437143 2026-07-15T16:05:21Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-analytics-product: apply 2437144 wikitext text/x-wiki 8x7a98cu76z62flud2fyvb7qibgf4nb 2437145 2437144 2026-07-15T16:05:29Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-analytics-test: apply 2437145 wikitext text/x-wiki ced4vrgivj80cnein0hd3uizjka8rnq 2437146 2437145 2026-07-15T16:05:36Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-analytics-test: apply 2437146 wikitext text/x-wiki 8qd52ypqa51s55itf0blf1iqs1qnzid 2437147 2437146 2026-07-15T16:05:57Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet 2437147 wikitext text/x-wiki ge3viudt1bpggbhcc9iy2wypsslbkox 2437148 2437147 2026-07-15T16:06:59Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2317-2326].codfw.wmnet 2437148 wikitext text/x-wiki plzjqd0r5xmo6t7sgu417gfh6ewf47q 2437149 2437148 2026-07-15T16:07:05Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2317-2326].codfw.wmnet 2437149 wikitext text/x-wiki 5kltd9lrvz3d6c2xsfvop4oxyfb1pxt 2437150 2437149 2026-07-15T16:07:30Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2327-2336].codfw.wmnet 2437150 wikitext text/x-wiki iqil3mr7x2kpg4yeiw1wjbwkjnlokhc 2437151 2437150 2026-07-15T16:08:36Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns3003.wikimedia.org 2437151 wikitext text/x-wiki bmakuyns8wcwdb0x3yumh7rq4eafoz6 2437152 2437151 2026-07-15T16:08:58Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4041.ulsfo.wmnet 2437152 wikitext text/x-wiki 6rfklf5cwwk53sr4nd58kx8d23374g1 2437153 2437152 2026-07-15T16:11:51Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet 2437153 wikitext text/x-wiki 15qdci7zg95rclkjq39z9ubwtn3f2x3 2437154 2437153 2026-07-15T16:12:32Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet 2437154 wikitext text/x-wiki bfwi0thrr0eiuy7ahbphi25jyrtxvfv 2437155 2437154 2026-07-15T16:12:34Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1065.eqiad.wmnet with OS trixie 2437155 wikitext text/x-wiki egfyl1mjsekqb5ltwvias2r7xipnzx4 2437156 2437155 2026-07-15T16:12:38Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet 2437156 wikitext text/x-wiki j4zis5kh43qigidzfp2zxmixquiy8ky 2437157 2437156 2026-07-15T16:12:40Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet 2437157 wikitext text/x-wiki 8f7yd71vx24owia0lr1lxwbs9zop4n1 2437158 2437157 2026-07-15T16:12:46Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet 2437158 wikitext text/x-wiki orguf4whuzq06t6f3l99udv6amqiafa 2437159 2437158 2026-07-15T16:13:15Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2327-2336].codfw.wmnet 2437159 wikitext text/x-wiki 2v56310yv0cj28wrvkqvs9cix9ooqzi 2437160 2437159 2026-07-15T16:17:58Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet 2437160 wikitext text/x-wiki kvtqyj1qqnx6sipyz3yh07how8841p0 2437161 2437160 2026-07-15T16:22:55Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2327-2336].codfw.wmnet 2437161 wikitext text/x-wiki dlbrwescfcnuhcfbflggfkfknazssd6 2437162 2437161 2026-07-15T16:23:01Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2327-2336].codfw.wmnet 2437162 wikitext text/x-wiki bfpqzy9d2vb1ogdsk1isjwmjcwn2ih8 2437163 2437162 2026-07-15T16:23:21Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2337-2346].codfw.wmnet 2437163 wikitext text/x-wiki 7uc5e437appsv3et3s3iuvlgqgtlj9a 2437164 2437163 2026-07-15T16:23:36Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns3004.wikimedia.org 2437164 wikitext text/x-wiki 7ucmj4q7dq9lzb15d96ta5wyo58xov7 2437165 2437164 2026-07-15T16:24:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet 2437165 wikitext text/x-wiki gippvjw5yy4uyv58ao3yueoibj2y7oi 2437166 2437165 2026-07-15T16:24:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet 2437166 wikitext text/x-wiki 6b61vukql716lp6vdg8zrb5jv7366g5 2437167 2437166 2026-07-15T16:24:47Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet 2437167 wikitext text/x-wiki 707bgtbz2v5ld75y8df8mq8gv61o3il 2437168 2437167 2026-07-15T16:29:40Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2337-2346].codfw.wmnet 2437168 wikitext text/x-wiki 3av9osj34628cxqjgap78x3mxy48g15 2437169 2437168 2026-07-15T16:29:59Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet 2437169 wikitext text/x-wiki ik0lpx62oopifz6b2osbea8nvwp3rka 2437170 2437169 2026-07-15T16:30:55Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dev: apply 2437170 wikitext text/x-wiki ik1y5b3i0e2v6xvjdwu1b5h9zq49iha 2437171 2437170 2026-07-15T16:32:35Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dev: apply 2437171 wikitext text/x-wiki am4gy1b7k7vie3814nydmto6obpee4a 2437172 2437171 2026-07-15T16:32:41Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply 2437172 wikitext text/x-wiki nluyzbq46e6hlsi7yvil7xg1lcqaiqm 2437173 2437172 2026-07-15T16:32:46Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply 2437173 wikitext text/x-wiki ebkef9j78o1n970vxye9zekzaq93coi 2437174 2437173 2026-07-15T16:32:52Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply 2437174 wikitext text/x-wiki d0skm8zba4r641w2qpcsk5lbvout9s2 2437175 2437174 2026-07-15T16:33:00Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply 2437175 wikitext text/x-wiki 2j1zetf5f0qddmqiiw1fa1facw4oqrg 2437176 2437175 2026-07-15T16:33:35Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-ml: apply 2437176 wikitext text/x-wiki r79lxblrsh1jjh6fvpdyno7vya1o228 2437177 2437176 2026-07-15T16:33:42Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-ml: apply 2437177 wikitext text/x-wiki 50iofvayn8ho9er4ztcmis3sbjc8rhe 2437178 2437177 2026-07-15T16:34:46Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns3004.wikimedia.org 2437178 wikitext text/x-wiki j6jl55yxsr82loc2x7hlhn94t770l9y 2437179 2437178 2026-07-15T16:35:20Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet 2437179 wikitext text/x-wiki gbj7n00cu0uncucr79don68xs8qckvf 2437180 2437179 2026-07-15T16:35:21Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet 2437180 wikitext text/x-wiki i3cfn208a44ccqqm1lzmtxf7d1jmlt6 2437181 2437180 2026-07-15T16:35:22Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw 2437181 wikitext text/x-wiki f3c02zhmeedpkkd70ofs1p7lag5kzjc 2437182 2437181 2026-07-15T16:36:49Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2337-2346].codfw.wmnet 2437182 wikitext text/x-wiki h1pehy90mjfz0hmr8cak8mjopi3xk3q 2437183 2437182 2026-07-15T16:36:55Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2337-2346].codfw.wmnet 2437183 wikitext text/x-wiki 55x14h66d6bjyfilyy2mjbp2b1cqyl4 2437184 2437183 2026-07-15T16:37:14Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2347-2356].codfw.wmnet 2437184 wikitext text/x-wiki d9dgcfmmsdjuysz4c9rjbnrykfm5icf 2437185 2437184 2026-07-15T16:43:01Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2347-2356].codfw.wmnet 2437185 wikitext text/x-wiki fl2zp5ir91hv6kgra35lu92y2s0xi2v 2437186 2437185 2026-07-15T16:43:09Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4050.ulsfo.wmnet 2437186 wikitext text/x-wiki p40a7rotue5y0o4q2wlrthxhur5skzt 2437187 2437186 2026-07-15T16:44:18Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-platform-eng: apply 2437187 wikitext text/x-wiki eqwwf87l2ja6tg3l3cf0wma6xwdcatn 2437188 2437187 2026-07-15T16:44:25Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-platform-eng: apply 2437188 wikitext text/x-wiki 58dhgly1gdbq0ylame83bhzkxej3kek 2437189 2437188 2026-07-15T16:44:31Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-research: apply 2437189 wikitext text/x-wiki noi9gfareudho6iai5rzbo4rmmjb3tp 2437190 2437189 2026-07-15T16:44:38Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-research: apply 2437190 wikitext text/x-wiki 15ovdrel623j42siv574vpb9v9dbqd7 2437191 2437190 2026-07-15T16:49:46Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns5003.wikimedia.org 2437191 wikitext text/x-wiki pjnef73kuogq2cdzdvknp18qvkqlxp3 2437192 2437191 2026-07-15T16:50:12Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2347-2356].codfw.wmnet 2437192 wikitext text/x-wiki naib4v3199u2lhmzmtaatc9ulpcy1gx 2437193 2437192 2026-07-15T16:50:18Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2347-2356].codfw.wmnet 2437193 wikitext text/x-wiki oplbdhun0zy3s2ftgvbncirt4b61n88 2437194 2437193 2026-07-15T16:50:39Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4042.ulsfo.wmnet 2437194 wikitext text/x-wiki tgrdozw7uambmqw6bkpevoduu2zqcq5 2437195 2437194 2026-07-15T16:50:41Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2357-2365].codfw.wmnet 2437195 wikitext text/x-wiki 4yk4gx9lvk6bfjbyhpw3rt0wol7ojjm 2437196 2437195 2026-07-15T16:51:38Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2437196 wikitext text/x-wiki euhxq0jslbfz3ai0gk8o0fab3r13xy8 2437197 2437196 2026-07-15T16:52:17Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2437197 wikitext text/x-wiki 26y76v3yppzio8i6mpc2s3ajcsmtyw2 2437198 2437197 2026-07-15T16:52:47Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet 2437198 wikitext text/x-wiki alvik7yzjr0z9ff4fyz8n8snftrhzzb 2437199 2437198 2026-07-15T16:52:56Z Stashbot 7414 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2437199 wikitext text/x-wiki oaeggso1z68aehaxvnvg8teg93lparr 2437200 2437199 2026-07-15T16:53:22Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet 2437200 wikitext text/x-wiki 9vh5x0svl8nh5qbc8bius6upsjowsxh 2437201 2437200 2026-07-15T16:55:52Z Stashbot 7414 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2437201 wikitext text/x-wiki hpmo3l80xnyjo8ka84uqo5dwu8w2132 2437202 2437201 2026-07-15T16:55:59Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2357-2365].codfw.wmnet 2437202 wikitext text/x-wiki 0medu7b0l3qtpawya8go2waju98qa0l 2437204 2437202 2026-07-15T16:57:24Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-search: apply 2437204 wikitext text/x-wiki e0mqig2neujbqolgtz190pxszg4flaz 2437205 2437204 2026-07-15T16:57:30Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-search: apply 2437205 wikitext text/x-wiki l1esaf8suryei7aqzcyxun5pwvuop5s 2437206 2437205 2026-07-15T16:57:37Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply 2437206 wikitext text/x-wiki ocokqpn4ntviblioxu2kratvs6azl78 2437207 2437206 2026-07-15T16:57:44Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply 2437207 wikitext text/x-wiki afp8akds1wr467tnuysjcy5j2iwowlg 2437208 2437207 2026-07-15T17:03:05Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply 2437208 wikitext text/x-wiki c87v7masvsr9agbgx7slb79r9lqyxl1 2437209 2437208 2026-07-15T17:03:06Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2357-2365].codfw.wmnet 2437209 wikitext text/x-wiki najddwt0ht9vt3rm67891382wwfs38x 2437210 2437209 2026-07-15T17:03:10Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply 2437210 wikitext text/x-wiki l3jb7unpd8uacj25lds46g1xaxqti6a 2437211 2437210 2026-07-15T17:03:12Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2357-2365].codfw.wmnet 2437211 wikitext text/x-wiki ksuud69l0gh6zchnfbgqqoa75wv605q 2437212 2437211 2026-07-15T17:03:16Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-wikidata: apply 2437212 wikitext text/x-wiki 27mu6jrzl1h5mezim8209ce39onnshp 2437213 2437212 2026-07-15T17:03:22Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-wikidata: apply 2437213 wikitext text/x-wiki nw1fwnbkfuzcn278tachhgeucs3n6lg 2437214 2437213 2026-07-15T17:03:27Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-wmde: apply 2437214 wikitext text/x-wiki ib5m3coqvdj80r2mh9i7hjfj5oqyuay 2437215 2437214 2026-07-15T17:03:29Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2366-2374].codfw.wmnet 2437215 wikitext text/x-wiki fgabd31rikejunna8euqp8u3u3nxzkd 2437216 2437215 2026-07-15T17:03:35Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-wmde: apply 2437216 wikitext text/x-wiki i78twz5s0p9afw25twf36qfrt46iw6o 2437217 2437216 2026-07-15T17:03:39Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply 2437217 wikitext text/x-wiki qlw28yrtbqusdz7t2x12vij2hzmz1e9 2437218 2437217 2026-07-15T17:03:45Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply 2437218 wikitext text/x-wiki hxmnqf61cc5vdhfxvgkfx02gqcenzsf 2437219 2437218 2026-07-15T17:03:49Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Security update 2437219 wikitext text/x-wiki ksnkm47v66t9f7e6771oml3e6iz1p7a 2437220 2437219 2026-07-15T17:04:42Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Security update 2437220 wikitext text/x-wiki fe1ln0wc2dzghmh9s075fxgo7r62sv0 2437221 2437220 2026-07-15T17:06:04Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns5003.wikimedia.org 2437221 wikitext text/x-wiki mg7mszul1fyh4n3mss4oi2quc140tz6 2437222 2437221 2026-07-15T17:06:08Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-reboot (exit_code=97) rolling reboot on A:dnsbox and not (A:ulsfo or A:magru) and (A:dnsbox) 2437222 wikitext text/x-wiki 3mgy762vyow4k6u5appvdrqkpvqy37x 2437223 2437222 2026-07-15T17:06:20Z Stashbot 7414 sukhe: sre.dns.roll-reboot to resume later 2437223 wikitext text/x-wiki 24xs6w9w4lp0spyqsa8o06bo7qhuce0 2437225 2437223 2026-07-15T17:08:49Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2366-2374].codfw.wmnet 2437225 wikitext text/x-wiki lux1h7sk1dkxraucdbb7lyinr86ot9f 2437226 2437225 2026-07-15T17:10:02Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply 2437226 wikitext text/x-wiki gn8jrgrs2hn0e55jdfo48ac1sogivgo 2437227 2437226 2026-07-15T17:10:08Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply 2437227 wikitext text/x-wiki o6x75wqx47idgfojf00v8csp689bkj6 2437229 2437227 2026-07-15T17:15:36Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2366-2374].codfw.wmnet 2437229 wikitext text/x-wiki lcnfk6lnzdd1wmb24g1i6grksiqfhs4 2437230 2437229 2026-07-15T17:15:41Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2366-2374].codfw.wmnet 2437230 wikitext text/x-wiki 6ze5qm6jjoh315xh6kru0qgide9mu0x 2437231 2437230 2026-07-15T17:15:44Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp2001.codfw.wmnet 2437231 wikitext text/x-wiki 52wkxtdpgytqw2mdxjiyrgmduqhk24m 2437232 2437231 2026-07-15T17:16:18Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp2001.codfw.wmnet 2437232 wikitext text/x-wiki 2u88e7pfidhzu9snkyg40kfe3h7p4yq 2437233 2437232 2026-07-15T17:17:19Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet 2437233 wikitext text/x-wiki mxwri7i2lx1icu2i567k08cvak273ec 2437234 2437233 2026-07-15T17:17:20Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet 2437234 wikitext text/x-wiki dcedjrljl6f394ugfupvvo3sj6ftvbv 2437235 2437234 2026-07-15T17:19:53Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Security update 2437235 wikitext text/x-wiki d18cam2d7wkw3ugh6gm8kr6peu61pyv 2437236 2437235 2026-07-15T17:22:02Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437236 wikitext text/x-wiki fm70ojy6j6dls2lnbnvdd3wvgzoq8tk 2437237 2437236 2026-07-15T17:22:16Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp2001.codfw.wmnet 2437237 wikitext text/x-wiki pjm988czws8x5mkb8w9ol7iw5k74aci 2437238 2437237 2026-07-15T17:22:17Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437238 wikitext text/x-wiki 2oyt0zgazj0ykipng53jh9gfqk3us1h 2437239 2437238 2026-07-15T17:22:19Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp2001.codfw.wmnet 2437239 wikitext text/x-wiki top2sf978xfui1nfuo6g4kf1k6bzhm5 2437240 2437239 2026-07-15T17:22:20Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-codfw 2437240 wikitext text/x-wiki fb1l187db6kofgc780z9lp0c3avlvha 2437241 2437240 2026-07-15T17:22:24Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437241 wikitext text/x-wiki ah4rqeg7jddpnn9s39ndissg5e2vb74 2437242 2437241 2026-07-15T17:25:50Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437242 wikitext text/x-wiki 8r1ohm9l6u6tuh24f3nd6bz3ay7ppp8 2437243 2437242 2026-07-15T17:27:44Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437243 wikitext text/x-wiki avggjvxrh557z9pt3gzm97liuvwvonh 2437244 2437243 2026-07-15T17:30:21Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4043.ulsfo.wmnet 2437244 wikitext text/x-wiki 36juctqvnk9p3etu6ebhobnsw3t00qk 2437246 2437244 2026-07-15T17:31:28Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply 2437246 wikitext text/x-wiki fm9cdmd9352z66lehmbx71xlf0s3eld 2437247 2437246 2026-07-15T17:32:34Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply 2437247 wikitext text/x-wiki flf1pxaeckr1dg5gjbrn2gsy2iz7dhy 2437248 2437247 2026-07-15T17:33:39Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6015.drmrs.wmnet 2437248 wikitext text/x-wiki jbqqkkvl4wyykijdqbik5tp3gloh9py 2437249 2437248 2026-07-15T17:34:35Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet 2437249 wikitext text/x-wiki cs6y3b8hl7bi2pgv7b4zeiwo1kjlqyz 2437250 2437249 2026-07-15T17:39:54Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply 2437250 wikitext text/x-wiki ryg7kpy8msgr1d7ceeuyeuqtswbow1t 2437251 2437250 2026-07-15T17:40:28Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply 2437251 wikitext text/x-wiki hhaayb2ynxe1aqvqe8sho0cyp8cwxzo 2437252 2437251 2026-07-15T17:42:24Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4051.ulsfo.wmnet 2437252 wikitext text/x-wiki 911cxgpro74cutzmk7whsru2s8y737g 2437253 2437252 2026-07-15T17:47:27Z Stashbot 7414 cdobbins@cumin2003: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS trixie 2437253 wikitext text/x-wiki 2e2fw7pl62h49djjou081f613a2a9un 2437254 2437253 2026-07-15T17:56:39Z Stashbot 7414 topranks: start draining traffic on cr1-eqiad ahead of line card installation T426343 2437254 wikitext text/x-wiki 76vj0vmnervdgk3mugtmrebc84vbsnn 2437255 2437254 2026-07-15T18:05:25Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Security update 2437255 wikitext text/x-wiki miw5shryl74oytlcj9zrsqxmmko9zqi 2437256 2437255 2026-07-15T18:10:11Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_drmrs 2437256 wikitext text/x-wiki 4uoojieiuzfm8uyrotei5lxu906oi82 2437257 2437256 2026-07-15T18:12:01Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4044.ulsfo.wmnet 2437257 wikitext text/x-wiki caeemsa5t012em4hgj3juvics5ti36h 2437258 2437257 2026-07-15T18:12:03Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_ulsfo 2437258 wikitext text/x-wiki re4e4ojadspbth7jf3bfiqyv0x4jbey 2437259 2437258 2026-07-15T18:14:43Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet 2437259 wikitext text/x-wiki hrkmlcvq1v0la7on14vuhniyrx1jja3 2437260 2437259 2026-07-15T18:14:45Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_drmrs 2437260 wikitext text/x-wiki dtqelrczmbgie95l2k7286669bprb0w 2437261 2437260 2026-07-15T18:15:36Z Stashbot 7414 cdobbins@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage 2437261 wikitext text/x-wiki gyryc33xg9xrpsg8g5rrgxgcbqofgwh 2437262 2437261 2026-07-15T18:19:50Z Stashbot 7414 cdobbins@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage 2437262 wikitext text/x-wiki jc98m1day4xwp6qe4nqz9mnrz2hdfxe 2437263 2437262 2026-07-15T18:22:12Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp4052.ulsfo.wmnet 2437263 wikitext text/x-wiki 7tnjmtlyzmydbnfnpw1q725ljk4871a 2437264 2437263 2026-07-15T18:22:13Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_ulsfo 2437264 wikitext text/x-wiki mvwzghuieri1dkeomq6m22xlkaywrrd 2437265 2437264 2026-07-15T18:27:32Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 14 hosts with reason: linecard install in cr1-eqad 2437265 wikitext text/x-wiki lmdhflk87rakm6ssi9vawjrx9csbrk2 2437266 2437265 2026-07-15T18:29:16Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs[1018-1020].eqiad.wmnet with reason: line card install in cr1-eqiad 2437266 wikitext text/x-wiki eod9og1mecmyejijw4xgk43ia6e1pnv 2437267 2437266 2026-07-15T18:29:38Z Stashbot 7414 topranks: pull power on cr1-eqiad to install new switch-control boards T426343 2437267 wikitext text/x-wiki 93ehe0joosiwwdpfd9tfeku62s0pkol 2437268 2437267 2026-07-15T18:42:14Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host ncmonitor1001.eqiad.wmnet 2437268 wikitext text/x-wiki 95kjjxmsu72bchrzhqkjb971lk16pkj 2437269 2437268 2026-07-15T18:44:13Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1262.eqiad.wmnet 2437269 wikitext text/x-wiki 7x1fwc0ebgm2p0bf0rul48uq7dnq92q 2437270 2437269 2026-07-15T18:44:18Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1262.eqiad.wmnet 2437270 wikitext text/x-wiki a1z6pn0qij3dkgvydw5i495rmcfy2uh 2437271 2437270 2026-07-15T18:45:13Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1262.eqiad.wmnet 2437271 wikitext text/x-wiki 6qq0r95dnrlvwm9w8361021e3yvjf4i 2437272 2437271 2026-07-15T18:45:30Z Stashbot 7414 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'. 2437272 wikitext text/x-wiki fjxatlcytmqrgrf41wa99zfp83win43 2437273 2437272 2026-07-15T18:45:34Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1262.eqiad.wmnet with OS trixie 2437273 wikitext text/x-wiki agc5p5w2a6e2lx51joa4rlscrvrvyyt 2437274 2437273 2026-07-15T18:46:00Z Stashbot 7414 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2437274 wikitext text/x-wiki cf6evqu90ff9hi2szkj9hcnx77ut9z9 2437275 2437274 2026-07-15T18:46:04Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ncmonitor1001.eqiad.wmnet 2437275 wikitext text/x-wiki 0pkje8ssxieniiy4dxyp9ax1j4ti1qi 2437276 2437275 2026-07-15T18:46:11Z Stashbot 7414 jhuneidi@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.11 refs T430830 2437276 wikitext text/x-wiki 02o7u3gho540rjy9hfyz5xkyz4gxe0d 2437277 2437276 2026-07-15T18:46:14Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1262 2437277 wikitext text/x-wiki du0dlokl9s42y6jlmvs14vn9lai2ata 2437278 2437277 2026-07-15T18:46:48Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.netbox 2437278 wikitext text/x-wiki qnzr82r2l4bqw72tdv7cphch1lb7ctf 2437279 2437278 2026-07-15T18:51:37Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1262 - kamila@cumin1003" 2437279 wikitext text/x-wiki tb2jcssdupn9b9ibpdlbicj30xyje8u 2437280 2437279 2026-07-15T18:51:41Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1262 - kamila@cumin1003" 2437280 wikitext text/x-wiki 38cvry42g8fhs26gkpw6mbke8frcqca 2437281 2437280 2026-07-15T18:51:42Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2437281 wikitext text/x-wiki 4kbh88vs33dvmmzobndr8kbughbj7bm 2437282 2437281 2026-07-15T18:51:44Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1262.eqiad.wmnet 72.32.64.10.in-addr.arpa 2.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437282 wikitext text/x-wiki 07sglfmcrhm04x0bnp2glxuj4aa4mlq 2437283 2437282 2026-07-15T18:51:45Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1262.eqiad.wmnet 72.32.64.10.in-addr.arpa 2.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437283 wikitext text/x-wiki otd39448yw0hkclet2ix1me8zelxn3t 2437284 2437283 2026-07-15T18:51:46Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1262 2437284 wikitext text/x-wiki r6cz4f1jw5b7cjc9z4jpqji3p02dpg5 2437285 2437284 2026-07-15T18:52:27Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1262 2437285 wikitext text/x-wiki 39pnvgiuwl18iky4bi2ewqn7paksmxi 2437286 2437285 2026-07-15T18:52:29Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1262 2437286 wikitext text/x-wiki rmus8pqf4rtn32ab3zk8bju3j8eqthx 2437287 2437286 2026-07-15T18:53:54Z Stashbot 7414 dcausse@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply 2437287 wikitext text/x-wiki h6pn2bg817046emr93sdgcu4uyvuy79 2437288 2437287 2026-07-15T18:54:10Z Stashbot 7414 dcausse@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437288 wikitext text/x-wiki 1pnunfz00hotue4v5e0d66sjia32hzv 2437290 2437288 2026-07-15T18:59:53Z Stashbot 7414 cdobbins@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7002.wikimedia.org with OS trixie 2437290 wikitext text/x-wiki ttkzhzh4qc4u27ntay8tvxqxnx0rn5c 2437291 2437290 2026-07-15T19:11:42Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1262.eqiad.wmnet with reason: host reimage 2437291 wikitext text/x-wiki 3t0eh5s3519a0j1rwn78su5u2kljzem 2437292 2437291 2026-07-15T19:17:20Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1262.eqiad.wmnet with reason: host reimage 2437292 wikitext text/x-wiki 8qijtyupgsyoveopwxb4cqa1o2vd1gf 2437293 2437292 2026-07-15T19:38:44Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1262.eqiad.wmnet with OS trixie 2437293 wikitext text/x-wiki 4ymeuqnmitofobviro4qco0tbb8fgky 2437294 2437293 2026-07-15T19:44:15Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host relforge1010.eqiad.wmnet 2437294 wikitext text/x-wiki j89rulbmcs7a7o1z1c3ikn6dnty6yl7 2437295 2437294 2026-07-15T19:45:27Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2437295 wikitext text/x-wiki 0ahumd0jvavkaammi40gsxn30l00om4 2437296 2437295 2026-07-15T19:45:29Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm 2437296 wikitext text/x-wiki jpgb9lwvseatcp0z6byl9xciq4lamzc 2437297 2437296 2026-07-15T19:46:11Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2437297 wikitext text/x-wiki r6duayea422pnk8s31e5io2s6z0vgzs 2437298 2437297 2026-07-15T19:47:50Z Stashbot 7414 arlolra@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2437298 wikitext text/x-wiki md8b3xodnk4o571grx06gv82ryzn096 2437299 2437298 2026-07-15T19:48:20Z Stashbot 7414 arlolra@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2437299 wikitext text/x-wiki 35185vnzo0bt1dgu906p8ro5ayz4hie 2437300 2437299 2026-07-15T19:48:22Z Stashbot 7414 arlolra@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2437300 wikitext text/x-wiki efxqaubutab7mld4i4jbs67fkmvxqm1 2437301 2437300 2026-07-15T19:48:46Z Stashbot 7414 arlolra@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2437301 wikitext text/x-wiki g51yepl4fr6qmgep2q9tw90fcl46xwj 2437302 2437301 2026-07-15T19:52:38Z Stashbot 7414 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply 2437302 wikitext text/x-wiki nmxwzqt9ijb3m89ghkd4t911nk3n9j6 2437303 2437302 2026-07-15T19:52:49Z Stashbot 7414 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437303 wikitext text/x-wiki mhdx8vcbu43pv2yal7l5wyyove706o6 2437304 2437303 2026-07-15T19:55:10Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host relforge1010.eqiad.wmnet 2437304 wikitext text/x-wiki nrmziiviokoubfrcof4w99s2sboem03 2437305 2437304 2026-07-15T20:01:41Z Stashbot 7414 jhathaway: disabling puppet fleet wide to roll out kafka patch 2437305 wikitext text/x-wiki 0shy6bh8hd2ix6h54k04vxuqzkeloz6 2437306 2437305 2026-07-15T20:02:16Z Stashbot 7414 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply 2437306 wikitext text/x-wiki gtod1207ivpuyfls8y3nyh43l4pfd4f 2437307 2437306 2026-07-15T20:02:21Z Stashbot 7414 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437307 wikitext text/x-wiki 02bcr2cqqxs8hzn7eqe3nrv5o6liqc8 2437308 2437307 2026-07-15T20:03:30Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs2001.codfw.wmnet with OS bookworm 2437308 wikitext text/x-wiki 0wz72dwewy2wmub073bw6ymwac9qxrd 2437309 2437308 2026-07-15T20:03:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm 2437309 wikitext text/x-wiki dngpa4rqqkguj1wp00q68nnaie313e5 2437310 2437309 2026-07-15T20:04:29Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs2001'] 2437310 wikitext text/x-wiki qxkjag3t5v1nzxlzsbrbsaak0ps28yk 2437311 2437310 2026-07-15T20:05:09Z Stashbot 7414 dcausse@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2437311 wikitext text/x-wiki 9pt3q4k53f2dkwn93pum1pazu4mm5p6 2437312 2437311 2026-07-15T20:05:17Z Stashbot 7414 dcausse@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437312 wikitext text/x-wiki gbw3o6z59xt7nbk7hsvkd40zo8uekn6 2437313 2437312 2026-07-15T20:08:49Z Stashbot 7414 dcausse@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2437313 wikitext text/x-wiki 8l1fry0q4mw7jqyfrp71w7l93bm65km 2437314 2437313 2026-07-15T20:09:00Z Stashbot 7414 dcausse@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437314 wikitext text/x-wiki sqmhbriluvjgx92630tbxocm95114vp 2437315 2437314 2026-07-15T20:11:29Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs2001'] 2437315 wikitext text/x-wiki csqkwcp0e1q0ek1foyjdv4de9bfn1gs 2437316 2437315 2026-07-15T20:12:17Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs2001'] 2437316 wikitext text/x-wiki pr2n8upl4i9bzw31tisfjbplz7jbihh 2437317 2437316 2026-07-15T20:17:26Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs2001'] 2437317 wikitext text/x-wiki 34zwwqjvonkxaowql1ir2uuy636lzx5 2437318 2437317 2026-07-15T20:18:05Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2437318 wikitext text/x-wiki dqsu0xfzi6ng4lk76wnezhooh09n87t 2437319 2437318 2026-07-15T20:21:36Z Stashbot 7414 jhathaway: puppet is re-enabled, have fun, but not too much fun! 2437319 wikitext text/x-wiki dzsdwwfgghiwx66tcgtdvw669j207y0 2437320 2437319 2026-07-15T20:45:44Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS bookworm 2437320 wikitext text/x-wiki t09lt0a7l3injevktaj93q6x2z6haka 2437321 2437320 2026-07-15T20:55:11Z Stashbot 7414 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2437321 wikitext text/x-wiki gu1j6bt76qjiyobc0g2lecjc5y4277s 2437322 2437321 2026-07-15T20:55:53Z Stashbot 7414 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2437322 wikitext text/x-wiki jhcs0z8qwvcydo0nw1c85cuzshfhk22 2437323 2437322 2026-07-15T20:59:37Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad 2437323 wikitext text/x-wiki 4zrdyxqqkfzxbbcfoweuvrgcfigp5vi 2437324 2437323 2026-07-15T20:59:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet 2437324 wikitext text/x-wiki faisylfs7eel0ixe13k79s0f5a3cjhw 2437325 2437324 2026-07-15T20:59:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet 2437325 wikitext text/x-wiki 340qovko8oa2pqcidut339chc7bdqgb 2437326 2437325 2026-07-15T21:00:39Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm 2437326 wikitext text/x-wiki 2uyblscue59vfhb7qydkh4sermmh2rk 2437327 2437326 2026-07-15T21:01:35Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2437327 wikitext text/x-wiki qv7h57dmiffh33t2mjmpzgld9kfa4gs 2437328 2437327 2026-07-15T21:01:42Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm 2437328 wikitext text/x-wiki knp8jplim6zozr6twnaujr3da0ql6zq 2437329 2437328 2026-07-15T21:02:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2437329 wikitext text/x-wiki 1pl3dssb76yal1232wuddbnptgao9qi 2437330 2437329 2026-07-15T21:04:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-druid-public cluster: Roll restart of jvm daemons. 2437330 wikitext text/x-wiki n7qb3840mxv8bivo9ef6eangrdsk40g 2437331 2437330 2026-07-15T21:05:02Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet 2437331 wikitext text/x-wiki cvut3qn832kh9t0q8o9jklt0sup8e8i 2437332 2437331 2026-07-15T21:05:03Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet 2437332 wikitext text/x-wiki 8yf1brvj693jdiu0rzt6iwvmpappw7m 2437333 2437332 2026-07-15T21:05:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet 2437333 wikitext text/x-wiki 4mskw6ysxomgfkoegsh8di00w09vka2 2437334 2437333 2026-07-15T21:05:44Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet 2437334 wikitext text/x-wiki f6e3u3fnfligsver2blf2q3q4tefs7t 2437335 2437334 2026-07-15T21:08:34Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] 2437335 wikitext text/x-wiki ioejkqytwfcwea6tk6yxi7izpevyn1g 2437336 2437335 2026-07-15T21:11:02Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-druid-public cluster: Roll restart of jvm daemons. 2437336 wikitext text/x-wiki e7ng561maiu5i14f9gdketrwn03be1q 2437337 2437336 2026-07-15T21:11:07Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet 2437337 wikitext text/x-wiki 56k7798kcy13zxd1lb472qkr3112n88 2437338 2437337 2026-07-15T21:11:08Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet 2437338 wikitext text/x-wiki ssn1wgswdnodav4gpsidbragor9mbfh 2437339 2437338 2026-07-15T21:11:15Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet 2437339 wikitext text/x-wiki kdt4vpmq0eenl3s8hkfrhjx1e46ybtf 2437340 2437339 2026-07-15T21:11:45Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet 2437340 wikitext text/x-wiki byeyzp3hhc6xaku534n1s0168ku3gv6 2437341 2437340 2026-07-15T21:15:45Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs1002'] 2437341 wikitext text/x-wiki h8t4j1o5hwsmbeo2kzj93f3i7hcynn3 2437342 2437341 2026-07-15T21:16:15Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] 2437342 wikitext text/x-wiki ar33eu7lxpbvoh9ky43qzf5uo4ec92q 2437343 2437342 2026-07-15T21:17:04Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet 2437343 wikitext text/x-wiki qr5uhor8v95l782i2m6svfqja982rl2 2437344 2437343 2026-07-15T21:17:05Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet 2437344 wikitext text/x-wiki 31tx16766t2nv7reab81c6xb4zgekwq 2437345 2437344 2026-07-15T21:17:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet 2437345 wikitext text/x-wiki 7zyg2xjyf2gbzg7tfmtmj54xehxz024 2437346 2437345 2026-07-15T21:17:11Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet 2437346 wikitext text/x-wiki 8v5gtxiz06jt1zhmt2chquwaugse1uy 2437347 2437346 2026-07-15T21:18:04Z Stashbot 7414 swfrench-wmf: reprepro include php8.3_8.3.32-1+wmf11u2 into component/php83 for bullseye-wikimedia 2437347 wikitext text/x-wiki k94rn88zn5xht2a0yb6xbdf7wbc92ab 2437350 2437347 2026-07-15T21:20:52Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs1002'] 2437350 wikitext text/x-wiki qkdscnl0usr8tkkjx3xsotyzfywm4kf 2437351 2437350 2026-07-15T21:22:03Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm 2437351 wikitext text/x-wiki 9vfw7nzugkzplih2ajyvk8pn37fu7l1 2437352 2437351 2026-07-15T21:23:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet 2437352 wikitext text/x-wiki r7gxgyv3eaef6wa4nxp5z67b7aawf9c 2437353 2437352 2026-07-15T21:23:25Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet 2437353 wikitext text/x-wiki fh4c8d93hrpjtel5yr5u9goqcwikvm5 2437354 2437353 2026-07-15T21:23:29Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet 2437354 wikitext text/x-wiki sofixojmdlerusp4jymx4ftysy9h3vi 2437355 2437354 2026-07-15T21:28:27Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs2001.codfw.wmnet with OS bookworm 2437355 wikitext text/x-wiki boionj7kfopaxs9p62d1mlbtxg7xyi3 2437356 2437355 2026-07-15T21:28:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2437356 wikitext text/x-wiki ldhm24oyfcwo59cxsfbeiy6803zriy9 2437357 2437356 2026-07-15T21:29:26Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs1002.eqiad.wmnet with OS bookworm 2437357 wikitext text/x-wiki fe40guor14oe2qeqmpsjnz7eemuxhad 2437358 2437357 2026-07-15T21:29:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] 2437358 wikitext text/x-wiki 9tsrr242h3lc8l5q8uvzeaj4hlf9dm5 2437359 2437358 2026-07-15T21:29:48Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs1002'] 2437359 wikitext text/x-wiki q8avf8b134jlrw7rnakzqp17z8uxa6a 2437360 2437359 2026-07-15T21:29:58Z Stashbot 7414 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437360 wikitext text/x-wiki 4rb045g8r6g28rcgl2vosuv1ryuveqd 2437361 2437360 2026-07-15T21:30:12Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] 2437361 wikitext text/x-wiki q3muw4zcdw1s5d0pbe45iizf6h1x3l0 2437362 2437361 2026-07-15T21:30:25Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['wcqs1002'] 2437362 wikitext text/x-wiki lxhqovlg5nqem0yrkhjotpu0zov9yk4 2437363 2437362 2026-07-15T21:30:43Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] 2437363 wikitext text/x-wiki pcuk7hl88hweg735t2o70h7oqknopds 2437364 2437363 2026-07-15T21:38:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs1002'] 2437364 wikitext text/x-wiki 17obji0sb31951dei6fte66qs05ath1 2437365 2437364 2026-07-15T21:43:29Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm 2437365 wikitext text/x-wiki jvd6nxyfl669addtn6kivmw5dccxsml 2437367 2437365 2026-07-15T21:50:07Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs1002.eqiad.wmnet with OS bookworm 2437367 wikitext text/x-wiki ij7456jmfjxz24bry72cu5s4tbnewvx 2437368 2437367 2026-07-15T21:50:27Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS trixie 2437368 wikitext text/x-wiki 4nf547o4yakexw73ais5z1uhjgssjwy 2437369 2437368 2026-07-15T21:50:30Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply 2437369 wikitext text/x-wiki qgep9jhsubqbxkx2njt6v6wwj88anmm 2437370 2437369 2026-07-15T21:52:15Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply 2437370 wikitext text/x-wiki 6h0po07phjfnwcrdm5bu9zaek2t9vjq 2437371 2437370 2026-07-15T21:53:43Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet 2437371 wikitext text/x-wiki dfxw8pjerrqxdgi6c2zenfern7ff7eg 2437372 2437371 2026-07-15T22:00:32Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet 2437372 wikitext text/x-wiki jixkv1x1qnzaw84b96d816lw9w9yrc6 2437373 2437372 2026-07-15T22:00:34Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet 2437373 wikitext text/x-wiki jb6okaext2qyaltfdr5rv5f5j9prf69 2437374 2437373 2026-07-15T22:00:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet 2437374 wikitext text/x-wiki s036qqo7th38do3k39srda50ueldds5 2437375 2437374 2026-07-15T22:01:16Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet 2437375 wikitext text/x-wiki hxmb03zqkfzgvxcr7chsrfk08x0sk1z 2437376 2437375 2026-07-15T22:01:30Z Stashbot 7414 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on datahubsearch[1001-1003].eqiad.wmnet with reason: Using datahubsearch1001 to test bookworm reimages 2437376 wikitext text/x-wiki 5lv3ari74jp0949mhduidcsczsl7mae 2437377 2437376 2026-07-15T22:02:56Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs2001.codfw.wmnet with OS bookworm 2437377 wikitext text/x-wiki ti4g4dhddi6ywqi0h2mk3gpklofvnf6 2437378 2437377 2026-07-15T22:05:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage 2437378 wikitext text/x-wiki hinbhf8z9q4scgdvg6rwzqarwgj06sd 2437379 2437378 2026-07-15T22:05:20Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host datahubsearch1001.eqiad.wmnet with OS bookworm 2437379 wikitext text/x-wiki dt3ahys1a3ui0slyi9kktd75meb44ow 2437380 2437379 2026-07-15T22:08:02Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet 2437380 wikitext text/x-wiki pkjc9kssoyxgi4xoxwou548ig2km4r8 2437381 2437380 2026-07-15T22:08:04Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet 2437381 wikitext text/x-wiki h49cm59rdh0ph4brrx4hdocii9hhjmo 2437382 2437381 2026-07-15T22:08:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet 2437382 wikitext text/x-wiki ivv7kse132factujbyg0cgckix24kla 2437383 2437382 2026-07-15T22:08:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet 2437383 wikitext text/x-wiki 55wr1gvvube28i8m8l4igyy2wj06g19 2437384 2437383 2026-07-15T22:09:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage 2437384 wikitext text/x-wiki 7w83tycy472p2p4erqs8pmukx4js3vu 2437385 2437384 2026-07-15T22:15:00Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet 2437385 wikitext text/x-wiki 3ho2y9jte6lkezg1dj9jolpdtntnphc 2437386 2437385 2026-07-15T22:15:01Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet 2437386 wikitext text/x-wiki ljvadboxe0dswgcroso3luoa5wpsgw0 2437387 2437386 2026-07-15T22:15:07Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet 2437387 wikitext text/x-wiki 5caldsu19ubg2ca98g4wfnqgxiayd3g 2437388 2437387 2026-07-15T22:16:06Z Stashbot 7414 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on datahubsearch[1002-1003].eqiad.wmnet with reason: Using datahubsearch1001 to test bookworm reimages 2437388 wikitext text/x-wiki mvwir8cw8fizflcn1kby2g60cfn8b4h 2437389 2437388 2026-07-15T22:34:03Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host datahubsearch1001.eqiad.wmnet with OS bookworm 2437389 wikitext text/x-wiki d2epa74qyth1rnh0737cm809xt19e1x 2437390 2437389 2026-07-15T22:34:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host datahubsearch1001.eqiad.wmnet with OS bookworm 2437390 wikitext text/x-wiki 4c8yjtqqb9hvy4ftzyd4it35bydyn31 2437391 2437390 2026-07-15T22:44:01Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs1002.eqiad.wmnet with OS trixie 2437391 wikitext text/x-wiki j19nu4ur0a5tv7t844oy94z3lwwysc3 2437392 2437391 2026-07-15T22:44:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm 2437392 wikitext text/x-wiki hlzfqgbzcskxd68jgiavvcxlzsf9vva 2437393 2437392 2026-07-15T22:45:08Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet 2437393 wikitext text/x-wiki r4cocy5zzvn4ew69pvmzsen59xx3qol 2437394 2437393 2026-07-15T22:51:35Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet 2437394 wikitext text/x-wiki gf4wmlmpui77sf38jfeburqs8m9j5h8 2437395 2437394 2026-07-15T22:51:37Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet 2437395 wikitext text/x-wiki ntki606bjjs3a3goysddai50n26t5pl 2437396 2437395 2026-07-15T22:51:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet 2437396 wikitext text/x-wiki avotjixsivqm1u7b8xiqk1khujjxhfr 2437397 2437396 2026-07-15T22:57:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm 2437397 wikitext text/x-wiki 2uwafst0muomkjww2b5eh5pdr924gkp 2437398 2437397 2026-07-15T23:00:10Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage 2437398 wikitext text/x-wiki kniesp7aqc6mkiwz5p3k8zny26x4k7n 2437399 2437398 2026-07-15T23:04:01Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage 2437399 wikitext text/x-wiki kappbpvvnn25r70bi1j6ybmw8dbpgw4 2437401 2437399 2026-07-15T23:15:15Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2001.codfw.wmnet with reason: host reimage 2437401 wikitext text/x-wiki 774c3gyunyytp02ucqvdlandfgrx5zr 2437402 2437401 2026-07-15T23:15:40Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host datahubsearch1001.eqiad.wmnet with OS bookworm 2437402 wikitext text/x-wiki l0yimnn6diyxocenq36d02tyw9kx02e 2437403 2437402 2026-07-15T23:20:17Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2001.codfw.wmnet with reason: host reimage 2437403 wikitext text/x-wiki la8apx0jgf3oh5pfmsgknnk9ggl7qsc 2437404 2437403 2026-07-15T23:21:46Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet 2437404 wikitext text/x-wiki kjq3vicj06572idu3i8snurbsg89t2f 2437405 2437404 2026-07-15T23:28:51Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs1002.eqiad.wmnet with OS bookworm 2437405 wikitext text/x-wiki ivu7f2dh0fb8x8nkrgkpvxo88901bxf 2437406 2437405 2026-07-15T23:28:56Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet 2437406 wikitext text/x-wiki duxr28fu0vfoy4pfrkzp8gskku8h7sd 2437407 2437406 2026-07-15T23:28:57Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet 2437407 wikitext text/x-wiki 7m1lxo86mickmwtpoizyxc3dtcyywnt 2437408 2437407 2026-07-15T23:29:02Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet 2437408 wikitext text/x-wiki 6l77jowdwxjuc0f7y0r2suin9a46kxi 2437409 2437408 2026-07-15T23:29:34Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet 2437409 wikitext text/x-wiki pa4z3f8gvnp9tigg3zr7ruoxv7r9694 2437410 2437409 2026-07-15T23:36:05Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet 2437410 wikitext text/x-wiki fb1qfruooh0ks3ueaemcpo33kkx615c 2437411 2437410 2026-07-15T23:36:06Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet 2437411 wikitext text/x-wiki l2nm3vpq3523in61svly8ohe68jo1i0 2437412 2437411 2026-07-15T23:36:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet 2437412 wikitext text/x-wiki ctzdcfvn9fejq141t3mwty1b9gma4tq 2437413 2437412 2026-07-15T23:36:44Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet 2437413 wikitext text/x-wiki ehw3pj63cawh9dfxu8fdsi5w6zgnexw 2437414 2437413 2026-07-15T23:43:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet 2437414 wikitext text/x-wiki 8s07jpj1b1lnirxxg5jchdinmhcs65y 2437415 2437414 2026-07-15T23:43:25Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet 2437415 wikitext text/x-wiki 50r069dkxbwflxk2wle5yy5yzmbeh69 2437416 2437415 2026-07-15T23:43:31Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet 2437416 wikitext text/x-wiki 46nctipzlatzf5htks7urshnxfjie2d 2437417 2437416 2026-07-15T23:44:05Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2001.codfw.wmnet with OS bookworm 2437417 wikitext text/x-wiki 8w11nae6cjq553qtwsqivvb8jvpto5l 2437420 2437417 2026-07-16T00:13:33Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet 2437420 wikitext text/x-wiki h4kyqccdcohdbjxczdcy2mmt2q3hx7e 2437421 2437420 2026-07-16T00:20:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet 2437421 wikitext text/x-wiki tuvyj99rh3s529dcfujxj2z83rzk9fk 2437422 2437421 2026-07-16T00:20:26Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet 2437422 wikitext text/x-wiki g9h0vdq305ngfoqnvdd1hpok92zr0h7 2437423 2437422 2026-07-16T00:20:31Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet 2437423 wikitext text/x-wiki 3rbnyk9hr65sbjw99lxlbm7l3e6oz0u 2437424 2437423 2026-07-16T00:50:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet 2437424 wikitext text/x-wiki nir6z5o2zqr7w0037e7xay8ajq6bwbi 2437425 2437424 2026-07-16T00:57:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet 2437425 wikitext text/x-wiki q6ru3yzvsuo86y6d6swq8hbk4z3woie 2437426 2437425 2026-07-16T00:57:11Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet 2437426 wikitext text/x-wiki 0ij23dt3ykb5ynv1cv3a7s2ai7wzmoq 2437427 2437426 2026-07-16T00:57:16Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet 2437427 wikitext text/x-wiki 7cd5hqg1y08u8e4uuafd9xw2r428kmr 2437428 2437427 2026-07-16T00:57:49Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet 2437428 wikitext text/x-wiki 8g3o7kthnkzppq9sasskniabj4l7y32 2437429 2437428 2026-07-16T01:04:15Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet 2437429 wikitext text/x-wiki pm97hk1a48ru9hi144hb04950c9yi4t 2437430 2437429 2026-07-16T01:04:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet 2437430 wikitext text/x-wiki 38cym8k8rjbwd345h4zq32ue80q4m2p 2437431 2437430 2026-07-16T01:04:22Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet 2437431 wikitext text/x-wiki fbfo4qsvmm1r02ozolgovifzg78ld5h 2437432 2437431 2026-07-16T01:04:55Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet 2437432 wikitext text/x-wiki as10oi739rzj4on6aawtgbg0ditewm4 2437433 2437432 2026-07-16T01:07:59Z Stashbot 7414 ryankemper@deploy2003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): scap deploy post bookworm reimage 2437433 wikitext text/x-wiki a4ove9pyxy82dmuvhes7e2144cda995 2437434 2437433 2026-07-16T01:08:07Z Stashbot 7414 ryankemper@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): scap deploy post bookworm reimage (duration: 00m 46s) 2437434 wikitext text/x-wiki jsvvprllgqxh42733e9bhug0ld5atwe 2437435 2437434 2026-07-16T01:08:46Z Stashbot 7414 ryankemper@deploy2003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 scap deploy post bookworm reimage 2437435 wikitext text/x-wiki lqv5au64738upbko4qtwouqrjyt1p8n 2437436 2437435 2026-07-16T01:08:53Z Stashbot 7414 ryankemper@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 scap deploy post bookworm reimage (duration: 00m 23s) 2437436 wikitext text/x-wiki 32qwg571fyw6i73pdu1b0t6092z52in 2437437 2437436 2026-07-16T01:11:01Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet 2437437 wikitext text/x-wiki nce900trq8rqwbsiv06839vma0uw54o 2437438 2437437 2026-07-16T01:11:02Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet 2437438 wikitext text/x-wiki pyjsvat3nxa24kebfnx84z2j0h6ft64 2437439 2437438 2026-07-16T01:11:07Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet 2437439 wikitext text/x-wiki iari40fxn5uq3s5cogab08v4db9a9cz 2437440 2437439 2026-07-16T01:11:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet 2437440 wikitext text/x-wiki rbsxe3nc1e3omuvspcr2sm36wsabtv8 2437441 2437440 2026-07-16T01:17:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet 2437441 wikitext text/x-wiki nqh0aviyislt05nqkervu8jqw0z3z6v 2437442 2437441 2026-07-16T01:17:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet 2437442 wikitext text/x-wiki 83dfo38p3yr3ofa5oo40vnwkfhhaydx 2437443 2437442 2026-07-16T01:17:47Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet 2437443 wikitext text/x-wiki 5alvrtf8kq6c5emeam371wnzlakknii 2437444 2437443 2026-07-16T01:20:14Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430879, restore data on newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1002.eqiad.wmnet, repooling both afterwards 2437444 wikitext text/x-wiki bfslwekasyxpzgc1b2yowun3p0boxbz 2437445 2437444 2026-07-16T01:20:27Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430879, restore data on newly-reimaged host) xfer commons from wcqs2003.codfw.wmnet -> wcqs2001.codfw.wmnet, repooling both afterwards 2437445 wikitext text/x-wiki tg5sdx3bklj5bm9ddshc3i2mnirat1s 2437446 2437445 2026-07-16T01:47:49Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet 2437446 wikitext text/x-wiki fa7i0ex7whxpl8wxrbo9dd8vfqktr87 2437448 2437446 2026-07-16T01:54:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet 2437448 wikitext text/x-wiki 82z3cwass94e1c29jnl1yomykasqoj7 2437449 2437448 2026-07-16T01:54:12Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet 2437449 wikitext text/x-wiki 1yo329koyfeaw0iagdvgm31fz5idvxj 2437450 2437449 2026-07-16T01:54:17Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet 2437450 wikitext text/x-wiki 4v5c87ut5y8n9x682v03969mdxt9ywn 2437451 2437450 2026-07-16T02:00:22Z Stashbot 7414 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image 2437451 wikitext text/x-wiki htrpez70lvtjqww1zm1ef9j96acuo56 2437452 2437451 2026-07-16T02:07:00Z Stashbot 7414 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 37s) 2437452 wikitext text/x-wiki 2sfb5oouu55lmslanfnlzeqez29tczj 2437453 2437452 2026-07-16T02:24:21Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet 2437453 wikitext text/x-wiki 4jgznrprdye5sx8k35ob6kg222pf88b 2437454 2437453 2026-07-16T02:30:19Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet 2437454 wikitext text/x-wiki o2phz974no8u9morjn58anc36no41o4 2437455 2437454 2026-07-16T02:30:20Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet 2437455 wikitext text/x-wiki n3qfl5ej32q3fv3f29oanv1tvdp527w 2437456 2437455 2026-07-16T02:30:25Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet 2437456 wikitext text/x-wiki a1vmixeok8drpuvl2ackxwpq4l0599o 2437458 2437456 2026-07-16T02:35:29Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet 2437458 wikitext text/x-wiki qoue1b06rfygg904ql5tvfyp20e5h1t 2437459 2437458 2026-07-16T02:36:08Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430879, restore data on newly-reimaged host) xfer commons from wcqs2003.codfw.wmnet -> wcqs2001.codfw.wmnet, repooling both afterwards 2437459 wikitext text/x-wiki jzmcvs8kx4qv2nki4l6p2ixov6hwag0 2437460 2437459 2026-07-16T02:36:41Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430879, restore data on newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1002.eqiad.wmnet, repooling both afterwards 2437460 wikitext text/x-wiki 3sz6k5bkpo6yy7awx8aljsmzvqw90e8 2437461 2437460 2026-07-16T02:41:32Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet 2437461 wikitext text/x-wiki 3ilymvd5l3d64ud8ys2tw7ed0s4261t 2437462 2437461 2026-07-16T02:41:33Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet 2437462 wikitext text/x-wiki 9h4leq8ki4maimkbna1wib3b6du146z 2437463 2437462 2026-07-16T02:41:38Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet 2437463 wikitext text/x-wiki kqx77ipre3brgbzqsuyydifkjaepqxk 2437464 2437463 2026-07-16T03:11:43Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet 2437464 wikitext text/x-wiki cnsvkxnnfcn38btban71u5d5htn81yx 2437465 2437464 2026-07-16T03:18:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet 2437465 wikitext text/x-wiki 7dao4x44bpijdfzrqzxyehkxid9qis9 2437466 2437465 2026-07-16T03:18:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet 2437466 wikitext text/x-wiki fl0gok6acai549cqjab75r6cldqzxdg 2437467 2437466 2026-07-16T03:18:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet 2437467 wikitext text/x-wiki 7aq38g22rrs2mz634kznci7cc9w13dm 2437468 2437467 2026-07-16T03:20:46Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet 2437468 wikitext text/x-wiki o030xdlth7dbscrhgtox0j8687qg255 2437469 2437468 2026-07-16T03:38:04Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet 2437469 wikitext text/x-wiki 3gm7fhhtnq692owziqvzzbir2irjn39 2437470 2437469 2026-07-16T03:38:05Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet 2437470 wikitext text/x-wiki a8r5kyr0crf8xkimilfjexgmmfgzdqq 2437471 2437470 2026-07-16T03:38:12Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet 2437471 wikitext text/x-wiki 60vozioknzgmyifzkr88dqnu8eoc60j 2437472 2437471 2026-07-16T04:08:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet 2437472 wikitext text/x-wiki oxh21qeg983p6y1qp6hsiq1xkrpi2wg 2437473 2437472 2026-07-16T04:19:06Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet 2437473 wikitext text/x-wiki ljaw88pd7cjtzj5btrcxq19alywz5gk 2437474 2437473 2026-07-16T04:19:08Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet 2437474 wikitext text/x-wiki 3qfles3j44uolvwp1ba7hlwsu7ee2i0 2437475 2437474 2026-07-16T04:19:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet 2437475 wikitext text/x-wiki 86zombbnsmm8xp3p9etorksox26finc 2437477 2437475 2026-07-16T04:49:19Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet 2437477 wikitext text/x-wiki bp0czptbhihdcshdvs9u7jj5zdblr0y 2437478 2437477 2026-07-16T04:56:31Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet 2437478 wikitext text/x-wiki 95aau3h8dgl5d2osr011bfjztwe7lh4 2437479 2437478 2026-07-16T04:56:32Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet 2437479 wikitext text/x-wiki 83uz2kufqnlvg2o5p37w6jmhkri3sba 2437480 2437479 2026-07-16T04:56:37Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet 2437480 wikitext text/x-wiki sss9x1ygilw2hgpcrokucpqv59n9ijj 2437481 2437480 2026-07-16T05:26:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet 2437481 wikitext text/x-wiki 06tn3pvypeu46532gmgbj35dl10xxd3 2437482 2437481 2026-07-16T05:37:14Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet 2437482 wikitext text/x-wiki cd6y7doilnf6bftatqeves1w7trmcu7 2437483 2437482 2026-07-16T05:37:16Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet 2437483 wikitext text/x-wiki rsn9zsbz9r7yp9fd9locwqw18mg0671 2437484 2437483 2026-07-16T05:37:21Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet 2437484 wikitext text/x-wiki 81wviuzqfyi3r07t6nej8xey1ck2tud 2437485 2437484 2026-07-16T06:07:25Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet 2437485 wikitext text/x-wiki 44nrw5im0zymf853v9cci33hsdua1ww 2437486 2437485 2026-07-16T06:14:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet 2437486 wikitext text/x-wiki ila3sqmi6ix7eqtpgfar8w63vaf577j 2437487 2437486 2026-07-16T06:14:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet 2437487 wikitext text/x-wiki nwvnn08y8pd87aoqf6dtq9s2ehk2ssa 2437488 2437487 2026-07-16T06:14:46Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet 2437488 wikitext text/x-wiki kbenjg3h9xheay5iclx0ic5guzuuqgy 2437489 2437488 2026-07-16T06:44:51Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet 2437489 wikitext text/x-wiki 2vhowq1deifu1n2qq64a2zmqdti3tyw 2437490 2437489 2026-07-16T06:47:03Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437490 wikitext text/x-wiki f7uun5fu2r9g8v6kpbjmt2rwhte63x2 2437491 2437490 2026-07-16T06:47:16Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437491 wikitext text/x-wiki fbt08u0l2y1bgxqgth0mfw5zbganxxc 2437492 2437491 2026-07-16T06:51:49Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet 2437492 wikitext text/x-wiki 53uwaoqva3ug5enf4qo68x4vtubb6ad 2437493 2437492 2026-07-16T06:51:50Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet 2437493 wikitext text/x-wiki mvu56p5tum6jci525hv5a1glfexswcg 2437494 2437493 2026-07-16T06:51:56Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet 2437494 wikitext text/x-wiki hmj09115ipj90fjjch6cd9ok4ct8a42 2437495 2437494 2026-07-16T07:18:55Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS trixie 2437495 wikitext text/x-wiki sp963p5pld6sxv8lnfp3f4tz6djw0hu 2437496 2437495 2026-07-16T07:19:35Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org 2437496 wikitext text/x-wiki 0qef8osgc74hxdw33iiru2hklbxqzuv 2437497 2437496 2026-07-16T07:20:23Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1066.eqiad.wmnet with OS trixie 2437497 wikitext text/x-wiki j8snhhedbysb2qqye1djhw36jlpuqit 2437498 2437497 2026-07-16T07:21:57Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet 2437498 wikitext text/x-wiki 4bhj8x6dv9mznoam9cypnbi9plsj33z 2437499 2437498 2026-07-16T07:28:08Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org 2437499 wikitext text/x-wiki 9c4vxcfuiikun5xufc1c3gxjpm1yb21 2437500 2437499 2026-07-16T07:28:38Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet 2437500 wikitext text/x-wiki l5aq20rjws3dlzqzwprs3914p9e1m68 2437501 2437500 2026-07-16T07:28:39Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet 2437501 wikitext text/x-wiki gqt1lf1uj5xk7x7ffpsftuoeooxe4kn 2437502 2437501 2026-07-16T07:28:44Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet 2437502 wikitext text/x-wiki 0lmjjegtowvp1b83vrgmo4s949ap2fd 2437503 2437502 2026-07-16T07:29:19Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet 2437503 wikitext text/x-wiki ttu8gc86gbqv9mxmbj1fl8s3qm4fsoz 2437504 2437503 2026-07-16T07:35:58Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet 2437504 wikitext text/x-wiki ovz255u3x8m8xu8flsi5rlzgjf3qo8r 2437505 2437504 2026-07-16T07:35:59Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet 2437505 wikitext text/x-wiki jpsexmqx9sdkeap9pauscj106jma270 2437506 2437505 2026-07-16T07:36:04Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet 2437506 wikitext text/x-wiki op5snwqc8elv1iiimn03u1bp40br17x 2437507 2437506 2026-07-16T07:36:37Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet 2437507 wikitext text/x-wiki dt8hb69rjuodb16eie0fzufjpwa5ia6 2437508 2437507 2026-07-16T07:38:11Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage 2437508 wikitext text/x-wiki q7xa3gytki1zftlwd9lbi23lkbqp3qe 2437509 2437508 2026-07-16T07:38:41Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage 2437509 wikitext text/x-wiki flbzxl3ykz17h02rahkp865p2s1ujc0 2437510 2437509 2026-07-16T07:43:37Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet 2437510 wikitext text/x-wiki fy59g89w8m5xfyd80ptaua8nwv0at6y 2437511 2437510 2026-07-16T07:43:39Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet 2437511 wikitext text/x-wiki jgz5oj1l2yuu764lkfdv6b5zx98ghr6 2437512 2437511 2026-07-16T07:43:40Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad 2437512 wikitext text/x-wiki lvnaygezsy8ajbvwcewhj90n8lxft41 2437513 2437512 2026-07-16T07:43:50Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage 2437513 wikitext text/x-wiki ktnf98i8akyzhgjtnax2vezx3bp5wr8 2437514 2437513 2026-07-16T07:44:18Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 T430925 2437514 wikitext text/x-wiki i6x2ay0rtql2c2q2mq2ik0wsee4ajy5 2437515 2437514 2026-07-16T07:45:10Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2241 with weight 0 T430925', diff saved to https://phabricator.wikimedia.org/P94868 and previous config saved to /var/cache/conftool/dbconfig/20260716-074507-cwilliams.json 2437515 wikitext text/x-wiki je79yej5saz78f3premctk8uwp5wzdc 2437517 2437515 2026-07-16T07:47:57Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage 2437517 wikitext text/x-wiki q3qtghnpc6c99btgtwyqd5ckcsbxrp3 2437518 2437517 2026-07-16T07:50:17Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437518 wikitext text/x-wiki k8dyqgxf7l8grq5u2297owiahkuzdad 2437519 2437518 2026-07-16T07:50:28Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2437519 wikitext text/x-wiki kql1fauq2wqzl6sbh9867bul21lw8gw 2437520 2437519 2026-07-16T07:52:22Z Stashbot 7414 cezmunsta: Starting x3 codfw failover from db2162 to db2241 - T430925 2437520 wikitext text/x-wiki c8k4xhlyppgoy84y0b4gh27om254xrm 2437521 2437520 2026-07-16T07:53:17Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2241 to x3 primary T430925', diff saved to https://phabricator.wikimedia.org/P94869 and previous config saved to /var/cache/conftool/dbconfig/20260716-075314-cwilliams.json 2437521 wikitext text/x-wiki asz18bppkqagkbp9b7bq48mm4ac0yt1 2437522 2437521 2026-07-16T07:55:36Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2162 T430925', diff saved to https://phabricator.wikimedia.org/P94870 and previous config saved to /var/cache/conftool/dbconfig/20260716-075530-cwilliams.json 2437522 wikitext text/x-wiki i03rd4c7qkeq5l9qg3f795mp847pmxo 2437523 2437522 2026-07-16T07:56:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2162: Repooling after switchover 2437523 wikitext text/x-wiki dnmp8jk098a1tpfvw6n4oprvhxzbpoo 2437524 2437523 2026-07-16T08:02:00Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2162: Repooling after switchover 2437524 wikitext text/x-wiki 77fqbn2hadyj7ogp87dfv3llas5lwdf 2437525 2437524 2026-07-16T08:05:20Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1066.eqiad.wmnet with OS trixie 2437525 wikitext text/x-wiki tkg27jq2gwqiqy93curcjh1gs7ze3vh 2437526 2437525 2026-07-16T08:10:46Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS trixie 2437526 wikitext text/x-wiki h9852khgbaxpavnw8b358zb4nfe2ces 2437527 2437526 2026-07-16T08:15:50Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-eqiad 2437527 wikitext text/x-wiki e2l5oweyodoqotkt06a9xyzg865tls0 2437528 2437527 2026-07-16T08:16:13Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1035,1038-1040].eqiad.wmnet 2437528 wikitext text/x-wiki p1286zd84zydinri97q9ofhlfianu6q 2437529 2437528 2026-07-16T08:16:23Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1067.eqiad.wmnet with OS trixie 2437529 wikitext text/x-wiki bcgjhl1irozd4eaydyhf0u2kpeexksr 2437533 2437529 2026-07-16T08:21:56Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2064.codfw.wmnet with OS trixie 2437533 wikitext text/x-wiki 4vzvds4p99tdbweu2cjvv7ajr3egy56 2437534 2437533 2026-07-16T08:21:58Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2162: Repooling after switchover 2437534 wikitext text/x-wiki jdjfs23unyig6hi2e2wgccttck0r082 2437535 2437534 2026-07-16T08:26:11Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1035,1038-1040].eqiad.wmnet 2437535 wikitext text/x-wiki h5nxln4q0gjlgdoqkpcauvmtfyop11f 2437536 2437535 2026-07-16T08:33:46Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage 2437536 wikitext text/x-wiki ox07w6iiwp0zrracqxjrn17srdsgo01 2437537 2437536 2026-07-16T08:33:49Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1035,1038-1040].eqiad.wmnet 2437537 wikitext text/x-wiki f22hn836amy3dk5skd9b38nmtp139nn 2437538 2437537 2026-07-16T08:33:52Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1035,1038-1040].eqiad.wmnet 2437538 wikitext text/x-wiki 7ari3s631pn91r68e6dczpxqec5nfjd 2437539 2437538 2026-07-16T08:34:12Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1041-1050].eqiad.wmnet 2437539 wikitext text/x-wiki adj9e8d8tcmi8apsyc0cqouovpornti 2437540 2437539 2026-07-16T08:39:25Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage 2437540 wikitext text/x-wiki 3kp6z1szoml7xxzc6z6qavugcndvkxy 2437541 2437540 2026-07-16T08:41:01Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1041-1050].eqiad.wmnet 2437541 wikitext text/x-wiki e6z69d0l5czuyyiwivgznfj3i9a074v 2437542 2437541 2026-07-16T08:41:48Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2064.codfw.wmnet with reason: host reimage 2437542 wikitext text/x-wiki 5q960o0nk5fxod9tmw6crm1is2ipl02 2437543 2437542 2026-07-16T08:48:10Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2064.codfw.wmnet with reason: host reimage 2437543 wikitext text/x-wiki ecffhn9sg5eklxcne0ig8dx2x47ziji 2437544 2437543 2026-07-16T08:48:29Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host ping1004.eqiad.wmnet 2437544 wikitext text/x-wiki ffpukgwxjlwircm2775k0yzool4ujp5 2437545 2437544 2026-07-16T08:49:01Z Stashbot 7414 dcausse@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply 2437545 wikitext text/x-wiki 8wdinpyiv5lcuhg5c7b6prbv9wb2plm 2437547 2437545 2026-07-16T08:51:18Z Stashbot 7414 dcausse@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437547 wikitext text/x-wiki tflnyk2p67jfiqoyhtl89vpdzmi15nn 2437548 2437547 2026-07-16T08:51:50Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1041-1050].eqiad.wmnet 2437548 wikitext text/x-wiki 7i98qng9yqpd0skj8moasyl6vzeq3lw 2437549 2437548 2026-07-16T08:51:53Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1041-1050].eqiad.wmnet 2437549 wikitext text/x-wiki n0hznfuhwxidxcmui1sxp61c4lan88y 2437550 2437549 2026-07-16T08:52:12Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1004.eqiad.wmnet 2437550 wikitext text/x-wiki b4r616jsprw47834adt30lautdxat4x 2437551 2437550 2026-07-16T08:52:13Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1051-1057,1064-1066].eqiad.wmnet 2437551 wikitext text/x-wiki n3onz2x89nl69bubosmdlyh2i13dimy 2437552 2437551 2026-07-16T08:52:52Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host ping2004.codfw.wmnet 2437552 wikitext text/x-wiki 58g0fm0u0a9e6k8505a949hjm7afzph 2437553 2437552 2026-07-16T08:56:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2004.codfw.wmnet 2437553 wikitext text/x-wiki j5zehqzjkqkjy6bln3uheda3xhganzv 2437554 2437553 2026-07-16T08:57:36Z Stashbot 7414 tappof: bump space for prometheus k8s-dse in eqiad 2437554 wikitext text/x-wiki q83fb959umx2xlfeltbs9rnvsr2l8hc 2437555 2437554 2026-07-16T08:57:40Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1051-1057,1064-1066].eqiad.wmnet 2437555 wikitext text/x-wiki cpxg8b11hulx7lvgedpsm10yog5n5wt 2437556 2437555 2026-07-16T08:59:50Z Stashbot 7414 dcausse@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply 2437556 wikitext text/x-wiki 6vt8bvkndjvzqewbr88tpd5pe1k2i8m 2437557 2437556 2026-07-16T09:00:09Z Stashbot 7414 dcausse@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437557 wikitext text/x-wiki t60g86tcv2aw9wr4kv6ob5v5qix5khu 2437558 2437557 2026-07-16T09:01:20Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1067.eqiad.wmnet with OS trixie 2437558 wikitext text/x-wiki gm6sqkjod33jquvbpqu8ouy6epa9sdd 2437559 2437558 2026-07-16T09:07:28Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2162: Repooling after switchover 2437559 wikitext text/x-wiki 6w84vkijbaelijcpxkyip8r6xki9p6k 2437560 2437559 2026-07-16T09:07:44Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1051-1057,1064-1066].eqiad.wmnet 2437560 wikitext text/x-wiki btys0vywy473282rdasu1bm94094djn 2437561 2437560 2026-07-16T09:07:46Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1051-1057,1064-1066].eqiad.wmnet 2437561 wikitext text/x-wiki lvy8yzfcrycceckrpf32ne1llqsep93 2437562 2437561 2026-07-16T09:08:05Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1067-1076].eqiad.wmnet 2437562 wikitext text/x-wiki gaip39e0haadmwfrjtot1q85xxw4pvk 2437563 2437562 2026-07-16T09:10:29Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2064.codfw.wmnet with OS trixie 2437563 wikitext text/x-wiki as84qgjk54miacmfcrmtbsmo7vwl89e 2437564 2437563 2026-07-16T09:13:32Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1067-1076].eqiad.wmnet 2437564 wikitext text/x-wiki dhwir5nfx67bt009y9vpwri9ikkwg35 2437565 2437564 2026-07-16T09:13:40Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1068.eqiad.wmnet with OS trixie 2437565 wikitext text/x-wiki 7wyeh9s9gdwqjc018hjp2g5wnosugwi 2437566 2437565 2026-07-16T09:20:18Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2065.codfw.wmnet with OS trixie 2437566 wikitext text/x-wiki 7npgljnnmtt4m9jqjvqxja449iiedva 2437567 2437566 2026-07-16T09:21:45Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437567 wikitext text/x-wiki oqku9pz5uv9h6x3rjam71lowla88l93 2437568 2437567 2026-07-16T09:24:07Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437568 wikitext text/x-wiki 29nhbezu97diji5hpfefym7n4g24l0d 2437569 2437568 2026-07-16T09:24:09Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437569 wikitext text/x-wiki rb81m9cdyzr1c1bu04ezagxn44umjtg 2437570 2437569 2026-07-16T09:24:19Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1067-1076].eqiad.wmnet 2437570 wikitext text/x-wiki l1fwtrioxkooamm4i5565jitxhdfkqc 2437571 2437570 2026-07-16T09:24:21Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1067-1076].eqiad.wmnet 2437571 wikitext text/x-wiki mzsf7xrfvunvmopiid189mv8l79u5nz 2437572 2437571 2026-07-16T09:24:41Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1077-1081,1084-1087,1093].eqiad.wmnet 2437572 wikitext text/x-wiki nayhqfcqhyxbdz2f28dohbttpvb5ay7 2437573 2437572 2026-07-16T09:25:28Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437573 wikitext text/x-wiki r4buh23o56lsrzlwbgay7p2d5fbxccs 2437574 2437573 2026-07-16T09:25:31Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437574 wikitext text/x-wiki r4ndnykmfsukbffxw1gn5rmdecdwh8n 2437575 2437574 2026-07-16T09:25:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netbox2003.codfw.wmnet 2437575 wikitext text/x-wiki bnsug8n6ulap7mzymxffsfdbr33vv8s 2437577 2437575 2026-07-16T09:29:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox2003.codfw.wmnet 2437577 wikitext text/x-wiki 0vkibp2xq7z4ujcz0ojav9qusfu60f5 2437578 2437577 2026-07-16T09:30:13Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1077-1081,1084-1087,1093].eqiad.wmnet 2437578 wikitext text/x-wiki kmgyrsvhd2hyl7lkfqmrb7ozlnrlryv 2437580 2437578 2026-07-16T09:37:10Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437580 wikitext text/x-wiki 8ynrgmd7d4cw2u0ioki7hpbta2hueuz 2437581 2437580 2026-07-16T09:39:31Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1077-1081,1084-1087,1093].eqiad.wmnet 2437581 wikitext text/x-wiki 5t8rixz1dqa01epozhbkg2h1wsqoic2 2437582 2437581 2026-07-16T09:39:34Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1077-1081,1084-1087,1093].eqiad.wmnet 2437582 wikitext text/x-wiki 8ysd08rgnui051snj25royvnz5jy8r2 2437583 2437582 2026-07-16T09:39:49Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437583 wikitext text/x-wiki jypg09ezfgucq1qzy9q5t93apzgo7mg 2437584 2437583 2026-07-16T09:39:54Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1094-1095,1113-1120].eqiad.wmnet 2437584 wikitext text/x-wiki 4n1twyb9qkehbp7xrmw7asv7a0mpx15 2437585 2437584 2026-07-16T09:40:05Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437585 wikitext text/x-wiki mtvd2rusc7pkbca2e5c5pe4phz4dsp9 2437586 2437585 2026-07-16T09:40:28Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2065.codfw.wmnet with reason: host reimage 2437586 wikitext text/x-wiki 8nirxtch2rfsvrkilfgkbf34d7fe59t 2437587 2437586 2026-07-16T09:43:25Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2065.codfw.wmnet with reason: host reimage 2437587 wikitext text/x-wiki e8h5egz4444difuae871a19uoq1pfxv 2437588 2437587 2026-07-16T09:46:33Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437588 wikitext text/x-wiki sk7t10vgeokkfjoea7h2v2sxmizerwl 2437589 2437588 2026-07-16T09:46:41Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437589 wikitext text/x-wiki 32wwoivdvlwyojgrrmimzi4yjpcoo06 2437590 2437589 2026-07-16T09:47:11Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437590 wikitext text/x-wiki 8bdec52xtd71jnhqhddonwhcxbu62eg 2437591 2437590 2026-07-16T09:49:54Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1094-1095,1113-1120].eqiad.wmnet 2437591 wikitext text/x-wiki jvqjz949mpvd1cbg9yti4qxj9gd8fr2 2437592 2437591 2026-07-16T09:52:15Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437592 wikitext text/x-wiki 3pn83l1nnvdip0b2yt7644w4qmbqynt 2437593 2437592 2026-07-16T09:57:18Z Stashbot 7414 dcausse@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply 2437593 wikitext text/x-wiki qbdpy10wehi2odwmf70nikt9d63fjf6 2437594 2437593 2026-07-16T09:57:25Z Stashbot 7414 dcausse@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437594 wikitext text/x-wiki 4laj6xn4mkuiuxw43nxxkznvogf7h66 2437595 2437594 2026-07-16T09:58:24Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1068.eqiad.wmnet with reason: host reimage 2437595 wikitext text/x-wiki r9oev6wxu9dpr714i3tx9m78kpg24r6 2437596 2437595 2026-07-16T09:59:45Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437596 wikitext text/x-wiki 8x961eygukpou0sdazksuiiqgwp2tkj 2437597 2437596 2026-07-16T10:00:03Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1094-1095,1113-1120].eqiad.wmnet 2437597 wikitext text/x-wiki 1nmgwpkipv6j9j5q58u4398rbzi1d7v 2437598 2437597 2026-07-16T10:00:06Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1094-1095,1113-1120].eqiad.wmnet 2437598 wikitext text/x-wiki hmyhucawbhqux5lvzwet9cmv2csuzh1 2437599 2437598 2026-07-16T10:00:26Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1121-1130].eqiad.wmnet 2437599 wikitext text/x-wiki ji6bls37an6sp7ditje91pldk9exukf 2437600 2437599 2026-07-16T10:03:31Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437600 wikitext text/x-wiki bvr7ducl21owz17vm655hmqed1jm3sx 2437601 2437600 2026-07-16T10:03:37Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437601 wikitext text/x-wiki nrprawschh2ucilq9b91be2cwebiqfa 2437602 2437601 2026-07-16T10:04:10Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1068.eqiad.wmnet with reason: host reimage 2437602 wikitext text/x-wiki cb19tfi2lui8h9sx45g3zpueatqzj73 2437603 2437602 2026-07-16T10:06:15Z Stashbot 7414 dcausse@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2437603 wikitext text/x-wiki 5i5fuc2jykis22wi42xbxlfgop8t6gb 2437604 2437603 2026-07-16T10:06:24Z Stashbot 7414 dcausse@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2437604 wikitext text/x-wiki cspp51wvhndxq5dc1xv6fvclbov0703 2437605 2437604 2026-07-16T10:06:33Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1121-1130].eqiad.wmnet 2437605 wikitext text/x-wiki gky00czuaek6i94lbrglazfa5lny9zv 2437606 2437605 2026-07-16T10:07:45Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2065.codfw.wmnet with OS trixie 2437606 wikitext text/x-wiki 70dyaiv5m779u9afok1nnodzxwfnh8f 2437607 2437606 2026-07-16T10:15:59Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1121-1130].eqiad.wmnet 2437607 wikitext text/x-wiki rqdonbuhb1x4jby5h46iyvqgje693jt 2437608 2437607 2026-07-16T10:16:01Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1121-1130].eqiad.wmnet 2437608 wikitext text/x-wiki ln1tejng9emot2lcv87y19s5xbort5i 2437609 2437608 2026-07-16T10:16:30Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1131-1140].eqiad.wmnet 2437609 wikitext text/x-wiki nteug55ia2rsut6tqqgvixxusylia5v 2437610 2437609 2026-07-16T10:21:08Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437610 wikitext text/x-wiki 7udkf81byw6qrgr4du92nzicoo2t4d5 2437611 2437610 2026-07-16T10:21:11Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437611 wikitext text/x-wiki g0ipr8ivwziwu5am8dkal7orlkd5v1m 2437612 2437611 2026-07-16T10:21:19Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437612 wikitext text/x-wiki pf6ivtnlrz9dp9hyo5d1oqvkxsgrzs1 2437613 2437612 2026-07-16T10:21:27Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437613 wikitext text/x-wiki 9doduiu2o8l1fy07wnenkyhaxpqu8u6 2437614 2437613 2026-07-16T10:22:27Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2066.codfw.wmnet with OS trixie 2437614 wikitext text/x-wiki r5d9kjz8ampfkunuods8377d7x53k21 2437615 2437614 2026-07-16T10:23:01Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1131-1140].eqiad.wmnet 2437615 wikitext text/x-wiki obzfv3hnqayhkbvekic0wz0fohkip0r 2437616 2437615 2026-07-16T10:33:30Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1131-1140].eqiad.wmnet 2437616 wikitext text/x-wiki 2xoo6yb4ka0ckxfkpbln0c48i764sbw 2437617 2437616 2026-07-16T10:33:32Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1131-1140].eqiad.wmnet 2437617 wikitext text/x-wiki rhhulz5x5k7dtf2hhq3yza1g2qkuk5q 2437618 2437617 2026-07-16T10:33:52Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1141-1150].eqiad.wmnet 2437618 wikitext text/x-wiki 6zhvdeij1v9lt2pofhfx1a9k48h8fv8 2437619 2437618 2026-07-16T10:36:27Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437619 wikitext text/x-wiki nzus802pk89n6i8c3fujy2bupie7q8z 2437620 2437619 2026-07-16T10:37:30Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437620 wikitext text/x-wiki f60wt6ybucy34d8qdbseqcoin83fooq 2437621 2437620 2026-07-16T10:39:31Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1141-1150].eqiad.wmnet 2437621 wikitext text/x-wiki 7h37b52v1jm5v9bbku3gkd4svpimdfm 2437623 2437621 2026-07-16T10:42:24Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2066.codfw.wmnet with reason: host reimage 2437623 wikitext text/x-wiki 6a9k7cv8dmvrt8eod2h5r49w725wkli 2437624 2437623 2026-07-16T10:47:02Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1141-1150].eqiad.wmnet 2437624 wikitext text/x-wiki lhgz6b27auiwhfhx1u2q5ykxo2oirzt 2437625 2437624 2026-07-16T10:47:05Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1141-1150].eqiad.wmnet 2437625 wikitext text/x-wiki mzjrf5duecdnva1hgemjmtck2ezxpyt 2437626 2437625 2026-07-16T10:47:26Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1151-1160].eqiad.wmnet 2437626 wikitext text/x-wiki mgs3jd83otsce6jks7sxoje61xrsmqb 2437627 2437626 2026-07-16T10:49:02Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2066.codfw.wmnet with reason: host reimage 2437627 wikitext text/x-wiki 53550i2foe28wesvdnt9ol1g4g4qp95 2437628 2437627 2026-07-16T10:52:59Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1151-1160].eqiad.wmnet 2437628 wikitext text/x-wiki g4lqg0cgsu4fzwzwb2qostvm29zughb 2437629 2437628 2026-07-16T10:54:32Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437629 wikitext text/x-wiki ngbowrr66ui9t41rfmhxfbbevallpk8 2437630 2437629 2026-07-16T10:55:33Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2437630 wikitext text/x-wiki hwdqze3v203wds500nv8sjc7mix3g44 2437631 2437630 2026-07-16T10:55:34Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2437631 wikitext text/x-wiki mfp0xrsou1ymf9y2uhmos1y8ebs92fb 2437632 2437631 2026-07-16T10:59:50Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1068.eqiad.wmnet with OS trixie 2437632 wikitext text/x-wiki ja78ig0zq5kmsg0b3yzu7gbwp0uly4r 2437633 2437632 2026-07-16T11:03:09Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1151-1160].eqiad.wmnet 2437633 wikitext text/x-wiki sfe136t0lsrldsi2gk3qwd8798khe8p 2437634 2437633 2026-07-16T11:03:12Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1151-1160].eqiad.wmnet 2437634 wikitext text/x-wiki gwfjypir57p84idln71t13re5r8uyu9 2437635 2437634 2026-07-16T11:03:36Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1161-1163,1165,1240-1245].eqiad.wmnet 2437635 wikitext text/x-wiki nnp5tr7g0lfs2j8zwqxipz3hah5zjwv 2437636 2437635 2026-07-16T11:05:25Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437636 wikitext text/x-wiki dzonnepte8m6koewsl0t9t9pebvl0fj 2437638 2437636 2026-07-16T11:08:56Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2066.codfw.wmnet with OS trixie 2437638 wikitext text/x-wiki 1azmlsdzww3bz4nccusj48eny3ygude 2437639 2437638 2026-07-16T11:09:37Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1161-1163,1165,1240-1245].eqiad.wmnet 2437639 wikitext text/x-wiki i9840l3k3isyxkx2xavwi36gj9cz07e 2437640 2437639 2026-07-16T11:13:52Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437640 wikitext text/x-wiki 0adktmh2dhb1ih8unbwlhothyh822tk 2437641 2437640 2026-07-16T11:13:55Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437641 wikitext text/x-wiki c7klxzdi7qs3jwn7adgalofhe57d3yo 2437642 2437641 2026-07-16T11:14:25Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437642 wikitext text/x-wiki hr2axbnnkbdt9n6g9u2ywldkid4mffs 2437643 2437642 2026-07-16T11:15:41Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437643 wikitext text/x-wiki 06dfbvqox1r0kyz4fd0ptsmggeoqrvo 2437644 2437643 2026-07-16T11:20:29Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1161-1163,1165,1240-1245].eqiad.wmnet 2437644 wikitext text/x-wiki r1fvckipa1rgd9sz4vqsmaqfurxovvi 2437645 2437644 2026-07-16T11:20:33Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1161-1163,1165,1240-1245].eqiad.wmnet 2437645 wikitext text/x-wiki f4d3d1ntv98oolbosu7sxivz7j29cvm 2437646 2437645 2026-07-16T11:20:52Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1246-1255].eqiad.wmnet 2437646 wikitext text/x-wiki rv5sqpu5gejovdpgy0dnkyp7wkdevdu 2437647 2437646 2026-07-16T11:22:29Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437647 wikitext text/x-wiki eu3ucqdryhj9idi0nrylodn795ijhmd 2437648 2437647 2026-07-16T11:23:59Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-codfw 2437648 wikitext text/x-wiki te3i648ux1j1c3l5mgf2ihxpy0vub78 2437649 2437648 2026-07-16T11:24:25Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad 2437649 wikitext text/x-wiki 9qlnn1y3pgzro2oae4o9gz62lrzq1z7 2437650 2437649 2026-07-16T11:31:28Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1246-1255].eqiad.wmnet 2437650 wikitext text/x-wiki grt3x1f0o2wgxo4bmjp3buakeljdlvn 2437651 2437650 2026-07-16T11:41:49Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1246-1255].eqiad.wmnet 2437651 wikitext text/x-wiki mo73fwxwtunfmjcux0p98yjuuz3ef7r 2437652 2437651 2026-07-16T11:41:52Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1246-1255].eqiad.wmnet 2437652 wikitext text/x-wiki b8a2dy7b3vdkkrlp7un681bcw6o33o5 2437653 2437652 2026-07-16T11:42:11Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1256-1261,1263-1266].eqiad.wmnet 2437653 wikitext text/x-wiki p2wzk755ckdtxwm8nmsazesfkmdsp7k 2437654 2437653 2026-07-16T11:43:38Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-codfw 2437654 wikitext text/x-wiki 6s89z95ibaorsk99q8xacyd6zpz23xr 2437655 2437654 2026-07-16T11:44:29Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad 2437655 wikitext text/x-wiki jsi8lglyxjlqf4kujtug21866l3w0hx 2437656 2437655 2026-07-16T11:47:48Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1256-1261,1263-1266].eqiad.wmnet 2437656 wikitext text/x-wiki 87y4ifmk5txetqmxlmem423nqu33rp0 2437657 2437656 2026-07-16T11:48:42Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin 2437657 wikitext text/x-wiki jt4kjd8thg6cqfudz1va9njmvc4544x 2437658 2437657 2026-07-16T11:48:59Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin 2437658 wikitext text/x-wiki 38r3mkeul10j7v38n9sni6hk55nfd2n 2437659 2437658 2026-07-16T11:49:38Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_esams 2437659 wikitext text/x-wiki 1d0gysiiodret6yo83m3upe2zq17cwi 2437660 2437659 2026-07-16T11:49:45Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_esams 2437660 wikitext text/x-wiki 74m85xex36tcm02fnav7ea08h7lpr0x 2437661 2437660 2026-07-16T11:50:31Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad 2437661 wikitext text/x-wiki jk4pxwnkn6ykdz1nizpagiscz37m5q8 2437662 2437661 2026-07-16T11:50:32Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2437662 wikitext text/x-wiki stoa2fogikod21lt071ysd616m74mr5 2437663 2437662 2026-07-16T11:50:33Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad 2437663 wikitext text/x-wiki ncbv61h9j8qkom8pz4afc007omn1sy5 2437664 2437663 2026-07-16T11:50:35Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2437664 wikitext text/x-wiki p0xu2a9f5qa96u3wrb3gl365k4zym10 2437665 2437664 2026-07-16T11:51:33Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437665 wikitext text/x-wiki jsbsx534sd9ukm78ywdt941uqs8ah3w 2437666 2437665 2026-07-16T11:51:50Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437666 wikitext text/x-wiki d465ldm7amgc840yrv8yjuzx9xehhj0 2437667 2437666 2026-07-16T11:53:10Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad 2437667 wikitext text/x-wiki p4dso2wuw17bhdg37qk9ljqhc7oc1dw 2437668 2437667 2026-07-16T11:53:17Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437668 wikitext text/x-wiki kgqy4f8oyq5wlc4iy1flfqhgr5iqbr3 2437669 2437668 2026-07-16T11:54:19Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437669 wikitext text/x-wiki eoxlr92iwy04ioqivh3yfbc1iz7i0fp 2437670 2437669 2026-07-16T11:54:58Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox and (A:eqsin or A:drmrs or A:magru) and not (P{dns5003*} or P{dns7002*}) and (A:dnsbox) 2437670 wikitext text/x-wiki isis09zkemsasviq2qoq1fru2lyq8yg 2437671 2437670 2026-07-16T11:55:00Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns5004.wikimedia.org 2437671 wikitext text/x-wiki hdc3zrwlnehdtvnus4qfo1tglnssaew 2437672 2437671 2026-07-16T11:58:46Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1256-1261,1263-1266].eqiad.wmnet 2437672 wikitext text/x-wiki kc9hhampun98avbo2iuues1qua3xoj1 2437673 2437672 2026-07-16T11:58:49Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1256-1261,1263-1266].eqiad.wmnet 2437673 wikitext text/x-wiki 35ko7o23u4t9q9yusvuow6zys2ssdz4 0 17290 2437289 2436712 2026-07-15T18:58:02Z Stashbot 7414 James_F: dev-images: Re-build PHP images for latest point releases, for T431099 2437289 wikitext text/x-wiki 4to26ibq69a2ojpl57xqm9v99kmfdzc 2 19379 2437476 2403059 2026-07-16T04:31:38Z Ranjithsiji 3352 add section 2437476 wikitext text/x-wiki 8ak5je5dbumfiry35qvu178vo1fjizw 498 30942 2436935 2436484 2026-07-15T13:07:40Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T431429) 2436935 wikitext text/x-wiki k3xq27uei6gxegct16fb3c083jmzk2u 2436936 2436935 2026-07-15T13:07:45Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T431429) 2436936 wikitext text/x-wiki 8er33io0vxayc5k2k9kuxfuj9fgw4u4 498 443060 2437074 2375117 2026-07-15T14:58:30Z Stashbot 7414 bd808: Hard reboot of logging-logstash-04 via Horizon (T432254) 2437074 wikitext text/x-wiki 13ufbsauoakp7cg7x79m78t13ci0p85 498 443628 2437088 2436655 2026-07-15T15:20:26Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29427447334 (https://github.com/cluebotng/component-configs/commits/0d2bc5e6c57ba81a40584a85fe4ea3868875999f) 2437088 wikitext text/x-wiki r1rw2q4b22aye77u4likr1200i2skof 2437106 2437088 2026-07-15T15:34:50Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29428509974 (https://github.com/cluebotng/component-configs/commits/842854f79b07dc4ec739505a99c8cb11009f8f4f) 2437106 wikitext text/x-wiki qhe5hgmnsualvmozcx32r4ul0iz22ee 2437141 2437106 2026-07-15T16:01:26Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29430449675 (https://github.com/cluebotng/component-configs/commits/ecf4f749293c44e72e74858bef8193eb29695c9a) 2437141 wikitext text/x-wiki 1z4j0csmhozpqup3avh1a24qrxp31nh 2437203 2437141 2026-07-15T16:56:55Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29434331433 (https://github.com/cluebotng/component-configs/commits/37dd33ec5b56f991e912ec44ce96a536735cb1a0) 2437203 wikitext text/x-wiki e1skojig9pocg0jq5vppl9tna73nnr7 2437245 2437203 2026-07-15T17:31:14Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29436632270 (https://github.com/cluebotng/component-configs/commits/ff37e7e6d3e126ba24d186be144aeae3d2048954) 2437245 wikitext text/x-wiki b1m0oszsehghxvm6l84iju0cxmwqfgp 2437532 2437245 2026-07-16T08:20:18Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29482938340 (https://github.com/cluebotng/component-configs/commits/6ddd234f9ddace81ab9e1e7e45b5f974c5938a11) 2437532 wikitext text/x-wiki k38akipd0ytrqnwpb1cvcagjyfjr4x8 2437546 2437532 2026-07-16T08:50:14Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29484727917 (https://github.com/cluebotng/component-configs/commits/328ca21897e26730c95c4dad77c34d5e1bbf9c31) 2437546 wikitext text/x-wiki 8iuygfeapf0px8i4m9kjob0pnpmdie4 498 443949 2437530 2426116 2026-07-16T08:18:36Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 94f04f3 (Refactor: extract get_column_label() from make_column_header()) 2437530 wikitext text/x-wiki e0ozougd2394wdzehwan2r3hgkhatx2 2437531 2437530 2026-07-16T08:18:40Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 651408e (Add ReferenceColumn for tracking sourced statements) for T428637 2437531 wikitext text/x-wiki 0zrbgu51f4nmaj3xsk4dq0ao6hn0vmp 0 447017 2436938 2417391 2026-07-15T13:09:41Z Ifrahkhanyaree WMDE 36875 /* Upcoming SLOs */ 2436938 wikitext text/x-wiki jc6c4yoz0z2d7ahtnh5302roehtc7s1 0 449160 2437419 2436613 2026-07-16T00:01:55Z Dexbot 30554 Bot: Updating the report 2437419 wikitext text/x-wiki qrl4sm8eqxgrg8fgiuajx1o57mu5kzn 2437516 2437419 2026-07-16T07:47:51Z Dexbot 30554 Bot: Updating the report 2437516 wikitext text/x-wiki lpsw76tiolxe0xn6hyp8a53l44rjdp2 498 452890 2437107 2433027 2026-07-15T15:35:54Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2437107 wikitext text/x-wiki qruy8nifsjqxuaifb84fy2btfzgnuua 2437224 2437107 2026-07-15T17:07:46Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29435050584 (https://github.com/cluebotng/component-configs/commits/912a8ea89e62cddf63a20263572b5c94900ef06a) 2437224 wikitext text/x-wiki fdidpatndfbymjn80fzdlaq51k1d5eq 0 459699 2437349 2436620 2026-07-15T21:19:35Z Stashbot 7414 eileen: SmashPig upgraded from 34082cca to 37a80ec4 2437349 wikitext text/x-wiki ggkz8gzjogxdw5eu36lv1s0ai14g9ya 2437366 2437349 2026-07-15T21:48:36Z Stashbot 7414 eileen: civicrm upgraded from d38ae4d6 to 2dd23034 2437366 wikitext text/x-wiki 683qw11f1kbi0ocm1byikh766ol9cqr 2437400 2437366 2026-07-15T23:12:45Z Stashbot 7414 eileen: SmashPig upgraded from 37a80ec4 to 2e07ce09 2437400 wikitext text/x-wiki e5jrskc45edbr7gt9cgpuad3e78wfjg 2437418 2437400 2026-07-15T23:48:26Z Stashbot 7414 eileen: config revision changed from 8078b0b0 to 7dddea38 2437418 wikitext text/x-wiki d9cwp0h3j39cb4lnq0ikb900gspohyz 2437447 2437418 2026-07-16T01:51:02Z Stashbot 7414 eileen: SmashPig upgraded from 2e07ce09 to 6921d862 2437447 wikitext text/x-wiki 01kh7vwl5ckjpwq46pjsfq7uwm3qiyh 2437457 2437447 2026-07-16T02:31:24Z Stashbot 7414 eileen: civicrm upgraded from 2dd23034 to 1c94b489 2437457 wikitext text/x-wiki pstg50gjznwfdf7o7l8klnirpzy861r 2437622 2437457 2026-07-16T10:41:01Z Stashbot 7414 laurabar: process-control config revision changed from 7dddea38 to 730715e1 2437622 wikitext text/x-wiki s37d4mzr8xl8mzjy0yyod3xhmgehr0h 2437637 2437622 2026-07-16T11:06:39Z Stashbot 7414 laurabar: process-control config revision changed from 730715e1 to c951e730 2437637 wikitext text/x-wiki 7p8ga2cscgpkos6m25cy1fijxeaj6uh 2 460066 2437348 2403109 2026-07-15T21:19:09Z Ranjithsiji 3352 /* Install uv */ 2437348 wikitext text/x-wiki qw9rutigpzk4hhzn8zhiaxzfdhfazbg 1 460416 2437081 2435207 2026-07-15T15:11:30Z BTracy-WMF 46548 /* federated MWAPI server ? */ Reply 2437081 wikitext text/x-wiki ntjofjzuukbung38j2ketgkpnu6lfoz 2437228 2437081 2026-07-15T17:11:30Z BTracy-WMF 46548 /* federated MWAPI server ? */ Reply 2437228 wikitext text/x-wiki bz1uu2vh6ls24mhvhngqqifjc4wvy9u 0 460417 2436916 2435004 2026-07-15T12:45:11Z AChou-WMF 15621 2436916 wikitext text/x-wiki sdi6y4i1qh7x1e04disgx1244aqojk6 2437576 2436916 2026-07-16T09:26:22Z AChou-WMF 15621 2437576 wikitext text/x-wiki gfr948bau5lyy493qnvitkzknaj4x0u 2437579 2437576 2026-07-16T09:36:27Z AChou-WMF 15621 /* Python (OpenAI SDK) */ 2437579 wikitext text/x-wiki 1kgw7xn54bpg8f7bmzigcdb35usxzfb 0 460447 2436937 2026-07-15T13:09:18Z Ifrahkhanyaree WMDE 36875 Created page with "Status: '''[[SLO/Runbook#Drafting|draft]]''' == Organizational == === Service === Wikibase GraphQL is a GraphQL API for Wikidata. It is a flexible alternative to a few simple [[Wikidata_Query_Service|Wikidata Query Service (WDQS)]] use cases, with a strong focus on developer experience, flexibility, and efficient data retrieval. See [[:d:wikidata:Wikibase GraphQL|Wikidata:Wikibase GraphQL]] === Teams === The [[:meta:Wikibase Reuse|Wikibase Reuse Team]] is responsible..." 2436937 wikitext text/x-wiki cwhzmctej01uw5rmza35mmmp7qr5yhf