Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.21 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2460954 2460859 2026-09-25T13:32:12Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-115 (T438255) 2460954 wikitext text/x-wiki a2x6i9qsaqsp7a8s0nskob884xvhiqn 2460955 2460954 2026-09-25T13:33:22Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-115 (T438255) 2460955 wikitext text/x-wiki 2knkcictpemp2tuiqg4nhjdgkbclw2y 2460956 2460955 2026-09-25T13:34:00Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster (T438255) 2460956 wikitext text/x-wiki fksqk79rss05xpstm9g625fdia2ft04 2460973 2460956 2026-09-25T13:45:51Z Stashbot 7414 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster 2460973 wikitext text/x-wiki 3rllv3tda46pq2pxhb1q7fnts1p4y1y 2460974 2460973 2026-09-25T13:45:52Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster 2460974 wikitext text/x-wiki 2fnntr112kolzv1p3i5k20bjo1kyee0 2461072 2460974 2026-09-25T18:18:46Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-87 2461072 wikitext text/x-wiki jjzrmbergdmi7p99315whgyxxvszmgq 2461073 2461072 2026-09-25T18:20:14Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-87 2461073 wikitext text/x-wiki 4mc90nicbrslfot24x5oh664ce5han3 2461074 2461073 2026-09-25T18:22:18Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) 2461074 wikitext text/x-wiki jqotmjys1dv7smbuygxxnb3zd4hq7zp 2461079 2461074 2026-09-25T18:31:52Z Stashbot 7414 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-90.tools.eqiad1.wikimedia.cloud to the cluster 2461079 wikitext text/x-wiki pcnadmkxc8ny7my3mocupo6vgcwalk9 2461080 2461079 2026-09-25T18:31:53Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster 2461080 wikitext text/x-wiki m219r695ub8eiv2urkwhdhktxqb8g1d 0 4108 2460910 2460898 2026-09-25T12:13:25Z ScheduleDeploymentBot 37566 Add [[gerrit:1343611]] to Monday, September 28 UTC morning backport window 2460910 wikitext text/x-wiki i6o1fu8qcb7fkr2niqnb90qr3kmmkmy 2460946 2460910 2026-09-25T13:20:41Z ScheduleDeploymentBot 37566 Add [[gerrit:1344594]] to Monday, September 28 UTC morning backport window 2460946 wikitext text/x-wiki 3f570l6c03xhoj0x91gpumh3ambn5y3 2460998 2460946 2026-09-25T13:59:40Z ScheduleDeploymentBot 37566 Add [[gerrit:1345130]] to Monday, September 28 UTC afternoon backport window 2460998 wikitext text/x-wiki 9ga8lsih45eiobpiztws98rhz6iccg2 2461082 2460998 2026-09-25T21:06:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1345052]] to Monday, September 28 UTC afternoon backport window 2461082 wikitext text/x-wiki ebfcxzp9o31shrofmd048k3fekrp89j 2461083 2461082 2026-09-25T21:27:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1344216]] to Monday, September 28 UTC afternoon backport window 2461083 wikitext text/x-wiki la953yh5wlu7fayuuq1qclbbuo6t83i 2461097 2461083 2026-09-26T02:00:42Z DeploymentCalendarTool 20896 Remove Week of September 21 2461097 wikitext text/x-wiki 2m7ovjeyd4k9ruslhyczat5cehj8713 0 7919 2460911 2460909 2026-09-25T12:13:44Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet 2460911 wikitext text/x-wiki 8bwhxtd649grj1pbgrlnauat1xe9e23 2460912 2460911 2026-09-25T12:18:51Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.netbox 2460912 wikitext text/x-wiki 00wc5sqzhpmgoqges07ym9gcp91ut7v 2460913 2460912 2026-09-25T12:20:18Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2460913 wikitext text/x-wiki 8l3642j8tlt1kv59bmgmqtjq6jdffmg 2460914 2460913 2026-09-25T12:20:29Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2460914 wikitext text/x-wiki aa91olfojow8kzurk09e7sr1p1h8v7y 2460915 2460914 2026-09-25T12:21:15Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2460915 wikitext text/x-wiki dnlpbf3pp0pbvnotfxxkw4lg7wci54l 2460916 2460915 2026-09-25T12:21:25Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2460916 wikitext text/x-wiki 2njidg135nyu6eg7hj174a12r104yp6 2460917 2460916 2026-09-25T12:26:00Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" 2460917 wikitext text/x-wiki phxu4tp17b6lgua30x1vbic9bu4724l 2460918 2460917 2026-09-25T12:33:49Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039 2460918 wikitext text/x-wiki kl8wwqv14olngblm5g8jrmui44nf0ip 2460919 2460918 2026-09-25T12:34:10Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.dns.netbox 2460919 wikitext text/x-wiki 052qczreiq4amotihhv0pvwxwn6cloa 2460920 2460919 2026-09-25T12:34:11Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" 2460920 wikitext text/x-wiki c8s1hmfkhkdjn6lc55dc401n3a6a6bf 2460921 2460920 2026-09-25T12:34:13Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2460921 wikitext text/x-wiki twgun2j69lv7bntowsywnzqbm13zoqw 2460922 2460921 2026-09-25T12:34:14Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet 2460922 wikitext text/x-wiki to75lmm06raoj8hivpv79264mq74oaw 2460923 2460922 2026-09-25T12:38:26Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" 2460923 wikitext text/x-wiki mmoi8880rudwo2sffpt8gfp7xbpe7i8 2460924 2460923 2026-09-25T12:39:43Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" 2460924 wikitext text/x-wiki o6zdaa4qy4goip66ezzopbasyf05u3m 2460925 2460924 2026-09-25T12:39:44Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2460925 wikitext text/x-wiki rupujyysm4sbmbb464eicx7mmqm29cx 2460926 2460925 2026-09-25T12:39:45Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors 2460926 wikitext text/x-wiki gf2f6rmw1p8gjl76iid99vyi6o0paxs 2460927 2460926 2026-09-25T12:39:47Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors 2460927 wikitext text/x-wiki mczpvahqt8q7sc6y5maratgsl47et9v 2460928 2460927 2026-09-25T12:39:48Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039 2460928 wikitext text/x-wiki d9yvp4smq78rhvrrtn092vuphiefw5d 2460929 2460928 2026-09-25T12:40:08Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039 2460929 wikitext text/x-wiki bw69anjy5elgsrdvogzizcrn3i943nv 2460930 2460929 2026-09-25T12:40:16Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039 2460930 wikitext text/x-wiki l62eum5hln1hf3ewv0q2fkznrbabc9l 2460931 2460930 2026-09-25T12:42:00Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm 2460931 wikitext text/x-wiki 14h9vg6a5cbnzvp9vf73mrsh1xiv4zc 2460932 2460931 2026-09-25T12:43:36Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2460932 wikitext text/x-wiki b6sns2cwzswl3fvkcb55gbldqle966m 2460933 2460932 2026-09-25T12:44:12Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2460933 wikitext text/x-wiki l4xr90ijri2bop42p4e76941wu1ribn 2460934 2460933 2026-09-25T12:44:44Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2460934 wikitext text/x-wiki 7wghnj83izmoos8v9q6yz475s85gobb 2460935 2460934 2026-09-25T12:45:42Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2460935 wikitext text/x-wiki c1tbbctk1qu4nrz9ood25cto1iuiva4 2460936 2460935 2026-09-25T12:47:14Z Stashbot 7414 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance 2460936 wikitext text/x-wiki f1qq8oe39mfu41mfjqx60wyl9gopbgm 2460937 2460936 2026-09-25T12:47:15Z Stashbot 7414 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance 2460937 wikitext text/x-wiki mcbpn7ikxdldivyfmkkdfq93gnm7n25 2460938 2460937 2026-09-25T12:52:25Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage 2460938 wikitext text/x-wiki 00bztlzl97andtx35w3gpp72paxh7sq 2460939 2460938 2026-09-25T12:55:51Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage 2460939 wikitext text/x-wiki cjneqt2v44mgo2ssygbef5ip05k85cd 2460940 2460939 2026-09-25T12:58:21Z Stashbot 7414 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2460940 wikitext text/x-wiki 8z7mfmj2kjqx4o3scud1mqw04avbzq1 2460941 2460940 2026-09-25T12:59:40Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet 2460941 wikitext text/x-wiki a0resixc3rjln33zwugd2jbfm4t9dh7 2460942 2460941 2026-09-25T13:03:20Z Stashbot 7414 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2460942 wikitext text/x-wiki mb87ttt8qyz8hi7262xrhrqqacx4gmr 2460943 2460942 2026-09-25T13:15:55Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm 2460943 wikitext text/x-wiki 2bueuvdcnpwyqjgrh8nxht2ht7y4c1k 2460944 2460943 2026-09-25T13:20:04Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply 2460944 wikitext text/x-wiki 4s8abn4vp9f075ewxlqlbdi29hk8xad 2460945 2460944 2026-09-25T13:20:13Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply 2460945 wikitext text/x-wiki 9zdt9mm25g9ya3ztz541396f1ulyvcg 2460947 2460945 2026-09-25T13:22:09Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply 2460947 wikitext text/x-wiki aomzbrkuibdynm7q9xxmhe4gmr6k69d 2460948 2460947 2026-09-25T13:22:19Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply 2460948 wikitext text/x-wiki lgnhgrb7mce34wvk3e7oum8ehrk5p3m 2460949 2460948 2026-09-25T13:24:27Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2460949 wikitext text/x-wiki mbk4grmwi2alyacw0jm6bgbsf40h6ul 2460950 2460949 2026-09-25T13:24:29Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2460950 wikitext text/x-wiki 5izyshm0e1r99tr45yjo3dlp3dnr0bn 2460951 2460950 2026-09-25T13:24:30Z Stashbot 7414 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active 2460951 wikitext text/x-wiki tabhx9r1wb7f8bs5ko9la2hh9x0e1s4 2460952 2460951 2026-09-25T13:24:46Z Stashbot 7414 Emperor: repool sessionstore in codfw 2460952 wikitext text/x-wiki im341loxkfsngavhqutguzlwys29215 2460953 2460952 2026-09-25T13:29:34Z Stashbot 7414 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active 2460953 wikitext text/x-wiki d1etri32188byj5zr4ot4cgjuowidq6 2460957 2460953 2026-09-25T13:34:37Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040 2460957 wikitext text/x-wiki 5nixsu53nmpek09pre9xqk2bk8oqrjl 2460958 2460957 2026-09-25T13:34:48Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.dns.netbox 2460958 wikitext text/x-wiki lifraxpualmcmpydkai6rozusz8et8g 2460959 2460958 2026-09-25T13:38:39Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" 2460959 wikitext text/x-wiki 1f9r88mextbjjyya8g7avvl858xamkw 2460960 2460959 2026-09-25T13:39:07Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" 2460960 wikitext text/x-wiki o51abwq3wsrc5ek1k6nbszsown7t35u 2460961 2460960 2026-09-25T13:39:08Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2460961 wikitext text/x-wiki i9m4vqbsjqgrvqz2aaor1ds7yh47413 2460962 2460961 2026-09-25T13:39:08Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors 2460962 wikitext text/x-wiki dgeu76s3w1dgxftiletx3pmpxkhcv7b 2460963 2460962 2026-09-25T13:39:10Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors 2460963 wikitext text/x-wiki 4li431wvveibxhfpapgfwk10pgl9ax0 2460964 2460963 2026-09-25T13:39:11Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040 2460964 wikitext text/x-wiki q41e9oyyp1c9c3cbdccksjqnhcl1p8e 2460965 2460964 2026-09-25T13:39:30Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040 2460965 wikitext text/x-wiki iu78xa2eoment8pz2nohefsxr2yqku1 2460966 2460965 2026-09-25T13:40:07Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040 2460966 wikitext text/x-wiki owsno3ou8q27el3qj83qniij7zxma5q 2460967 2460966 2026-09-25T13:42:10Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2460967 wikitext text/x-wiki dwsjtv1esb4cnkjhpowjf4rmnzh8wor 2460968 2460967 2026-09-25T13:42:19Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2460968 wikitext text/x-wiki 6scpgqvsxjlpccupl08il6psfm8bm6s 2460969 2460968 2026-09-25T13:44:18Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020.*} and A:lvs 2460969 wikitext text/x-wiki mg20co4jea2zoe660l768rmfpx057gr 2460970 2460969 2026-09-25T13:44:51Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020.*} and A:lvs 2460970 wikitext text/x-wiki q2jm6hv1rs9dn61iy88n3ycw2k7bvqc 2460971 2460970 2026-09-25T13:45:30Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm 2460971 wikitext text/x-wiki 7nvc2zy414bkxo9x7hh7j0171zfujc2 2460972 2460971 2026-09-25T13:45:37Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet 2460972 wikitext text/x-wiki oes7lqfhok69vrktliwgydj5xyp5ml9 2460975 2460972 2026-09-25T13:46:36Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019.*} and A:lvs 2460975 wikitext text/x-wiki atexkh2a3r6kinok2mseyooo1u0qn8e 2460976 2460975 2026-09-25T13:46:58Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019.*} and A:lvs 2460976 wikitext text/x-wiki 3cp0qa7bea4h1a8ik7bgdwgnj5jk2cs 2460977 2460976 2026-09-25T13:47:31Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041 2460977 wikitext text/x-wiki 2349odvif5em15d7pmjgo570zk2sa0d 2460978 2460977 2026-09-25T13:47:55Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.dns.netbox 2460978 wikitext text/x-wiki 2bai66an8bs8v4jcnvhkfzs6bw84ybp 2460979 2460978 2026-09-25T13:49:00Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply 2460979 wikitext text/x-wiki os7id8lbc6sihaw6vsifves5rc88m13 2460980 2460979 2026-09-25T13:49:10Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply 2460980 wikitext text/x-wiki dmopf9pqnbk2x8re4cmdck7h4i89as4 2460981 2460980 2026-09-25T13:50:51Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet 2460981 wikitext text/x-wiki tf7fyoyl5ndjn717r7c9iw9own0oi7z 2460982 2460981 2026-09-25T13:51:25Z Stashbot 7414 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # T438750 2460982 wikitext text/x-wiki i33t25ff5ybpivfsu2q0vk346wbh93a 2460983 2460982 2026-09-25T13:51:54Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" 2460983 wikitext text/x-wiki hvs6lvvlq20eyldhmtjuot4ofcip1g2 2460984 2460983 2026-09-25T13:52:39Z Stashbot 7414 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply 2460984 wikitext text/x-wiki a3tj344eau3scw0p35c1mhf2vx1qy0d 2460985 2460984 2026-09-25T13:52:42Z Stashbot 7414 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply 2460985 wikitext text/x-wiki edalhlrwsghqd5ph6edyki2kdtzznu1 2460986 2460985 2026-09-25T13:52:48Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" 2460986 wikitext text/x-wiki 9syw6d29r7zba2c9mpdicu7xfzld15f 2460987 2460986 2026-09-25T13:52:49Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2460987 wikitext text/x-wiki dej3rmuy5cke4kuvzv05rbqx2lutry5 2460988 2460987 2026-09-25T13:52:50Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors 2460988 wikitext text/x-wiki 65dwtjhfv7jr7jmlbrft7gksrv6r7n8 2460989 2460988 2026-09-25T13:52:52Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors 2460989 wikitext text/x-wiki n9sjs3leml2ur8z8tffp7qsf06oc4bf 2460990 2460989 2026-09-25T13:52:53Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041 2460990 wikitext text/x-wiki 37s18psbw0varybd2pu9nafcejhx96b 2460991 2460990 2026-09-25T13:53:03Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041 2460991 wikitext text/x-wiki 9zluzfo9p3fsq6291eg7nyc2qtwpvb4 2460992 2460991 2026-09-25T13:53:40Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041 2460992 wikitext text/x-wiki p77oxpfhhhps9nohhzs27munann13k0 2460993 2460992 2026-09-25T13:54:38Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm 2460993 wikitext text/x-wiki f28hjc5ankdn1wgejac1yosoukk0kgw 2460994 2460993 2026-09-25T13:55:22Z Stashbot 7414 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2460994 wikitext text/x-wiki 2mcudyjatp9bzoziihe7w6c3y01bo31 2460995 2460994 2026-09-25T13:55:31Z Stashbot 7414 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2460995 wikitext text/x-wiki 3se1yxstnxgs6ecbox8ajii2m79mf89 2460996 2460995 2026-09-25T13:58:43Z Stashbot 7414 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2460996 wikitext text/x-wiki tvsvue37iva4zly1wu9jbsmb4zmbko0 2460997 2460996 2026-09-25T13:58:47Z Stashbot 7414 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply 2460997 wikitext text/x-wiki c7ahcmk1kpaw4mffyt86hzg8xytwlg4 2460999 2460997 2026-09-25T13:59:45Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage 2460999 wikitext text/x-wiki elt8zure3swb44b2ma21abts4celt4u 2461000 2460999 2026-09-25T14:00:22Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2461000 wikitext text/x-wiki 9x0po390m66i891qc9n43qu0pbbxhf3 2461001 2461000 2026-09-25T14:00:35Z Stashbot 7414 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw 2461001 wikitext text/x-wiki cgrsw7ovenyu1ek7pdlh8kip5eru6um 2461002 2461001 2026-09-25T14:00:36Z Stashbot 7414 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw 2461002 wikitext text/x-wiki bxrwsqzdkzbzg4domcimt8x7og0hcac 2461003 2461002 2026-09-25T14:00:37Z Stashbot 7414 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw 2461003 wikitext text/x-wiki kpdnez058ej7m2ab9rh2s9xojcww3kj 2461004 2461003 2026-09-25T14:00:38Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2461004 wikitext text/x-wiki ex6y3nbsm30nhfnrrpuputq220hnwm5 2461005 2461004 2026-09-25T14:00:39Z Stashbot 7414 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw 2461005 wikitext text/x-wiki p7u3pmza937qlmmg87j8rsg70tmv5r8 2461006 2461005 2026-09-25T14:01:06Z Stashbot 7414 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet 2461006 wikitext text/x-wiki f3j6il1da0yfltqk4zu0zxpzuwjcy4a 2461007 2461006 2026-09-25T14:02:17Z Stashbot 7414 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet 2461007 wikitext text/x-wiki mvlno1wx13f2dpc2yvp5wqwzod70hy5 2461008 2461007 2026-09-25T14:05:28Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage 2461008 wikitext text/x-wiki 7h2iejyy8mkcf58g50058n7qii26hu9 2461010 2461008 2026-09-25T14:06:46Z Stashbot 7414 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage 2461010 wikitext text/x-wiki oyddsrzsnm499n20hh5kgag3n4i31yx 2461011 2461010 2026-09-25T14:06:48Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage 2461011 wikitext text/x-wiki mkeifo7ckcjfp5ndsdid2rgo7ub9qpy 2461012 2461011 2026-09-25T14:13:21Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2461012 wikitext text/x-wiki izcfiwwnr5epvq78v668tz0ldldodk6 2461013 2461012 2026-09-25T14:13:31Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2461013 wikitext text/x-wiki 0w709dij7k0sylbg62c8na4lr7ae8p9 2461014 2461013 2026-09-25T14:13:45Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2461014 wikitext text/x-wiki 5fzuu1nnckut5zn3cuq024w9mt2fwxf 2461015 2461014 2026-09-25T14:13:55Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2461015 wikitext text/x-wiki hkkbui1e9yfytieobo4y8tpim5qmweg 2461016 2461015 2026-09-25T14:14:20Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie 2461016 wikitext text/x-wiki dwurluytjef31iv3nd0zkehflxbhglp 2461017 2461016 2026-09-25T14:14:47Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply 2461017 wikitext text/x-wiki d8im18yvx14y8x6wke1k69o4duwzddt 2461018 2461017 2026-09-25T14:14:57Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply 2461018 wikitext text/x-wiki d35gdmgg4ozw024xr1fziseqnixs3zk 2461019 2461018 2026-09-25T14:15:16Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply 2461019 wikitext text/x-wiki 3z8jc98j6lbk7q0obaj4gnv7t0ytvj4 2461020 2461019 2026-09-25T14:15:26Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply 2461020 wikitext text/x-wiki o1em1q1uc80nxb2uszzkn6mm5tn55ie 2461021 2461020 2026-09-25T14:17:19Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply 2461021 wikitext text/x-wiki opadrey7a4nxgvmbn6dwau7mkihe5d0 2461022 2461021 2026-09-25T14:17:30Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply 2461022 wikitext text/x-wiki ikauw1rjjzx4cttmpskycosxzs3v5sp 2461023 2461022 2026-09-25T14:17:39Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2461023 wikitext text/x-wiki j60sjji6m8q1o2q0gc9noc39yrilxvc 2461024 2461023 2026-09-25T14:17:49Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2461024 wikitext text/x-wiki 90mi302jzaubcprwnh6ijthl123mduj 2461025 2461024 2026-09-25T14:17:57Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply 2461025 wikitext text/x-wiki i5hgcib8n9dsklhb79qmj5mf746qkeq 2461026 2461025 2026-09-25T14:18:08Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply 2461026 wikitext text/x-wiki hgw070ixh8og7d4xhcps676kp1i9nuu 2461027 2461026 2026-09-25T14:23:45Z Stashbot 7414 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.* 2461027 wikitext text/x-wiki nmwutjc3ohjwaz9l5di6vjpxvk8ax9u 2461028 2461027 2026-09-25T14:29:29Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm 2461028 wikitext text/x-wiki 0b06gm6fxvyu92ql1f9029c559pbdfl 2461029 2461028 2026-09-25T14:30:16Z Stashbot 7414 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - T343000 2461029 wikitext text/x-wiki bmet8u7hkm3x3k3nn2hs01qelxlgywr 2461031 2461029 2026-09-25T14:32:45Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm 2461031 wikitext text/x-wiki 4v2td5bqeto3mmo7usf3ahzzv7dsswy 2461032 2461031 2026-09-25T14:33:45Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage 2461032 wikitext text/x-wiki ortfconpmtaszm1u7iqi96si6e5vxlp 2461033 2461032 2026-09-25T14:38:25Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage 2461033 wikitext text/x-wiki ow8wd3fr6kl3fgnude0dcmb0gevqo41 2461034 2461033 2026-09-25T14:41:11Z Stashbot 7414 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply 2461034 wikitext text/x-wiki qp3jf43ms8votiv2k3g3t5cyzbgq04l 2461035 2461034 2026-09-25T14:41:14Z Stashbot 7414 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply 2461035 wikitext text/x-wiki f4xba1oa4l80zrsd78hus8cjus97bnx 2461036 2461035 2026-09-25T14:43:55Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet 2461036 wikitext text/x-wiki 9f8od22j718dunxo3jug71mp55t6wjd 2461037 2461036 2026-09-25T14:44:22Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2461037 wikitext text/x-wiki mf2gisco3kipqb3vduj2hyru1090pxy 2461038 2461037 2026-09-25T14:44:31Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2461038 wikitext text/x-wiki h6x129pfn1w0k48cgu9vgxk16kmevjq 2461039 2461038 2026-09-25T14:46:31Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet 2461039 wikitext text/x-wiki gqkong5ubxzwm0d2b55klx5220l7qwt 2461040 2461039 2026-09-25T14:49:09Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet 2461040 wikitext text/x-wiki nmomhpf8zpj0jgz3e52sv4go0lmlipa 2461041 2461040 2026-09-25T14:49:26Z Stashbot 7414 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet 2461041 wikitext text/x-wiki 5iluof8607yyduiozb0rkhyrsq1bpdb 2461042 2461041 2026-09-25T14:49:31Z Stashbot 7414 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet 2461042 wikitext text/x-wiki 29mb6dp8u3spc3x2n01bqctnyl6532e 2461043 2461042 2026-09-25T14:51:35Z Stashbot 7414 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet 2461043 wikitext text/x-wiki 0wte2h4cfkn2s68cjfr9jz30ad8uye7 2461044 2461043 2026-09-25T14:51:40Z Stashbot 7414 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet 2461044 wikitext text/x-wiki jlnhdceg0dt0iqt9qdc856cbp6hqm87 2461045 2461044 2026-09-25T14:51:44Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet 2461045 wikitext text/x-wiki tkdfqu2gra1d1i29qnbopt8e7ncmdtp 2461046 2461045 2026-09-25T14:56:42Z Stashbot 7414 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet 2461046 wikitext text/x-wiki 0007cr3ue8ut2iapnp8yzw904qnxwha 2461047 2461046 2026-09-25T14:56:46Z Stashbot 7414 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet 2461047 wikitext text/x-wiki ot4ppu6k0ehtxu8q5gkrrj4oi21yaxz 2461048 2461047 2026-09-25T14:57:38Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie 2461048 wikitext text/x-wiki tdsegeb227jknuqx1wtjb506sjj1o7j 2461049 2461048 2026-09-25T15:01:30Z Stashbot 7414 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2461049 wikitext text/x-wiki 1uh8t8hu1yatds2x7c1hhq5g75gqes2 2461050 2461049 2026-09-25T15:01:38Z Stashbot 7414 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2461050 wikitext text/x-wiki m8xgjnm0uis3bb69qpp8stmrfcp8mq8 2461051 2461050 2026-09-25T15:01:59Z Stashbot 7414 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2461051 wikitext text/x-wiki hduw06rx3ualrgm51lnllhdcj9z6lvr 2461052 2461051 2026-09-25T15:02:03Z Stashbot 7414 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2461052 wikitext text/x-wiki qe1oap9j7pg205dmsrstmj32mytx522 2461053 2461052 2026-09-25T15:03:34Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2461053 wikitext text/x-wiki 59v313y3fxmxqg66yrgb8myql0imsnd 2461054 2461053 2026-09-25T15:03:43Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2461054 wikitext text/x-wiki 1g97bwz1kw498onh5i3t0ww5mds196j 2461056 2461054 2026-09-25T15:07:32Z Stashbot 7414 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.* 2461056 wikitext text/x-wiki 8ujdvxvs0wtumkanw7xtxpcbrzqngap 2461066 2461056 2026-09-25T17:57:16Z Stashbot 7414 cdanis@dns1004: START - running authdns-update 2461066 wikitext text/x-wiki 9qzpq8yiya4uuuq5gk5s8fh3kjz2u64 2461067 2461066 2026-09-25T17:59:41Z Stashbot 7414 cdanis@dns1004: END - running authdns-update 2461067 wikitext text/x-wiki kiogq3ka6trvjxytxh0t1wn91tsh08h 2461075 2461067 2026-09-25T18:27:20Z Stashbot 7414 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: T439183: Accept dot, plus, hyphen in label values 2461075 wikitext text/x-wiki o9q6ssa014mvjy8mx9kjjjb5hh950tl 2461076 2461075 2026-09-25T18:27:31Z Stashbot 7414 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: T439183: Accept dot, plus, hyphen in label values (duration: 00m 11s) 2461076 wikitext text/x-wiki 8s3d6zacy1rx5rrzj1rzdtx13qckh0l 2461085 2461076 2026-09-25T22:39:12Z Stashbot 7414 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2461085 wikitext text/x-wiki du5xqwxd5xfl0ghann3vz44nejfkxqc 2461086 2461085 2026-09-25T22:46:20Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2461086 wikitext text/x-wiki ei3ct403pax9gg9e9arzxpkitntzw2o 2461087 2461086 2026-09-25T22:46:49Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2461087 wikitext text/x-wiki 157g0comayrio70ku418yjtclzma3pn 2461088 2461087 2026-09-25T22:46:50Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2461088 wikitext text/x-wiki 5iggzztp1gvg37bol7k8rlq1phrxnpw 2461089 2461088 2026-09-25T22:47:31Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2461089 wikitext text/x-wiki fji9qaz6v9412l907jlb54mb8q0xwb5 2461090 2461089 2026-09-25T22:51:52Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2461090 wikitext text/x-wiki rkt5cbm8w6blk6ula54lqcirk9iqkhi 2461091 2461090 2026-09-25T22:51:54Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2461091 wikitext text/x-wiki 2n8i5u2g2qsy116xafaf5edeqwapz0a 2461092 2461091 2026-09-25T22:51:55Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2461092 wikitext text/x-wiki synez85757c3s92bq9nebrcpbmrx9br 2461093 2461092 2026-09-25T22:51:59Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2461093 wikitext text/x-wiki rg1keiq11jvcrqr8o26zz423hfwo07b 2461094 2461093 2026-09-25T23:15:35Z Stashbot 7414 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2461094 wikitext text/x-wiki qonwad16dzqol9g7d8belosy91lvgx8 2461096 2461094 2026-09-26T02:00:30Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2461096 wikitext text/x-wiki 1ph0l3c80my7bvrt2corvuqujau4u73 2461099 2461096 2026-09-26T02:07:43Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) 2461099 wikitext text/x-wiki e3cn7nyc7f7djvufp0h2n5tqdxevkd8 2461100 2461099 2026-09-26T03:26:31Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1345252|Disable Score exec (T439297 T438443)]] 2461100 wikitext text/x-wiki 6eg6oo3tyvr3g0d14ss6849vtrkyd8n 2461101 2461100 2026-09-26T03:30:43Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1345252|Disable Score exec (T439297 T438443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461101 wikitext text/x-wiki nkpiop89uox9qh5vql5padydyz77vzz 2461102 2461101 2026-09-26T03:31:49Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2461102 wikitext text/x-wiki 3egyauw7baolsnltp1y3ydjbz5zr90w 2461103 2461102 2026-09-26T03:37:31Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345252|Disable Score exec (T439297 T438443)]] (duration: 11m 01s) 2461103 wikitext text/x-wiki o95ia5up20rp57ephgvbe84hdjd66pk 2461104 2461103 2026-09-26T06:27:32Z Stashbot 7414 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2461104 wikitext text/x-wiki gz5pdzje7h9k945d1jb72qckyamrg37 2461105 2461104 2026-09-26T06:28:09Z Stashbot 7414 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2461105 wikitext text/x-wiki ajyx8jij3gly8ybykquvlqasztxbjni 2461106 2461105 2026-09-26T06:29:21Z Stashbot 7414 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2461106 wikitext text/x-wiki 3xzut8hf0e7g3kf8mp1tgvuh9k04jpm 2461107 2461106 2026-09-26T06:29:30Z Stashbot 7414 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2461107 wikitext text/x-wiki m84vy5n5chiadwx0x9ecfecbzo8cp39 2461108 2461107 2026-09-26T06:30:18Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply 2461108 wikitext text/x-wiki gr8tp0brxin1em3f1bntnr898sjurqo 2461109 2461108 2026-09-26T06:30:57Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply 2461109 wikitext text/x-wiki nur3gvr07d7me15pnbmx2p85jwvzob8 2461110 2461109 2026-09-26T07:42:05Z Stashbot 7414 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2461110 wikitext text/x-wiki bkgftt45xkbbz1mi6dyyl36xmce27lc 2461111 2461110 2026-09-26T07:42:12Z Stashbot 7414 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2461111 wikitext text/x-wiki drwd0ctpebzcbimndqq3f5v1ooszifi 2461112 2461111 2026-09-26T07:44:00Z Stashbot 7414 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2461112 wikitext text/x-wiki t4ru5x72bs1bafo4w4jsdy0d7hx8xyy 2461113 2461112 2026-09-26T07:46:05Z Stashbot 7414 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2461113 wikitext text/x-wiki 902qlcybkjsuniob5tv40kd6avp18hr 2461114 2461113 2026-09-26T07:50:29Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2461114 wikitext text/x-wiki rm5d3hlosucs9dyifuqqyva05usm500 2461115 2461114 2026-09-26T07:52:29Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2461115 wikitext text/x-wiki 8axeg1umbiwij2rrsxdcus5oym8g2qg 2461116 2461115 2026-09-26T07:56:52Z Stashbot 7414 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1345258|Revert "Disable Score exec"]] 2461116 wikitext text/x-wiki j4jt9rob9i5laya5yk9x745ky3x6tvc 2461117 2461116 2026-09-26T08:00:54Z Stashbot 7414 oblivian@deploy1003: oblivian: Backport for [[gerrit:1345258|Revert "Disable Score exec"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461117 wikitext text/x-wiki 2x8c8x1a6xmfqmqe1kxfkf41sjq0to1 2461118 2461117 2026-09-26T08:02:11Z Stashbot 7414 oblivian@deploy1003: oblivian: Continuing with deployment 2461118 wikitext text/x-wiki aifvqrnsgmds6a018352ahbonpc49pc 2461119 2461118 2026-09-26T08:07:45Z Stashbot 7414 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345258|Revert "Disable Score exec"]] (duration: 10m 53s) 2461119 wikitext text/x-wiki 11uao8kw75gwswd8acwexedy80gmgki 0 17290 2461009 2460573 2026-09-25T14:05:50Z Stashbot 7414 hashar: Tag Quibble 1.22.0 @ 0d0bce00eedf44dd235703cb3eb1f60083d64752 # T437752 T426684 2461009 wikitext text/x-wiki exnmog59lhkivjsdshcak46we0pacfx 2461055 2461009 2026-09-25T15:05:10Z Stashbot 7414 dancy: Upgrading gitlab-cloud-runners (prod) from 1.35.1-do.6 to 1.35.7-do.5 (T439035) 2461055 wikitext text/x-wiki 5hay9ybtclmuhzcn211zptyrw9ft548 2461057 2461055 2026-09-25T15:20:29Z Stashbot 7414 dancy: Upgraded gitlab-cloud-runners (prod) from 1.35.1-do.6 to 1.35.7-do.5 (T439035) 2461057 wikitext text/x-wiki ed1vvyd6kix9kbzkmukpd1jn9j56xnn 2461058 2461057 2026-09-25T15:23:41Z Stashbot 7414 dancy: Upgrading gitlab-cloud-runners (prod) from 1.35.7-do.5 to 1.36.3-do.5 (T439035) 2461058 wikitext text/x-wiki 79ka7jgb4yzbgp8jscbfoh9p711lu38 2461059 2461058 2026-09-25T15:43:37Z Stashbot 7414 dancy: Upgraded gitlab-cloud-runners (prod) from 1.35.7-do.5 to 1.36.3-do.5 (T439035) 2461059 wikitext text/x-wiki jff8fbfcqw8u4h13utf276pl144a65c 0 91251 2461065 2459736 2026-09-25T17:54:01Z SSingh (WMF) 13928 added instructions on how to query for urldownloader stuff 2461065 wikitext text/x-wiki bivs7l2sdj7js8j9r5ozf3w4w0v9pj3 2461068 2461065 2026-09-25T18:03:44Z SSingh (WMF) 13928 /* Troubleshooting */ add services which depend on urldownloaders 2461068 wikitext text/x-wiki 6jgb3ab6c9fosbbhqthla1m0bs3jx06 2461069 2461068 2026-09-25T18:04:14Z SSingh (WMF) 13928 /* Troubleshooting */ 2461069 wikitext text/x-wiki 42p9d41mwqg51e5w29hx4yr2ldijo99 2461070 2461069 2026-09-25T18:15:28Z SSingh (WMF) 13928 move services to top 2461070 wikitext text/x-wiki 23xca6u89b6d69d288n3ju2t80fyu0h 2461071 2461070 2026-09-25T18:15:42Z SSingh (WMF) 13928 /* Service */ 2461071 wikitext text/x-wiki 4bvb4kk9xxm4185j03e7cbs5naet2m6 2461077 2461071 2026-09-25T18:28:47Z SSingh (WMF) 13928 add dashboards information (needs more work) 2461077 wikitext text/x-wiki 65oe5flzokrhhd70i8q3occdt960i28 0 442308 2461078 2195373 2026-09-25T18:29:15Z XCollazo-WMF 31846 2461078 wikitext text/x-wiki 4rpaba652wjhc21ms9ui2xtmpdirz0u 0 449160 2461095 2460834 2026-09-26T00:02:41Z Dexbot 30554 Bot: Updating the report 2461095 wikitext text/x-wiki 1029mj1h1n743385h9dr0d11g2gql97 0 455989 2461063 2459120 2026-09-25T17:40:34Z KReid-WMF 49588 /* Plan */ Add intro to the plan section 2461063 wikitext text/x-wiki enmq5u5njgqcfcfj9hlz2hhfocvcj5j 2461064 2461063 2026-09-25T17:47:22Z KReid-WMF 49588 /* Plan */ Add note with link to list of experiments 2461064 wikitext text/x-wiki bqkblvqnbjq5jzgb2we1bxulfwnhayv 0 459290 2461030 2459649 2026-09-25T14:31:25Z XCollazo-WMF 31846 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2461030 wikitext text/x-wiki palplepjd27aes2j3ct18858oe794yq 498 459970 2461061 2408743 2026-09-25T16:15:17Z Stashbot 7414 wmbot~difronzo@tools-bastion-15: deployed v0.10.0 (by DiFronzo on GitHub) 2461061 wikitext text/x-wiki tifljxpejzs11rq9pkaleweuvwew1ph 0 460652 2461098 2458688 2026-09-26T02:00:52Z DeploymentCalendarTool 20896 Add last week 2461098 wikitext text/x-wiki j0ww7z0c46crteoisrgioy35trepsqq 12 460726 2461062 2460265 2026-09-25T16:20:02Z BryanDavis 1604 /* Before you begin */ <TOOL NAME> placeholder 2461062 wikitext text/x-wiki dp0gmkey1oag7kc7kploaacke0y4w4e 498 460727 2461060 2026-09-25T16:12:52Z Stashbot 7414 wmbot~difronzo@tools-bastion-15: deployed face-service-v0.10.0 (by DiFronzo on GitHub) 2461060 wikitext text/x-wiki 1let8y78qt1j7640r31pvwdejaml480