Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.14 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2444949 2444632 2026-08-06T13:15:40Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager (T408786) 2444949 wikitext text/x-wiki c9a2n92ldwkgadybtidmfvo5ppfp35d 2444961 2444949 2026-08-06T13:36:43Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager (T408786) 2444961 wikitext text/x-wiki 2af7nb8kyi7llf7gnjudodpfrxn6rg0 2445009 2444961 2026-08-06T15:39:17Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli 2445009 wikitext text/x-wiki lalrtek9pyi6qtfe0i7i95gmxokqpx9 2445010 2445009 2026-08-06T15:42:29Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli 2445010 wikitext text/x-wiki 4jhb90p0cdj0j43ksvne2w4udfyafcf 2445021 2445010 2026-08-06T16:26:42Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2445021 wikitext text/x-wiki j6cizsm95ft098gjubq22nwo5mcvd7d 2445024 2445021 2026-08-06T16:41:24Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2445024 wikitext text/x-wiki 5mm5heui46je8o53u0c9t416dxpbgqj 2445028 2445024 2026-08-06T16:50:29Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2445028 wikitext text/x-wiki g057o62dovnrmw8cdbxzw2zyjgeoiqn 2445030 2445028 2026-08-06T16:55:33Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2445030 wikitext text/x-wiki gboawdt2lud6cay4hn4kwhq2xezsbj2 0 4108 2444948 2444790 2026-08-06T13:14:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1315128]] to Thursday, August 06 UTC late backport window 2444948 wikitext text/x-wiki ghjaubhed3r6wtn8yhhhsoxc3is50y5 2445023 2444948 2026-08-06T16:37:00Z ScheduleDeploymentBot 37566 Add [[gerrit:1319939]] to Thursday, August 06 UTC late backport window 2445023 wikitext text/x-wiki bo4dph9qdfyc8ht51t1rv1urq2lktb2 2445029 2445023 2026-08-06T16:51:55Z ScheduleDeploymentBot 37566 Add [[gerrit:1320230]] to Thursday, August 06 UTC late backport window 2445029 wikitext text/x-wiki nmvlgcyijca4w7n5qh3vtj8dha27725 2445044 2445029 2026-08-06T18:34:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1322060]] to Thursday, August 06 UTC late backport window 2445044 wikitext text/x-wiki nhja58ctksi4tzzhsn40sdg3iv00pij 2445045 2445044 2026-08-06T18:34:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1321639]] to Thursday, August 06 UTC late backport window 2445045 wikitext text/x-wiki sys7yey7ktklkbp5qg6duv1yy6mmod0 2445046 2445045 2026-08-06T18:52:11Z ScheduleDeploymentBot 37566 Add [[gerrit:1322054]] to Thursday, August 06 UTC late backport window 2445046 wikitext text/x-wiki j3h85gdv7zsksxagb2eysrsvmzkyo0x 2445047 2445046 2026-08-06T18:52:59Z ScheduleDeploymentBot 37566 Add [[gerrit:1322038]] to Thursday, August 06 UTC late backport window 2445047 wikitext text/x-wiki avwf5gxmbzdfyrn1k1qbv84xscfye0b 2445049 2445047 2026-08-06T19:11:15Z CMing (WMF) 23277 /* {{Deployment_day|date=2026-08-05}} */ 2445049 wikitext text/x-wiki omfvo2ycufpdmpg8wr9nqsobkqi3xxu 498 6726 2444996 2444596 2026-08-06T14:51:09Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli 2444996 wikitext text/x-wiki okldpelptvzzfep84yuivmnjykwn96b 2444997 2444996 2026-08-06T14:51:14Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli 2444997 wikitext text/x-wiki avqeniglzpw8i3mwdcj4idmf4q8jr7p 2445007 2444997 2026-08-06T15:31:20Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli 2445007 wikitext text/x-wiki 7z7cg2vx4pq8x8zz014alqh85arigez 2445008 2445007 2026-08-06T15:34:09Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli 2445008 wikitext text/x-wiki 56befwkgrpcbhsw8lx7l5w3ubsez2tt 2445014 2445008 2026-08-06T15:55:02Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2445014 wikitext text/x-wiki 45jbs4ni2fro5unjuj592to7emp0xm8 2445018 2445014 2026-08-06T16:07:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2445018 wikitext text/x-wiki bd5sr9v2xtzf02fmhv5hp5t7acujkbf 2445025 2445018 2026-08-06T16:43:21Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2445025 wikitext text/x-wiki 1vbhwxl00zqswv34kpshkwvd1t2zhoi 2445026 2445025 2026-08-06T16:47:58Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2445026 wikitext text/x-wiki fhp9o7zzo07pype7uii51j03vru290n 0 7919 2444913 2444912 2026-08-06T12:08:54Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2444913 wikitext text/x-wiki fnbvj85x0g40ec8qmurvqpspfjgq8rm 2444914 2444913 2026-08-06T12:09:59Z Stashbot 7414 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2444914 wikitext text/x-wiki q4igmsr4p3eku4y7s6czojgd8zxhjin 2444915 2444914 2026-08-06T12:12:21Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply 2444915 wikitext text/x-wiki 6jbft6l8hqsmif6udmz3vjxxj4eoliq 2444916 2444915 2026-08-06T12:12:26Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply 2444916 wikitext text/x-wiki hsfl3l7qsl1j0gwqc3o61gjki1y211u 2444917 2444916 2026-08-06T12:38:05Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-client1002.eqiad.wmnet with OS bookworm 2444917 wikitext text/x-wiki rtt79oqka402vwyjs8q1u5ffdmlxkqn 2444918 2444917 2026-08-06T12:39:36Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . 2444918 wikitext text/x-wiki aesto1m8nrw7sibciuugw83uruicoty 2444919 2444918 2026-08-06T12:41:05Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . 2444919 wikitext text/x-wiki 44gurbtn0wgsvstl39sfcvujwwxg0ua 2444920 2444919 2026-08-06T12:45:08Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . 2444920 wikitext text/x-wiki qlmltotz550v1e55zmp65dv8fy4pwgl 2444921 2444920 2026-08-06T12:46:13Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . 2444921 wikitext text/x-wiki 0klii3xl3r0s1mc2hnubpk45cwtbqv2 2444922 2444921 2026-08-06T12:48:22Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2444922 wikitext text/x-wiki abr70zefiyfqq27ta9wpfyw8hvy8vkd 2444923 2444922 2026-08-06T12:50:28Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2444923 wikitext text/x-wiki 7egbnjadvc6e5g7u5h007tm5qyvyt2w 2444924 2444923 2026-08-06T12:51:51Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2444924 wikitext text/x-wiki acwxbnuuu6gch0tuyescktaofkev8kj 2444925 2444924 2026-08-06T12:53:20Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2444925 wikitext text/x-wiki 8pr0h92zvprxl4dbopugsruuoftbwb2 2444926 2444925 2026-08-06T12:53:25Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-client1002.eqiad.wmnet with reason: host reimage 2444926 wikitext text/x-wiki 00lafzprx25hufhny11e6d44wnheubl 2444927 2444926 2026-08-06T12:54:47Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2444927 wikitext text/x-wiki mluc2mwu2lmthhkjt0g14put9kflcm8 2444928 2444927 2026-08-06T12:55:30Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . 2444928 wikitext text/x-wiki av0mvnsgxtl5jss4q1buy74jgljvjrw 2444929 2444928 2026-08-06T12:57:03Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2444929 wikitext text/x-wiki g3r5jf1lzw2dro36wiaqqf2sd6n9dlm 2444930 2444929 2026-08-06T12:57:05Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . 2444930 wikitext text/x-wiki t7razl5sfd7yqllw63nwpxesiakhixr 2444931 2444930 2026-08-06T12:57:52Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . 2444931 wikitext text/x-wiki 9q75k49knac23lhn7468vj3nhilfk3q 2444932 2444931 2026-08-06T12:58:23Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-client1002.eqiad.wmnet with reason: host reimage 2444932 wikitext text/x-wiki qy28vnlq1m9dmt2vqgup76rk3ae9k2j 2444933 2444932 2026-08-06T12:59:02Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . 2444933 wikitext text/x-wiki jqzra2e4r4x195ts82jrbtpczi0wj74 2444934 2444933 2026-08-06T13:00:38Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . 2444934 wikitext text/x-wiki 1dqbzrj27b7g4p4xntl2rgngfrrp5io 2444935 2444934 2026-08-06T13:02:01Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2444935 wikitext text/x-wiki j8kd8apmak14mlcyf970t5xh53a7h1a 2444936 2444935 2026-08-06T13:03:23Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2444936 wikitext text/x-wiki q4l3uvykbmcyeqqkoblp4xfn9fng7lm 2444937 2444936 2026-08-06T13:04:00Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2444937 wikitext text/x-wiki f3w4lp09rw3kg43el9foz58qajgt101 2444938 2444937 2026-08-06T13:04:33Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-client1002.eqiad.wmnet with OS bookworm 2444938 wikitext text/x-wiki mpyi3x1m5vbfmpwvl0rmm9idx0g4336 2444939 2444938 2026-08-06T13:05:02Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2444939 wikitext text/x-wiki pj5mrk3kln2fdv05a9r6zgscrfmxtkz 2444940 2444939 2026-08-06T13:05:47Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2444940 wikitext text/x-wiki rkcnjllo7e3548xfgcggkpj34e8axfd 2444941 2444940 2026-08-06T13:05:51Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2444941 wikitext text/x-wiki 1x4petlrztjil0pq89bqlz1043z752g 2444942 2444941 2026-08-06T13:06:36Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-client1002.eqiad.wmnet with OS bookworm 2444942 wikitext text/x-wiki kdzdj7u69du29u7aijweso9fz8jmdkm 2444943 2444942 2026-08-06T13:06:52Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2444943 wikitext text/x-wiki n04g32qrjeqy19eqe44afvulrc0jmg1 2444944 2444943 2026-08-06T13:09:20Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2444944 wikitext text/x-wiki jdo9mgm4jw7mnemxlj5k0tj0zhso1sy 2444945 2444944 2026-08-06T13:11:56Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2444945 wikitext text/x-wiki numt3112eqz3g9svvbx4rmn2ftvqfb9 2444947 2444945 2026-08-06T13:13:59Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2444947 wikitext text/x-wiki 7xxhanzalhb7lfjdl02n4e94qyno3r7 2444950 2444947 2026-08-06T13:16:18Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2444950 wikitext text/x-wiki kh2c3jimq7u8xtq89o2mm8to3460b1m 2444951 2444950 2026-08-06T13:17:17Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2444951 wikitext text/x-wiki h7470pw59vtilzeebzm75uyp8jx8ikp 2444952 2444951 2026-08-06T13:18:34Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2444952 wikitext text/x-wiki dxrq0623c5dxvs56ulyu3x8ttbekrhr 2444953 2444952 2026-08-06T13:18:37Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-client1002.eqiad.wmnet with reason: host reimage 2444953 wikitext text/x-wiki 3gvt4ebllg9qk7aagrdchtq7wh3g23s 2444954 2444953 2026-08-06T13:19:49Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2444954 wikitext text/x-wiki a0vvxj6yzockpq5hifukbnhjj6pshs5 2444955 2444954 2026-08-06T13:22:23Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2444955 wikitext text/x-wiki surtcua4kp7l7s3dmpqa2xziykx026u 2444956 2444955 2026-08-06T13:23:20Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-client1002.eqiad.wmnet with reason: host reimage 2444956 wikitext text/x-wiki ho1jia8t796w89opglw1c2klgqwcqq9 2444958 2444956 2026-08-06T13:26:11Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2444958 wikitext text/x-wiki fso8mcpg0ho6w4w36fxavf8dsupy5e4 2444959 2444958 2026-08-06T13:29:12Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2444959 wikitext text/x-wiki 4u0pl9wpnqyd13lijdsrsnwvvo8o28w 2444960 2444959 2026-08-06T13:31:48Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2444960 wikitext text/x-wiki tnnt2o9xp9piiedmszihqk12dgamr7o 2444963 2444960 2026-08-06T13:44:02Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2444963 wikitext text/x-wiki 51zrj4j01m33xtu5k0z160expyk0pzi 2444964 2444963 2026-08-06T13:49:15Z Stashbot 7414 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2444964 wikitext text/x-wiki 00asjbf854qkre7isw0cu2nonleb3my 2444966 2444964 2026-08-06T13:56:09Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2444966 wikitext text/x-wiki 8xa39lyzp68ql0jobmngvep2cfkfvos 2444967 2444966 2026-08-06T13:58:08Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2444967 wikitext text/x-wiki dh83oxuqz1vhen4kmvyuuvs6yrqwxfw 2444968 2444967 2026-08-06T13:58:15Z Stashbot 7414 swfrench-wmf: authdns update to direct eqiad-associated etcd clients back to eqiad - T428495 2444968 wikitext text/x-wiki p32ajgier1wgvh0vqkqfkjhh4oe80sw 2444969 2444968 2026-08-06T14:02:23Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-client1002.eqiad.wmnet with OS bookworm 2444969 wikitext text/x-wiki 8ugjl4tjksw3e9by48edc1levrbhads 2444970 2444969 2026-08-06T14:04:23Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1002.eqiad.wmnet with OS bookworm 2444970 wikitext text/x-wiki 7jxx3ij0durjsowgd4mpddkc244u1s4 2444971 2444970 2026-08-06T14:04:31Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1001.eqiad.wmnet with OS bookworm 2444971 wikitext text/x-wiki thzkj7dz9y4kn3fs8r2y7hp8zfdo7uy 2444972 2444971 2026-08-06T14:04:45Z Stashbot 7414 swfrench-wmf: begin rolling restart of confd in drmrs, eqiad, esams, magru - T428495 2444972 wikitext text/x-wiki ggbytvdf2hret12xc5bsbz7mthw7b5y 2444974 2444972 2026-08-06T14:11:51Z Stashbot 7414 swfrench-wmf: restarted navtiming on webperf1003 - T428495 2444974 wikitext text/x-wiki ozwfv95b9xskck1hv9n3m0yf1uopt2b 2444975 2444974 2026-08-06T14:12:32Z Stashbot 7414 sukhe: sudo cumin 'A:cp-text' "disable-puppet 'merging CR 1290731'": T425441 2444975 wikitext text/x-wiki 1vk2xaafme2i9xubgdba1knpwptqxyq 2444976 2444975 2026-08-06T14:14:56Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-ui1001.eqiad.wmnet with OS bookworm 2444976 wikitext text/x-wiki 9ncx21fyz50xvxdn53kr4c831i9vmss 2444977 2444976 2026-08-06T14:16:14Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-presto1001.eqiad.wmnet with OS bookworm 2444977 wikitext text/x-wiki 0vdr25jo95yxy13k36ma99aaomona40 2444978 2444977 2026-08-06T14:18:57Z Stashbot 7414 swfrench-wmf: begin rolling restart of confd in codfw, eqsin, ulsfo for hosts in the wikimedia.org domain - T428495 2444978 wikitext text/x-wiki cg5zlrshlk7n3bibe3la8nwpw12w012 2444979 2444978 2026-08-06T14:23:04Z Stashbot 7414 sukhe: sudo cumin -b2 'A:cp-text' "run-puppet-agent --enable 'merging CR 1290731'": T425441 2444979 wikitext text/x-wiki 4frzr3bfsirnkryy4bo86i590a71wlh 2444980 2444979 2026-08-06T14:27:45Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-presto1001.eqiad.wmnet with reason: host reimage 2444980 wikitext text/x-wiki hgc9vq1kjl1oy7ig4zhy0klh49570v9 2444981 2444980 2026-08-06T14:28:04Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-ui1001.eqiad.wmnet with reason: host reimage 2444981 wikitext text/x-wiki bi1x1w88q2q3cche9dnokt4itbfq39a 2444982 2444981 2026-08-06T14:34:49Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-presto1001.eqiad.wmnet with reason: host reimage 2444982 wikitext text/x-wiki eft92kstf1zy38wh7y905q5bku6aahx 2444984 2444982 2026-08-06T14:38:26Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-ui1001.eqiad.wmnet with reason: host reimage 2444984 wikitext text/x-wiki tgyhjtvqxfcq60adlmj5cttgj1u09l5 2444985 2444984 2026-08-06T14:40:31Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-worker1002.eqiad.wmnet with OS bookworm 2444985 wikitext text/x-wiki ffff4i7m59ra43f4yxwx30wwqfwitqu 2444986 2444985 2026-08-06T14:41:00Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T428495) 2444986 wikitext text/x-wiki 0kcoabi3adyevqu0agj6tw9zg8wjuxf 2444987 2444986 2026-08-06T14:42:39Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T428495) 2444987 wikitext text/x-wiki fb6lem0z9w5q7w4et2lcoo9btm0jdic 2444988 2444987 2026-08-06T14:42:45Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1002.eqiad.wmnet with OS bookworm 2444988 wikitext text/x-wiki e1azak6dqlr26dszngisfrviklhmiac 2444989 2444988 2026-08-06T14:42:48Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-worker1002.eqiad.wmnet with OS bookworm 2444989 wikitext text/x-wiki b5xud6d8owftxiqnxe8qdpc774p14fi 2444990 2444989 2026-08-06T14:43:14Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1002.eqiad.wmnet with OS bookworm 2444990 wikitext text/x-wiki jnxatt9s99ersb8ub7j3ls51ytbb37y 2444991 2444990 2026-08-06T14:44:22Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-esams (T428495) 2444991 wikitext text/x-wiki 6yi0yfd4kdaliz9emxfns8qhqr9obr2 2444993 2444991 2026-08-06T14:46:16Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-esams (T428495) 2444993 wikitext text/x-wiki q4k1kdr2duligkbrx3vsi6ycv2jisc6 2444994 2444993 2026-08-06T14:48:43Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-worker1001.eqiad.wmnet with OS bookworm 2444994 wikitext text/x-wiki rt86jltc9ogox1t73qbsxfc0e6gbn23 2444995 2444994 2026-08-06T14:49:30Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-magru (T428495) 2444995 wikitext text/x-wiki nik3t9ddtd1m7n4nwc6rcre3kdjbekr 2444999 2444995 2026-08-06T14:52:23Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-magru (T428495) 2444999 wikitext text/x-wiki gmrqo5mueba46rygy5tunt315lrn4ru 2445000 2444999 2026-08-06T14:54:42Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-presto1001.eqiad.wmnet with OS bookworm 2445000 wikitext text/x-wiki in53365wmx9tg2wdxh9twkpo7zr4m7w 2445001 2445000 2026-08-06T14:55:16Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-ui1001.eqiad.wmnet with OS bookworm 2445001 wikitext text/x-wiki n3sjwu3ighgib188u9xc3475coda5pu 2445002 2445001 2026-08-06T14:55:48Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-drmrs (T428495) 2445002 wikitext text/x-wiki ntwimtklvnyp29234ma1dkgnq4ggun9 2445003 2445002 2026-08-06T14:57:45Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-drmrs (T428495) 2445003 wikitext text/x-wiki 0xrqi6kxl87dqneim9c59hhu657ptz7 2445004 2445003 2026-08-06T14:58:35Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1001.eqiad.wmnet with OS bookworm 2445004 wikitext text/x-wiki r42abzaswgl03z9gascc7q2kqiozv5o 2445005 2445004 2026-08-06T15:29:03Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-worker1002.eqiad.wmnet with OS bookworm 2445005 wikitext text/x-wiki 7pfts32b62lrqvy2dbhh4byf38fs761 2445006 2445005 2026-08-06T15:29:10Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-worker1001.eqiad.wmnet with OS bookworm 2445006 wikitext text/x-wiki gikrhnwrosqldqg89bem1e2kuii6uky 2445011 2445006 2026-08-06T15:43:21Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1001.eqiad.wmnet with OS bookworm 2445011 wikitext text/x-wiki ptsgbcbgyjltz0xq9atdu3nzvsi028q 2445015 2445011 2026-08-06T15:57:54Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1002.eqiad.wmnet with OS bookworm 2445015 wikitext text/x-wiki 1d8afhc7at2v54i524f7v6lpcy1srqe 2445016 2445015 2026-08-06T16:00:02Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-worker1001.eqiad.wmnet with reason: host reimage 2445016 wikitext text/x-wiki hg56cadujpfjjlpoquatui84g3w6ys0 2445017 2445016 2026-08-06T16:05:45Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-worker1001.eqiad.wmnet with reason: host reimage 2445017 wikitext text/x-wiki 5wwqyag6fhsdbd64rcswepcq3avbm7n 2445019 2445017 2026-08-06T16:16:04Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-worker1002.eqiad.wmnet with reason: host reimage 2445019 wikitext text/x-wiki dstn2cn19vlonjpxc8g6o0eh1108v57 2445020 2445019 2026-08-06T16:19:50Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-worker1002.eqiad.wmnet with reason: host reimage 2445020 wikitext text/x-wiki 6q9rptq4d8opqvcs21yjdeedroq7dly 2445022 2445020 2026-08-06T16:35:28Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-worker1001.eqiad.wmnet with OS bookworm 2445022 wikitext text/x-wiki olo9whjm5zt20o891rqytw2qah59lap 2445027 2445022 2026-08-06T16:50:10Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-worker1002.eqiad.wmnet with OS bookworm 2445027 wikitext text/x-wiki f9094jiojqdh78m8ba35ypc8v57tjoh 2445034 2445027 2026-08-06T17:55:59Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie 2445034 wikitext text/x-wiki 5zker8yrkk26xz85q3n6u3ybqla3sv5 2445037 2445034 2026-08-06T18:17:07Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Deploying gerrit:1321603 (T107188) 2445037 wikitext text/x-wiki sd29kjexls7ucyps97l0zaz4kcd6tgw 2445038 2445037 2026-08-06T18:18:15Z Stashbot 7414 ladsgroup@deploy1003: Stopping before sync operations 2445038 wikitext text/x-wiki k6mmhy76lx1szu9f16r5hu9jz4n2hgr 2445040 2445038 2026-08-06T18:19:48Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Deploying gerrit:1321603 (T107188) 2445040 wikitext text/x-wiki 0da4y1rqrr4118oyfvezps0tjecnrr7 2445041 2445040 2026-08-06T18:25:30Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Deploying gerrit:1321603 (T107188) (duration: 06m 08s) 2445041 wikitext text/x-wiki slw300djye4knz22k99ctzjr6ice8cx 2445048 2445041 2026-08-06T19:00:45Z Stashbot 7414 cdobbins@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie 2445048 wikitext text/x-wiki 0kh2mk2cn9u4kgralhzcrytw7k2tgwn 2445051 2445048 2026-08-06T19:31:28Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2445051 wikitext text/x-wiki hwsfxtnc02yodz283nys3utzc0lvc3p 2445052 2445051 2026-08-06T19:31:55Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2445052 wikitext text/x-wiki afyu4wjfn8p6vtpcl0n8q73v80me7p2 2445053 2445052 2026-08-06T19:41:00Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2445053 wikitext text/x-wiki sazt6pm1q20iev6ugydxer4n1i6ct99 2445054 2445053 2026-08-06T19:41:29Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2445054 wikitext text/x-wiki dt9ccnh4biee12bues8qjbc4hkr2ptv 2445055 2445054 2026-08-06T20:09:26Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1315128|Add new Apple app site association file for Test Wiki (T432412)]] 2445055 wikitext text/x-wiki 85921cuclmbyhf1iv7hxr5px66hjjxs 2445056 2445055 2026-08-06T20:11:17Z Stashbot 7414 cjming@deploy1003: cjming, tsev: Backport for [[gerrit:1315128|Add new Apple app site association file for Test Wiki (T432412)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2445056 wikitext text/x-wiki o1t1bx5yojrfjp4h8s47s7wn2oidcbq 2445057 2445056 2026-08-06T20:14:41Z Stashbot 7414 cjming@deploy1003: cjming, tsev: Continuing with deployment 2445057 wikitext text/x-wiki 7gk4tr2b5dxyxsxwqq9cnjogrtkz9zt 2445058 2445057 2026-08-06T20:18:51Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1315128|Add new Apple app site association file for Test Wiki (T432412)]] (duration: 09m 22s) 2445058 wikitext text/x-wiki tq5l3ra24leiv0mtfxc3zeefnyoswi9 2445059 2445058 2026-08-06T20:24:29Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1319939|Revert "frwiki: change to Wikipedia 25 logo" (T430409)]], [[gerrit:1320230|cirrus: Enable building redirect documents (T204089)]] 2445059 wikitext text/x-wiki 3ld1d42ffwct363upgztlyz2xp1gi05 2445062 2445059 2026-08-06T20:26:19Z Stashbot 7414 cjming@deploy1003: cjming, ebernhardson, chlod: Backport for [[gerrit:1319939|Revert "frwiki: change to Wikipedia 25 logo" (T430409)]], [[gerrit:1320230|cirrus: Enable building redirect documents (T204089)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2445062 wikitext text/x-wiki qnh9q49ptbnpseygmg9y8cvcct7r9r8 2445063 2445062 2026-08-06T20:27:05Z Stashbot 7414 cjming@deploy1003: cjming, ebernhardson, chlod: Continuing with deployment 2445063 wikitext text/x-wiki ri86m1fjhu5ggi1d2mkfwneunkrn047 2445067 2445063 2026-08-06T20:31:11Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319939|Revert "frwiki: change to Wikipedia 25 logo" (T430409)]], [[gerrit:1320230|cirrus: Enable building redirect documents (T204089)]] (duration: 06m 41s) 2445067 wikitext text/x-wiki d0yjbn9vf0yobkn4v76e8tlrriao4en 2445068 2445067 2026-08-06T20:32:05Z Stashbot 7414 ebernhardson@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply 2445068 wikitext text/x-wiki b1ixo9i901vjih59mqyv1b1ct4xs969 2445069 2445068 2026-08-06T20:32:09Z Stashbot 7414 ebernhardson@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply 2445069 wikitext text/x-wiki ad29kviuwo0twbcley9wwxlio3hx0e6 2445070 2445069 2026-08-06T20:37:01Z Stashbot 7414 ebernhardson@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply 2445070 wikitext text/x-wiki lr16eyxlr5hn7wn6qly2604txwmb12h 2445071 2445070 2026-08-06T20:37:09Z Stashbot 7414 ebernhardson@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply 2445071 wikitext text/x-wiki gyui3u5lne6lzj3gedn6hxzhw1n9h5z 2445072 2445071 2026-08-06T20:40:02Z Stashbot 7414 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1322038|outreachwiki: disable bureaucrats ability to locally remove users from importer usergroup (T431959)]], [[gerrit:1321639|Turn on feature flag for custom lists for betawiki (T434027)]], [[gerrit:1322054|tcywiki: update logos for 10years anniversary (T434176)]] 2445072 wikitext text/x-wiki 913lw436ithhx8scxt6eqkgwlt3w9a4 2445073 2445072 2026-08-06T20:41:35Z Stashbot 7414 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2445073 wikitext text/x-wiki sas07hjillzo6t6r7au2w3vjeu2vomk 2445074 2445073 2026-08-06T20:41:39Z Stashbot 7414 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2445074 wikitext text/x-wiki klscdy1e8qrmt7vpi80oqc6e47sn5tg 2445075 2445074 2026-08-06T20:41:52Z Stashbot 7414 aude@deploy1003: lmora, aude, anzx: Backport for [[gerrit:1322038|outreachwiki: disable bureaucrats ability to locally remove users from importer usergroup (T431959)]], [[gerrit:1321639|Turn on feature flag for custom lists for betawiki (T434027)]], [[gerrit:1322054|tcywiki: update logos for 10years anniversary (T434176)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be 2445075 wikitext text/x-wiki qlkwdcdwzlwiqwzcmv31z0rikxmn6kl 2445076 2445075 2026-08-06T20:44:07Z Stashbot 7414 aude@deploy1003: lmora, aude, anzx: Continuing with deployment 2445076 wikitext text/x-wiki gdylsjrpho594gzfj9yy06ma3dpc8w4 2445077 2445076 2026-08-06T20:48:14Z Stashbot 7414 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1322038|outreachwiki: disable bureaucrats ability to locally remove users from importer usergroup (T431959)]], [[gerrit:1321639|Turn on feature flag for custom lists for betawiki (T434027)]], [[gerrit:1322054|tcywiki: update logos for 10years anniversary (T434176)]] (duration: 08m 12s) 2445077 wikitext text/x-wiki j67nzvyhyjodlbbg8sbpy2ffslhgmcd 2445080 2445077 2026-08-06T21:29:14Z Stashbot 7414 maryum: Deploy security patch for T434189 2445080 wikitext text/x-wiki 3njrpl7b9ki9totv4xlea9kdaq4jdla 2445081 2445080 2026-08-06T21:39:07Z Stashbot 7414 maryum: Deploy security patch for T433070 2445081 wikitext text/x-wiki 5s05lj9z3t0l4418z3k24j0e3nieih8 2445085 2445081 2026-08-07T02:00:51Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2445085 wikitext text/x-wiki t6byg95eg5kr8v23eex9wkot8zae74p 2445086 2445085 2026-08-07T02:07:47Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 56s) 2445086 wikitext text/x-wiki jghfha9lxihupwpqhiz0pwfva0ski2d 2445088 2445086 2026-08-07T02:37:56Z Stashbot 7414 denisse: Extended volume on prometheus2007 tor the disk space alert as per https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_host_running_out_of_space 2445088 wikitext text/x-wiki kaawqkbt6sd6x5rn2916or8epw6m1px 2445089 2445088 2026-08-07T02:42:33Z Stashbot 7414 denisse: Extended volume on prometheus2008 for the disk space alert as per https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_host_running_out_of_space 2445089 wikitext text/x-wiki 0c3z1dqquxd5qtzqrguqkhugydpio40 2445090 2445089 2026-08-07T05:45:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.decommission 2445090 wikitext text/x-wiki e9go9jwr7ibayng4ppp3b0p87wnfsvl 2445091 2445090 2026-08-07T05:46:27Z Stashbot 7414 marostegui@cumin1003: Removing db1178 from zarcillo T433471 2445091 wikitext text/x-wiki prdkbt98zu9gom29bfqqfsp7x2zd22x 2445092 2445091 2026-08-07T05:46:30Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) 2445092 wikitext text/x-wiki 3j8h31dvdzav5zcoa418kag5qi653ie 2445093 2445092 2026-08-07T05:49:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1178.eqiad.wmnet 2445093 wikitext text/x-wiki td6enwbfxw58fh2y7uam5e1anzkiynp 2445094 2445093 2026-08-07T05:55:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2445094 wikitext text/x-wiki slra0dmzgixy0jyr25xpcya4vt00xnp 2445095 2445094 2026-08-07T06:06:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1178.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2445095 wikitext text/x-wiki bsxr67oibjfqr1ubk7yk6bad48x3dlq 2445096 2445095 2026-08-07T06:08:38Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1178.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2445096 wikitext text/x-wiki aceptl0n63aqwsmb5edr15pqov6vhn7 2445097 2445096 2026-08-07T06:08:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2445097 wikitext text/x-wiki 5v7xx91xls47t5ixm64bj597108x916 2445098 2445097 2026-08-07T06:08:43Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1178.eqiad.wmnet 2445098 wikitext text/x-wiki cbfa0678khsrbazmisysn335jy0b4ss 2445099 2445098 2026-08-07T06:36:00Z Stashbot 7414 jayme: updated istio to 1.29.4 on wikikube eqiad - T427401 2445099 wikitext text/x-wiki gup2sls4frodcjeb4tf415lnw1393h1 2445102 2445099 2026-08-07T07:37:29Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2445102 wikitext text/x-wiki despudbxjjirqazz8oy0gk28l523klg 2445103 2445102 2026-08-07T07:38:41Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2445103 wikitext text/x-wiki mjynaypwzcw7qjd14zwksbuk18rympj 2445104 2445103 2026-08-07T07:41:20Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2445104 wikitext text/x-wiki ad0jwi1i249q3q02w57htrtuu3hjgub 2445105 2445104 2026-08-07T07:45:05Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2445105 wikitext text/x-wiki dxllhgax0lhxs4996htvc83e7b1r4ub 2445106 2445105 2026-08-07T07:45:50Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2445106 wikitext text/x-wiki 0e53b5d5owi6q2bok476ct2tdm1c28r 2445107 2445106 2026-08-07T07:47:03Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2445107 wikitext text/x-wiki 5lziktd3snxjfqaqupj5dq36uqtkyrz 2445108 2445107 2026-08-07T07:48:20Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2445108 wikitext text/x-wiki lgdfz0zaatt8o98z75oifvx9jfcid4d 2445109 2445108 2026-08-07T07:49:09Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2445109 wikitext text/x-wiki 69ayeaxg4oqk31q1e941uithkrz2uly 2445110 2445109 2026-08-07T07:50:34Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2445110 wikitext text/x-wiki 962i1enn2ncpdjr6bdp6amomtbj8s89 2445111 2445110 2026-08-07T07:51:47Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2445111 wikitext text/x-wiki 8ikp3z7bd6jtxepas1h4drd989nwu3g 2445112 2445111 2026-08-07T07:52:21Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2445112 wikitext text/x-wiki 489pg3ah1vf62tlyc744hwkbxtc08e6 2445113 2445112 2026-08-07T07:53:06Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2445113 wikitext text/x-wiki d77nhwokmbgucznwgftksg0hg2ml3uw 2445114 2445113 2026-08-07T07:54:06Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2445114 wikitext text/x-wiki 8mggy17knzexk1vkffae1urdamsxo8s 2445115 2445114 2026-08-07T07:54:43Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2445115 wikitext text/x-wiki fe64g0g8civh2307gyq2vji2id9mfxx 2445116 2445115 2026-08-07T08:00:09Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2445116 wikitext text/x-wiki 1fqtso9uk0r21ts6duagmtsos67yv55 2445117 2445116 2026-08-07T08:00:13Z Stashbot 7414 klausman@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime to get full picture of current BIOS settings beyond what Redfish shows 2445117 wikitext text/x-wiki jnlmk6y0x8039ffcsffk2fbbgas9ddk 2445118 2445117 2026-08-07T08:06:00Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2445118 wikitext text/x-wiki rtrotaftryggq1z0ww7k0riutxy840a 2445119 2445118 2026-08-07T08:06:55Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2445119 wikitext text/x-wiki 011l5p1isq4n0egf8nu82och68gnpww 2445120 2445119 2026-08-07T08:10:42Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2445120 wikitext text/x-wiki 4kcl2ih6w9i16x38uk6d86tcivccbmt 2445121 2445120 2026-08-07T08:13:44Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2445121 wikitext text/x-wiki ip6el7r16r3gjh5v5mv96yomzgouh1v 2445122 2445121 2026-08-07T08:14:53Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2445122 wikitext text/x-wiki ofssuh6epmpf1mu0jvr60sol1u4h27h 2445123 2445122 2026-08-07T08:16:23Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2445123 wikitext text/x-wiki lzk00lx3ueurk5rj7df310ctit41acd 2445124 2445123 2026-08-07T08:18:06Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2445124 wikitext text/x-wiki npjmc42jymh6gh7ppxjjhrdp3qblxk8 2445125 2445124 2026-08-07T08:20:00Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2445125 wikitext text/x-wiki 5gtizv7dgmyt9l3w562sodjxckt7zet 2445129 2445125 2026-08-07T09:15:48Z Stashbot 7414 jynus: started stress testing db1245 dbs T431115 2445129 wikitext text/x-wiki pa5a0b1rpnyklw2bnqzy1peztul07re 2445130 2445129 2026-08-07T09:59:42Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1156.eqiad.wmnet onto db1271.eqiad.wmnet 2445130 wikitext text/x-wiki mnvvwbalmud6hnh3j57ua1jyuc129p1 2445131 2445130 2026-08-07T09:59:46Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1156: Depool db1156.eqiad.wmnet to then clone it to db1271.eqiad.wmnet - marostegui@cumin1003 2445131 wikitext text/x-wiki 1kb620gof2hy9c51ek4ea86n3b7y6ar 2445133 2445131 2026-08-07T10:01:03Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1156: Depool db1156.eqiad.wmnet to then clone it to db1271.eqiad.wmnet - marostegui@cumin1003 2445133 wikitext text/x-wiki p2ty85ke900gfo36u5755kwys6qnjf5 2445134 2445133 2026-08-07T10:06:37Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 21 hosts with reason: cloning 2445134 wikitext text/x-wiki janq24aemceksype0jmhe0fu68pkrbe 2445135 2445134 2026-08-07T10:18:27Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2445135 wikitext text/x-wiki 1588jg8x2ac5kw0v2xg14ztl3wfhhri 2445136 2445135 2026-08-07T10:19:48Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2445136 wikitext text/x-wiki epus5whwk3qgujaebj878yt3c0cn3xq 2445137 2445136 2026-08-07T10:20:12Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2445137 wikitext text/x-wiki ci36nxt90hpspeb1nlli943aq4pvu0s 2445138 2445137 2026-08-07T10:20:48Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2445138 wikitext text/x-wiki 7jxbin1kj7m8k08b1s1pfb84pt5ogrd 2445139 2445138 2026-08-07T10:21:08Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2445139 wikitext text/x-wiki bmso32erg74pukkkj2yfgdgxwc1u2o2 2445140 2445139 2026-08-07T10:22:02Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2445140 wikitext text/x-wiki corunfjv13wxaaw2pinweazc5hlbmnu 2445141 2445140 2026-08-07T10:22:38Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2445141 wikitext text/x-wiki f30ah1c2ou3qqcbo5zlst6k6ujerxfh 2445142 2445141 2026-08-07T10:23:20Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2445142 wikitext text/x-wiki jeo7dtk2mbcaoyrrxj7c9pxnvdvab3i 2445143 2445142 2026-08-07T11:12:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1156: Pool db1156.eqiad.wmnet in after cloning 2445143 wikitext text/x-wiki bmx6yt0cdcqgfoabj1thop1vhr0miv5 2445146 2445143 2026-08-07T11:30:08Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Adding db1271 to dbctl', diff saved to https://phabricator.wikimedia.org/P95945 and previous config saved to /var/cache/conftool/dbconfig/20260807-113006-marostegui.json 2445146 wikitext text/x-wiki bh2r1pm6atbloi4gcamlkkzps7yr496 2445147 2445146 2026-08-07T11:35:03Z Stashbot 7414 bjensen: sudo -i reprepro -C main include bookworm-wikimedia ${HOME}/httpbb/bookworm/httpbb_${VERSION?}-1_amd64.changes #T434052 2445147 wikitext text/x-wiki 3blyocdvovci7xfy6oqs9wtu5wofcuy 2445148 2445147 2026-08-07T11:38:31Z Stashbot 7414 bjensen: sudo -i reprepro -C main include trixie-wikimedia ${HOME}/httpbb/trixie/httpbb_${VERSION?}-1+deb13u1_amd64.changes #T434052 2445148 wikitext text/x-wiki l43p7vv7x0w726gqdtubgvpdhpguz2k 0 15652 2445083 2397097 2026-08-06T23:15:47Z LSandergreen-WMF 46345 /* Upgrading CiviCRM */ fill in details on upgrade 2445083 wikitext text/x-wiki n99kql5huzg5mlj29ofzh9phicrra0q 0 17290 2445033 2444798 2026-08-06T17:55:37Z Stashbot 7414 dancy: Restarted gitlab-webhooks (T430410) 2445033 wikitext text/x-wiki kyhg9haxlp17y6ctq86uykv5b07grjz 2445036 2445033 2026-08-06T18:02:09Z Stashbot 7414 dancy: Restarted gitlab-webhooks (T430410) (revert) 2445036 wikitext text/x-wiki iuaz7p6081o8cb3gxzsthrfnjhr4btt 2445101 2445036 2026-08-07T07:25:41Z Stashbot 7414 hashar: Tag Quibble 1.19.0 @ a8a84ed1c0adb34688ce22066ebde8a2d584a7d4 # T432934 T432943 T427922 2445101 wikitext text/x-wiki lc5rayxl16nc2w6kujv407lbdm0wi02 498 30942 2445050 2444781 2026-08-06T19:21:48Z Stashbot 7414 andrewbogott: restarting osds one by one to pick up config changes 2445050 wikitext text/x-wiki mmwk0pmrzesv6gd5krnw2r4satb8m6e 2445060 2445050 2026-08-06T20:24:34Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_restart_mon_daemons 2445060 wikitext text/x-wiki luk0ra2bp5x8rgzgsmjkk0susegvw6d 2445061 2445060 2026-08-06T20:24:54Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_restart_mon_daemons (exit_code=0) 2445061 wikitext text/x-wiki 5qsrlap4ynkcnzpatavjpq59u366den 0 440867 2444962 2444859 2026-08-06T13:41:25Z JMeybohm (WMF) 16709 /* Updating */ add major update instructions 2444962 wikitext text/x-wiki 84eg56k37unng7u9dy5eju08wkkdsg7 2445126 2444962 2026-08-07T08:25:50Z JMeybohm (WMF) 16709 /* The Debian Packages */ fix publishing instructions 2445126 wikitext text/x-wiki tuvlqszawsmznuwsj1l3wfmchq1rtmq 0 441014 2444983 2444843 2026-08-06T14:35:10Z SFrench-WMF 37362 /* Depool etcd client traffic from a cluster */ Move pybal restart instructions into Move PyBals section 2444983 wikitext text/x-wiki k8g24o2lm5bmh4v1uhxbun9kns2q1m7 2444998 2444983 2026-08-06T14:51:47Z SFrench-WMF 37362 /* Usage in production */ Add a couple of clarifications on failure modes 2444998 wikitext text/x-wiki jc8n9y531d3obdwzh27dnlxg7m42yca 2445031 2444998 2026-08-06T17:32:17Z SFrench-WMF 37362 /* Reimage cluster */ Refresh from T428495 2445031 wikitext text/x-wiki 59gici4l7ij3iq7pdv6dgu7tj69va2k 2445035 2445031 2026-08-06T17:59:38Z SFrench-WMF 37362 /* Reimage */ Fix typos, add clarifications 2445035 wikitext text/x-wiki 76mykvtxuu3vkjscbq6bafirt4cbiam 0 449160 2445084 2444841 2026-08-07T00:01:35Z Dexbot 30554 Bot: Updating the report 2445084 wikitext text/x-wiki leuipfwem2yczs3g688jfavd3ddts9g 0 449339 2444957 2429023 2026-08-06T13:25:01Z JMeybohm (WMF) 16709 Add link to istio-cni packaging instructions 2444957 wikitext text/x-wiki 9250a9gerf74t1m7xizqdqzmbun3xfm 0 450294 2445127 2398545 2026-08-07T09:08:49Z BRouberol-WMF 39141 /* List of instances */ 2445127 wikitext text/x-wiki bbm6z5qno91v845w6b45tw5fth5iuxr 2445128 2445127 2026-08-07T09:11:59Z BRouberol-WMF 39141 2445128 wikitext text/x-wiki 047bgrd0ezohv543jrvxg7jndhw620z 0 453707 2445132 2443891 2026-08-07T10:00:24Z MArostegui (WMF) 8620 /* Move the host in the right groups with the right role */ 2445132 wikitext text/x-wiki 28fizxr9pipb1c0wmsyqcw2qnbn6t9e 2445144 2445132 2026-08-07T11:19:14Z MArostegui (WMF) 8620 /* Cloning */ 2445144 wikitext text/x-wiki scwhxbchnqxqttknp5ybq7tytb3rxnt 116 453906 2445100 2443201 2026-08-07T06:36:12Z Gitlabaccountapprovalbot 37332 brianbybyby was rejected. 2445100 wikitext text/x-wiki 77b4kg8o4fkhwbuhri7lb6f5r1e4run 0 454010 2445042 2420531 2026-08-06T18:28:31Z ADancy (WMF) 17694 /* Administration */ Added monitoring section 2445042 wikitext text/x-wiki 0muzc1e2dks5o7oy173mpk0pcxhfgoh 2445043 2445042 2026-08-06T18:29:57Z ADancy (WMF) 17694 /* Testing */ added another actions that should trigger a phab mention 2445043 wikitext text/x-wiki rdxtbfrd1hwpcsv6k12ovkrtdeaqry7 2445065 2445043 2026-08-06T20:28:09Z ADancy (WMF) 17694 /* Testing */ more examples of actions that result in a phab notification 2445065 wikitext text/x-wiki dpnsgb95843vd27gqb95y17d9lpt9jv 2445082 2445065 2026-08-06T21:40:55Z BryanDavis 1604 /* Testing */ link to test task 2445082 wikitext text/x-wiki 3cwrq8zo5svlhjoxcwatu1b9c391uyy 498 455134 2445078 2393703 2026-08-06T21:27:42Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 64 cores, 1600 gigabytes, 4 instances, 131072 ram, 8 volumes 2445078 wikitext text/x-wiki kmk7ecsh5knjxpoxeyre3jty9ovbxg0 2445079 2445078 2026-08-06T21:27:49Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 64 cores, 1600 gigabytes, 4 instances, 131072 ram, 8 volumes 2445079 wikitext text/x-wiki q6lpdv78bzrq2ky8z6yptk1uz7qe6ka 0 459044 2445064 2444599 2026-08-06T20:27:49Z BKing (WMF) 29017 /* Cluster Quorum Loss Recovery Procedure */ 2445064 wikitext text/x-wiki gl44w795vm6j7rihk8dpvqt0kw1fr8u 2445066 2445064 2026-08-06T20:30:27Z BKing (WMF) 29017 /* Cluster Quorum Loss Recovery Procedure */ 2445066 wikitext text/x-wiki em643169m59nu6eehcmz3xna4n9b1ox 0 459699 2444992 2444844 2026-08-06T14:44:45Z Stashbot 7414 larssandergreen: civicrm upgraded from 56c8de1f to 78e2ecac 2444992 wikitext text/x-wiki qaf93pwr17ecrwdll4r081bmmg9dfz9 2 460394 2445039 2441958 2026-08-06T18:19:47Z Milimetric 612 2445039 wikitext text/x-wiki pwohm7wf96g48ct952x2c32b0ven23c 1 460409 2444946 2444753 2026-08-06T13:13:09Z BTracy-WMF 46548 /* status of label service queries */ Reply 2444946 wikitext text/x-wiki i7c4jgyphpmu121g4h8cjzg16wvj6mc 2444973 2444946 2026-08-06T14:09:44Z Peter F. Patel-Schneider 42591 /* status of label service queries */ Reply 2444973 wikitext text/x-wiki s8hxsyio7rr1g0ap85iarytmhftirm3 0 460470 2445087 2444742 2026-08-07T02:11:20Z DLynch (WMF) 6438 2445087 wikitext text/x-wiki eh2r02az9w7vef1vho9ay89gsth7th2 2 460515 2445013 2444547 2026-08-06T15:49:05Z Ottomata 88 /* 1. Write a lambda */ 2445013 wikitext text/x-wiki tbg26em7vhk09wr4o3md0zb9aom9mdc 0 460522 2445032 2444733 2026-08-06T17:42:52Z DChan (WMF) 54599 Add repo URL 2445032 wikitext text/x-wiki hjy6jcp3ihgorad3134lmohqdnncnbm 1 460533 2444965 2026-08-06T13:50:30Z Diegodlh 29411 /* What it can and can't do */ new section 2444965 wikitext text/x-wiki ny9kp0n6lmdvnnppsv5rs3t6ch6coqo 2 460534 2445012 2026-08-06T15:47:20Z Taavi-WMF 41365 init 2445012 wikitext text/x-wiki efzo142w8dzfcehrel0oth0195oznx7