Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.3 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 328 2418295 2401850 2026-05-21T17:28:59Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm util-abogott (cluster codfw1dev, project testlabs) 2418295 wikitext text/x-wiki 61ermmju5gj9xynerw8lrtodcd6tbnt 2418296 2418295 2026-05-21T17:29:02Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm util-abogott (cluster codfw1dev, project testlabs) 2418296 wikitext text/x-wiki 3mbo7azzle3ikyhz4mhhdpylcajxzzf 498 3086 2417919 2417771 2026-05-21T12:46:59Z Stashbot 7414 taavi: renew prometheus certificate 2417919 wikitext text/x-wiki 45oxy6pg9gb6ydm5bqcl92g7leriyrq 0 4108 2417903 2417842 2026-05-21T12:37:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1290743]] to Thursday, May 21 UTC afternoon backport window 2417903 wikitext text/x-wiki evz8lpdix580yan2v8enrix19xi23r9 2417906 2417903 2026-05-21T12:38:22Z ScheduleDeploymentBot 37566 Add [[gerrit:1289347]] to Thursday, May 21 UTC afternoon backport window 2417906 wikitext text/x-wiki mv3pl6dc8qsjw9wiudkci5edw8hh5h1 2417907 2417906 2026-05-21T12:39:02Z ScheduleDeploymentBot 37566 Add [[gerrit:1289342]] to Thursday, May 21 UTC afternoon backport window 2417907 wikitext text/x-wiki pwizwt9y8g3v1g4mgz8e2vn9rtlz8nj 2418060 2417907 2026-05-21T14:01:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1290781]] to Thursday, May 21 UTC late backport window 2418060 wikitext text/x-wiki sehxjgujhjzhjj451niicyeyyzea1af 2418201 2418060 2026-05-21T15:49:27Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-05-21}} */ UTC-late infra window 2418201 wikitext text/x-wiki 3d1vdpklfpogpwyyz3qnszogy1infn4 2418357 2418201 2026-05-21T21:06:51Z ScheduleDeploymentBot 37566 Add [[gerrit:1290924]] to Tuesday, May 26 UTC afternoon backport window 2418357 wikitext text/x-wiki ozswalsgttjawlvfxrwbweb3am1676n 2418358 2418357 2026-05-21T21:07:02Z ScheduleDeploymentBot 37566 Add [[gerrit:1290926]] to Tuesday, May 26 UTC afternoon backport window 2418358 wikitext text/x-wiki n357r313bgci4lnb2ktsl7w89aq5o6v 2418417 2418358 2026-05-22T07:50:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1289898]] to Thursday, May 28 UTC morning backport window 2418417 wikitext text/x-wiki bfs1ovw8zsyintk0x2t3r1iqrqvbbqu 2418499 2418417 2026-05-22T11:08:29Z ScheduleDeploymentBot 37566 Add [[gerrit:1291951]] to Tuesday, May 26 UTC morning backport window 2418499 wikitext text/x-wiki 8oli1mebuhwmdtkhyhd4ul8xp0tapui 0 7919 2417845 2417843 2026-05-21T12:00:52Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet 2417845 wikitext text/x-wiki sh4wg8au74yxahsura67ii5fvv2b1yt 2417846 2417845 2026-05-21T12:00:54Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1017.eqiad.wmnet with OS trixie 2417846 wikitext text/x-wiki a5xbfrpydhsugt8tgvfydpinmnibmau 2417847 2417846 2026-05-21T12:01:38Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet 2417847 wikitext text/x-wiki jjn4owk5y8obhu5k7jku256ktrywcwh 2417848 2417847 2026-05-21T12:01:40Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp601[3-4].drmrs.wmnet} and A:cp 2417848 wikitext text/x-wiki 63aajyg12ed6pzma9wf9vdas5wd931i 2417849 2417848 2026-05-21T12:02:15Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp600[7-8].drmrs.wmnet} and A:cp 2417849 wikitext text/x-wiki fov4uc0lzs57or4cxoxz8ztawu86476 2417850 2417849 2026-05-21T12:02:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet 2417850 wikitext text/x-wiki rnusis49ego88abwa0tt2c83a12dxf3 2417851 2417850 2026-05-21T12:03:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet 2417851 wikitext text/x-wiki f8kchwrtn0yauvw857k934ffsg9fnw3 2417852 2417851 2026-05-21T12:07:37Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet 2417852 wikitext text/x-wiki mh6yee3h2bxnrykv2dkvzudlowboy3b 2417853 2417852 2026-05-21T12:07:39Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet 2417853 wikitext text/x-wiki 1pvvncxjawqgsqdrml6ddvj1mnga554 2417854 2417853 2026-05-21T12:07:42Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet 2417854 wikitext text/x-wiki pm71g2foq8zt9v7w2jtq7h1zhf9s0y0 2417855 2417854 2026-05-21T12:08:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92764 and previous config saved to /var/cache/conftool/dbconfig/20260521-120824-fceratto.json 2417855 wikitext text/x-wiki lrac8h38kyw9rmxfxi6rn4ugvg121o7 2417856 2417855 2026-05-21T12:10:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet 2417856 wikitext text/x-wiki 24tb5ts8q4m77cmffz9ug2m5217dymp 2417857 2417856 2026-05-21T12:12:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet 2417857 wikitext text/x-wiki b78njncppdljdsb87x5qd13qbnsh6aa 2417858 2417857 2026-05-21T12:13:02Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet 2417858 wikitext text/x-wiki 7zp6shomj7antse4lje2li7y7lzfre9 2417859 2417858 2026-05-21T12:14:18Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet 2417859 wikitext text/x-wiki bgcnkofsksi1dxp4woldbatc03soo1x 2417860 2417859 2026-05-21T12:15:22Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage 2417860 wikitext text/x-wiki h2h9ug8ix7e8dvj4kye4f8mlgxfk2z7 2417861 2417860 2026-05-21T12:16:42Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet 2417861 wikitext text/x-wiki 8g5mzfkrowsvpgg0p6a2cdkhn3cprdi 2417862 2417861 2026-05-21T12:17:52Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet 2417862 wikitext text/x-wiki 49512wf25ufqrkpbh3lrmanofw7jitw 2417863 2417862 2026-05-21T12:18:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92765 and previous config saved to /var/cache/conftool/dbconfig/20260521-121832-fceratto.json 2417863 wikitext text/x-wiki 0asq69pyhg1qlopdddl4xsspuxybgeg 2417864 2417863 2026-05-21T12:18:59Z Stashbot 7414 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-staging-codfw: maintenance 2417864 wikitext text/x-wiki ef67c4fngjib2fey630jip1ucn31uce 2417865 2417864 2026-05-21T12:19:01Z Stashbot 7414 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-staging-codfw: maintenance 2417865 wikitext text/x-wiki icdyma1u06bhwoii9piy8u263yvlll3 2417866 2417865 2026-05-21T12:19:14Z Stashbot 7414 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-staging-codfw: maintenance 2417866 wikitext text/x-wiki dn606iusulry9536qfnqp8zmfj7j3t7 2417867 2417866 2026-05-21T12:19:17Z Stashbot 7414 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-staging-codfw: maintenance 2417867 wikitext text/x-wiki ldedofkfhb9ijrgtdgneukosubnaezf 2417868 2417867 2026-05-21T12:19:31Z Stashbot 7414 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-serve-codfw: maintenance 2417868 wikitext text/x-wiki dn7prlndov1jt8nf4q8xpbc1gzygn36 2417869 2417868 2026-05-21T12:19:41Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet 2417869 wikitext text/x-wiki 93gkdadkpgqwr5iixu81h3vyzs4qj9m 2417870 2417869 2026-05-21T12:19:45Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage 2417870 wikitext text/x-wiki 3pkkexhjrvu1vltbfkoorzoqd5zedsp 2417871 2417870 2026-05-21T12:20:12Z Stashbot 7414 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-serve-codfw: maintenance 2417871 wikitext text/x-wiki 48c7rh41z8cmuc9e49iaxb0vd50u0uz 2417872 2417871 2026-05-21T12:20:20Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet 2417872 wikitext text/x-wiki 773y1byxu5icb4ejmxhgwt37lnllkm4 2417873 2417872 2026-05-21T12:21:30Z Stashbot 7414 moritzm: installing nginx security updates 2417873 wikitext text/x-wiki 3l3naedfhe5y86jo0jue3atykradc42 2417874 2417873 2026-05-21T12:21:36Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply 2417874 wikitext text/x-wiki 4x6o168cbkl3za0ks30l825rjquzo83 2417875 2417874 2026-05-21T12:21:59Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply 2417875 wikitext text/x-wiki o7zarpfa4f2793kp27p7jtb2w7pfc3c 2417876 2417875 2026-05-21T12:22:40Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet 2417876 wikitext text/x-wiki 8ybfap6w6k9cbo2tr127h293151v2pr 2417877 2417876 2026-05-21T12:23:51Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet 2417877 wikitext text/x-wiki jf8mqxanix1cdp0ljkov9d7vge24pwe 2417878 2417877 2026-05-21T12:23:53Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet 2417878 wikitext text/x-wiki 5155k94ueudx4qn1h9edfdj1jf9dzsp 2417879 2417878 2026-05-21T12:23:56Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker 2417879 wikitext text/x-wiki 3cp57p36xn62ufv3rd265hliw6ey8ro 2417880 2417879 2026-05-21T12:26:47Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417880 wikitext text/x-wiki 2lovf0gx3e37w7apprw1e3ruknqpq1a 2417881 2417880 2026-05-21T12:27:26Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply 2417881 wikitext text/x-wiki q2vmrpgi8d8ws42v8boj182jhysjvni 2417882 2417881 2026-05-21T12:28:00Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply 2417882 wikitext text/x-wiki botnk0jbnbd45ymsnfj6gfrhfqypjui 2417883 2417882 2026-05-21T12:28:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 (T426633)', diff saved to https://phabricator.wikimedia.org/P92766 and previous config saved to /var/cache/conftool/dbconfig/20260521-122839-fceratto.json 2417883 wikitext text/x-wiki j8b5k44wij1h6j19ed8dl2ok723mwbw 2417884 2417883 2026-05-21T12:29:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance 2417884 wikitext text/x-wiki a67bigjtiha1zq9q5hkpcsd1zj6k6sh 2417885 2417884 2026-05-21T12:29:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 (T426633)', diff saved to https://phabricator.wikimedia.org/P92767 and previous config saved to /var/cache/conftool/dbconfig/20260521-122905-fceratto.json 2417885 wikitext text/x-wiki fa74n6q7kq5a2n4oi1ct2md78anvuzk 2417886 2417885 2026-05-21T12:29:12Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417886 wikitext text/x-wiki bmjhexvlumllnyfosfe7dxou6cg4y4c 2417887 2417886 2026-05-21T12:30:02Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet 2417887 wikitext text/x-wiki 9rg0taj75r1hy8st3hf349uk6k0nvqa 2417888 2417887 2026-05-21T12:30:12Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply 2417888 wikitext text/x-wiki t7k6h0noebthy2x4sixao9citkwos8z 2417890 2417888 2026-05-21T12:30:45Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply 2417890 wikitext text/x-wiki j71gfshdo94kx7263734og02qtjrz9r 2417891 2417890 2026-05-21T12:32:32Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet 2417891 wikitext text/x-wiki 4n36tdzs0b7nim2j0cu72ecf3ae0i3v 2417892 2417891 2026-05-21T12:34:05Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet 2417892 wikitext text/x-wiki cxhcla76qhywg7hqx55ah66xg5ikccp 2417893 2417892 2026-05-21T12:34:15Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417893 wikitext text/x-wiki 8h20ia3bvpo072ugat4r99og8fq1m5f 2417894 2417893 2026-05-21T12:34:24Z Stashbot 7414 kart_: Updated cxserver to 2026-05-20-034002-production (T388690, T404295, T391703, T426605) 2417894 wikitext text/x-wiki 0ke3z8i2f6fdthrm6kxmlhgelnpxvvd 2417895 2417894 2026-05-21T12:34:46Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet with OS trixie 2417895 wikitext text/x-wiki qqelj9jybkdlscq6x71h3uwty94lv7k 2417896 2417895 2026-05-21T12:35:27Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417896 wikitext text/x-wiki 1307fpxfkt1kvp1ma9xy4bb9ddq0gzd 2417897 2417896 2026-05-21T12:35:33Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1290727|hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] 2417897 wikitext text/x-wiki sv56hweivgdoggthdnf45kpz6vok0gh 2417898 2417897 2026-05-21T12:35:36Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417898 wikitext text/x-wiki oy2r3n05h5qe424wypxee4zl1avylis 2417899 2417898 2026-05-21T12:36:17Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp3066.esams.wmnet} and A:cp 2417899 wikitext text/x-wiki i8dhwq5r8hqd31eqkp4rcwy3zljwsda 2417900 2417899 2026-05-21T12:36:46Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417900 wikitext text/x-wiki g6xpjus8s09cgxkjd3q98yf70ra2n3i 2417901 2417900 2026-05-21T12:37:23Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1290727|hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417901 wikitext text/x-wiki a68w3rvknxac753xvw4i7bqxxi2o6dw 2417902 2417901 2026-05-21T12:37:43Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1018.eqiad.wmnet with OS trixie 2417902 wikitext text/x-wiki oue93b2bu6bfrqk0eh4gywsauhkgklp 2417904 2417902 2026-05-21T12:37:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet 2417904 wikitext text/x-wiki ha1xldpfsd6xu3x2atixlvg10rmfw2d 2417905 2417904 2026-05-21T12:38:03Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet 2417905 wikitext text/x-wiki 7ozhi5pzfqwosrhn6hyzr3du4rc713i 2417908 2417905 2026-05-21T12:39:12Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2417908 wikitext text/x-wiki qoeu7ajg751n28eca8qkkv1jcqu4qgt 2417909 2417908 2026-05-21T12:40:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 (T426633)', diff saved to https://phabricator.wikimedia.org/P92768 and previous config saved to /var/cache/conftool/dbconfig/20260521-124014-fceratto.json 2417909 wikitext text/x-wiki aukpd6jq24wau83h30e3o164x80gnq2 2417910 2417909 2026-05-21T12:42:43Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417910 wikitext text/x-wiki qysp6pxymcie8vp7dwfz8lexzbk03an 2417911 2417910 2026-05-21T12:43:29Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290727|hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] (duration: 07m 54s) 2417911 wikitext text/x-wiki nbv6xbot0x0netzsahc3uv8lj30reyk 2417912 2417911 2026-05-21T12:43:53Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417912 wikitext text/x-wiki n39lv9fno33x3rv0p7s45x3j6my7adq 2417913 2417912 2026-05-21T12:44:19Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417913 wikitext text/x-wiki hjmpaqc5p3otz6kjrjutq12v6cw1o4a 2417914 2417913 2026-05-21T12:45:30Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417914 wikitext text/x-wiki f3z00lz7hfdm8nrhpa2lms1wbmre8hx 2417915 2417914 2026-05-21T12:45:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet 2417915 wikitext text/x-wiki 5e99mulvncgreui4wffb2l0zzma1e7i 2417917 2417915 2026-05-21T12:46:12Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417917 wikitext text/x-wiki o3j2u91z9nyc64khpue99nz9764dk9n 2417918 2417917 2026-05-21T12:46:15Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1039: Repooling 2417918 wikitext text/x-wiki jut78tsrzp1t6gjexmipsumif4bvp7m 2417920 2417918 2026-05-21T12:47:01Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance 2417920 wikitext text/x-wiki gqd4y90rcucus91q7fohj88op2art05 2417921 2417920 2026-05-21T12:47:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 (T426633)', diff saved to https://phabricator.wikimedia.org/P92770 and previous config saved to /var/cache/conftool/dbconfig/20260521-124707-fceratto.json 2417921 wikitext text/x-wiki 4blsz6c9mbu47dbecd2pzd2ors7k59u 2417922 2417921 2026-05-21T12:47:23Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417922 wikitext text/x-wiki d7nkp9ty5969sg6fdfrxspdo6xsu37r 2417923 2417922 2026-05-21T12:48:28Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet 2417923 wikitext text/x-wiki asqh88r4m3zqfd8rxlkzm6r3hl9kerx 2417924 2417923 2026-05-21T12:48:31Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp3066.esams.wmnet} and A:cp 2417924 wikitext text/x-wiki nuk4f1w9zpxcjzew1slpzibev74eov3 2417925 2417924 2026-05-21T12:48:40Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet 2417925 wikitext text/x-wiki plzwsjlhetv24ylaljywds7yc8opc2x 2417926 2417925 2026-05-21T12:49:16Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad 2417926 wikitext text/x-wiki na9gwy0puuobj0ncmr3fmfjmkvlutgw 2417927 2417926 2026-05-21T12:49:20Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet 2417927 wikitext text/x-wiki 4mz6wjjjuxxxfd4ygkjxbfmzb9rg6rj 2417928 2417927 2026-05-21T12:51:53Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417928 wikitext text/x-wiki etd7de3rw78t3t7j7kbwuy3p6zedjt0 2417929 2417928 2026-05-21T12:52:30Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage 2417929 wikitext text/x-wiki tmnx2pngml9iy4x7ljm6uik1l8h7x0v 2417930 2417929 2026-05-21T12:53:04Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417930 wikitext text/x-wiki 840il4lf992rcqgcy8fb3p8w3ios9cr 2417931 2417930 2026-05-21T12:54:15Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet 2417931 wikitext text/x-wiki qh8ne9ygxgivjiwc98md1ud7pslw8y2 2417932 2417931 2026-05-21T12:54:18Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp600[7-8].drmrs.wmnet} and A:cp 2417932 wikitext text/x-wiki 63hiakwqqe1b63zgyil1bze0ni65rav 2417933 2417932 2026-05-21T12:54:23Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet 2417933 wikitext text/x-wiki or9ksl9v7z2m5alw992q3tqu5i9ie9x 2417934 2417933 2026-05-21T12:54:29Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp3074.esams.wmnet} and A:cp 2417934 wikitext text/x-wiki hoysbr0e1mjnvqfxqjvv7yklbhtsw7r 2417935 2417934 2026-05-21T12:54:43Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet 2417935 wikitext text/x-wiki kqos1dkbz7n0xncj5r6o466k3tivrdc 2417936 2417935 2026-05-21T12:55:08Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet 2417936 wikitext text/x-wiki bdvelk5t11sk4cjdzxw2lxg1ra9oth6 2417937 2417936 2026-05-21T12:56:05Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417937 wikitext text/x-wiki p3wqf4rc4dcv26vz6uqsncz7paoj7k5 2417938 2417937 2026-05-21T12:56:14Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 T426936 2417938 wikitext text/x-wiki qwakbep0kc1wl2vbcoc24d1lf9rr0ae 2417939 2417938 2026-05-21T12:56:49Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2162 with weight 0 T426936', diff saved to https://phabricator.wikimedia.org/P92771 and previous config saved to /var/cache/conftool/dbconfig/20260521-125645-cwilliams.json 2417939 wikitext text/x-wiki beek7ey3vccwbgihlmma9y9crnrth74 2417941 2417939 2026-05-21T12:57:53Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417941 wikitext text/x-wiki ltaxavtz0jfvu5sevoa8hufw1pgatr2 2417942 2417941 2026-05-21T12:58:02Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2417942 wikitext text/x-wiki oub05aoapjgh91p56yudoetb8d1ldd6 2417943 2417942 2026-05-21T12:59:05Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet 2417943 wikitext text/x-wiki jhwhec81u067rmrbzu6ywjhfh0xkdkz 2417944 2417943 2026-05-21T12:59:07Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet 2417944 wikitext text/x-wiki t4y37z2vibgkin4byc4e1f7zz6jyc9u 2417945 2417944 2026-05-21T12:59:11Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage 2417945 wikitext text/x-wiki o9oqbyjdphpy0x0ezae7gy404rxhdy6 2417946 2417945 2026-05-21T12:59:13Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet 2417946 wikitext text/x-wiki j4ywql7ow3xvncbtfzwyb3j3vlih7zd 2417947 2417946 2026-05-21T13:00:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 (T426633)', diff saved to https://phabricator.wikimedia.org/P92772 and previous config saved to /var/cache/conftool/dbconfig/20260521-130018-fceratto.json 2417947 wikitext text/x-wiki 7p7o81v2h58tu4e5xskiui1uo51g9a6 2417948 2417947 2026-05-21T13:00:25Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2417948 wikitext text/x-wiki kp5mwkje40rkwt8qdc312pw5myid49o 2417949 2417948 2026-05-21T13:03:58Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet 2417949 wikitext text/x-wiki 4d724z1c18a1z0ijwmfxnf1ti5q1vie 2417950 2417949 2026-05-21T13:04:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet 2417950 wikitext text/x-wiki r52a6j2fg3wyc3lp2962mngys506rgh 2417951 2417950 2026-05-21T13:04:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet 2417951 wikitext text/x-wiki 1xern4rdp2psnq63mvsrknh0hraj5us 2417952 2417951 2026-05-21T13:04:43Z Stashbot 7414 cezmunsta: Starting x3 codfw failover from db2241 to db2162 - T426936 2417952 wikitext text/x-wiki ar4dxlulwqx6v5ba0atp96sf2h97ylr 2417953 2417952 2026-05-21T13:05:01Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2417953 wikitext text/x-wiki als1i5bzu770rcbay69lidfig8kyazu 2417954 2417953 2026-05-21T13:06:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2162 to x3 primary T426936', diff saved to https://phabricator.wikimedia.org/P92774 and previous config saved to /var/cache/conftool/dbconfig/20260521-130609-cwilliams.json 2417954 wikitext text/x-wiki 1w6rn4g2m6po004tmxp1i28heckmsd2 2417955 2417954 2026-05-21T13:06:15Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet 2417955 wikitext text/x-wiki hubihp87yj6jks6663cm9hj09vnviow 2417956 2417955 2026-05-21T13:06:17Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp3074.esams.wmnet} and A:cp 2417956 wikitext text/x-wiki fzito9ufmb2ffiv8ighuvd8mv6zlww6 2417957 2417956 2026-05-21T13:06:38Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp601[5-6].drmrs.wmnet} and A:cp 2417957 wikitext text/x-wiki aommrgzrsvrdbl3u2ilehyqsr0ip0kb 2417958 2417957 2026-05-21T13:07:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet 2417958 wikitext text/x-wiki cmjd15gms960qh9ew3rytwtubi4c1cv 2417959 2417958 2026-05-21T13:08:17Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290088|Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032|Enable 'flood' user group at en.wikiversity (T426882)]] 2417959 wikitext text/x-wiki 0u34vkybjxzlnd4a9giacf4b3wvkkz8 2417960 2417959 2026-05-21T13:09:17Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . 2417960 wikitext text/x-wiki m9mv8f368xgtwdogpqbdaj5lgvgg8lm 2417961 2417960 2026-05-21T13:09:22Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2417961 wikitext text/x-wiki jb6o4tl2t9un7ksm5453da9goiquvxz 2417962 2417961 2026-05-21T13:09:27Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . 2417962 wikitext text/x-wiki h7xxdrvymp8yweehjhc74w9fdnp12vp 2417963 2417962 2026-05-21T13:09:32Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2417963 wikitext text/x-wiki s3xeu021jcl8h7ot79nzqy9eriooq3u 2417964 2417963 2026-05-21T13:09:37Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2417964 wikitext text/x-wiki 2z6gpstfwoxd5o7nqyvp7b2omf0m0cs 2417965 2417964 2026-05-21T13:09:44Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2417965 wikitext text/x-wiki mjedqfaf9xq1i1g268ifhdwm5tmyen3 2417966 2417965 2026-05-21T13:09:49Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2417966 wikitext text/x-wiki ntxven09k6zsyuctih98ndgjc1eedp2 2417967 2417966 2026-05-21T13:09:58Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2417967 wikitext text/x-wiki swwmyow683rm0rkm9985p30t4n2m1pe 2417968 2417967 2026-05-21T13:10:03Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1290088|Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032|Enable 'flood' user group at en.wikiversity (T426882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2417968 wikitext text/x-wiki kalqr83f46a7hpwfspvvdhc93953bcr 2417969 2417968 2026-05-21T13:10:05Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . 2417969 wikitext text/x-wiki d40mv8u1s3x0awnri3sf0q56dub5woq 2417970 2417969 2026-05-21T13:10:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet 2417970 wikitext text/x-wiki 5zm7uzjogselrj5t4k8gy826di18bc1 2417971 2417970 2026-05-21T13:10:09Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2417971 wikitext text/x-wiki 1rzycwlyzsy20guq2ii2ug1cscx7r35 2417972 2417971 2026-05-21T13:10:13Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . 2417972 wikitext text/x-wiki 3n1uhyh31yvces3rbe3zrjky7179bt7 2417973 2417972 2026-05-21T13:10:18Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . 2417973 wikitext text/x-wiki 9lq006v2yez5swummsvp2mrrw8d4m36 2417974 2417973 2026-05-21T13:10:23Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2417974 wikitext text/x-wiki ez4l1yv3w65vb0v8nznf4l6424xj148 2417975 2417974 2026-05-21T13:10:29Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2241 T426936', diff saved to https://phabricator.wikimedia.org/P92775 and previous config saved to /var/cache/conftool/dbconfig/20260521-131025-cwilliams.json 2417975 wikitext text/x-wiki d0oph8mdg1fmc4c3sowdewdhfqy8cqm 2417976 2417975 2026-05-21T13:10:31Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2417976 wikitext text/x-wiki 7iec592jiwf29fy5uyexzbh357n678f 2417977 2417976 2026-05-21T13:10:33Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2417977 wikitext text/x-wiki ctibepc51m7ifbdpydf4wrbdn07lux9 2417978 2417977 2026-05-21T13:10:35Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet 2417978 wikitext text/x-wiki 4pp4ktn6nvxl1ncu1yzwfi9dcu3axlt 2417979 2417978 2026-05-21T13:10:38Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet 2417979 wikitext text/x-wiki 3h5jetc7z0egm1me80ilci09gz47ml6 2417980 2417979 2026-05-21T13:10:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92776 and previous config saved to /var/cache/conftool/dbconfig/20260521-131033-fceratto.json 2417980 wikitext text/x-wiki 4r999yw5jb7qylg8q4e5jltoss84ydx 2417981 2417980 2026-05-21T13:10:42Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2417981 wikitext text/x-wiki bbie4cidi63i2ycbnj8hauf6rt1trmv 2417982 2417981 2026-05-21T13:10:45Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet 2417982 wikitext text/x-wiki r9ldleeszolj2x7jpecvoaj2kzmomp8 2417983 2417982 2026-05-21T13:10:47Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2417983 wikitext text/x-wiki d98e6tni320ytnpurwatjwn53x4r18q 2417984 2417983 2026-05-21T13:11:25Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:restbase 2417984 wikitext text/x-wiki iz1m8gq9wlg8akq1uhxl85tqjcas4th 2417985 2417984 2026-05-21T13:14:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet 2417985 wikitext text/x-wiki e39qwn6kn6kr6vx8qyh46wx3027tpzk 2417986 2417985 2026-05-21T13:15:42Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet 2417986 wikitext text/x-wiki 1pzh3hpa11t1o4k06yznxt2tqftdoz5 2417987 2417986 2026-05-21T13:15:57Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment 2417987 wikitext text/x-wiki k8fg2tcc23fcke0pkrztv2gwy1d6ww6 2417988 2417987 2026-05-21T13:16:24Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet 2417988 wikitext text/x-wiki pzs3oivmk5sdrhrqq84w8d0ygl4vwjw 2417989 2417988 2026-05-21T13:16:32Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet 2417989 wikitext text/x-wiki owuvwt0ppmiusri7habhqp5tkk6zne0 2417990 2417989 2026-05-21T13:16:42Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1039: Repooling 2417990 wikitext text/x-wiki mg3j18is1kk27wvyb7v838g7b6uka7d 2417991 2417990 2026-05-21T13:16:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet 2417991 wikitext text/x-wiki b4gu2fymblslbuo1usa55uw9p3ez466 2417992 2417991 2026-05-21T13:17:40Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1018.eqiad.wmnet with OS trixie 2417992 wikitext text/x-wiki exb2i657vyhk6bq1zuamwuaww9tvqul 2417993 2417992 2026-05-21T13:18:43Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet 2417993 wikitext text/x-wiki 5tsl5xuja6fwbw4g9ex41feor8bb0sp 2417994 2417993 2026-05-21T13:20:12Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290088|Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032|Enable 'flood' user group at en.wikiversity (T426882)]] (duration: 11m 55s) 2417994 wikitext text/x-wiki 0mtk8mnnqb45v2c47fmb4xjtl2xeh2x 2417995 2417994 2026-05-21T13:20:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet 2417995 wikitext text/x-wiki pqszd52k4s5d4s9vwimorxlvr0lv76j 2417996 2417995 2026-05-21T13:20:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92778 and previous config saved to /var/cache/conftool/dbconfig/20260521-132041-fceratto.json 2417996 wikitext text/x-wiki avuwpmp6uztv7nhuwgwyqa9eslhaeqj 2417997 2417996 2026-05-21T13:22:12Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet 2417997 wikitext text/x-wiki ijanasyfhnxit0msdt73ysjt2f4fevq 2417998 2417997 2026-05-21T13:22:15Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet 2417998 wikitext text/x-wiki k13a3ty0468djm401wkezsgmyqi71pt 2417999 2417998 2026-05-21T13:22:19Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet 2417999 wikitext text/x-wiki tdo21d1yarn4bm7y8thr3xntrok8dvx 2418000 2417999 2026-05-21T13:22:22Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290014|Enable AG on phase 2 wikis (T426871)]] 2418000 wikitext text/x-wiki g7yssopf6un5e27kxp7pi060j6nlxoz 2418001 2418000 2026-05-21T13:23:18Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2418001 wikitext text/x-wiki 2yyc30xl2z1v7fakd89nbnjjzhjgq9j 2418002 2418001 2026-05-21T13:23:29Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Upgrading db2241.codfw.wmnet 2418002 wikitext text/x-wiki t3kgdhq10aisaardl8wcvxplkpb2txe 2418003 2418002 2026-05-21T13:23:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Upgrading db2241.codfw.wmnet 2418003 wikitext text/x-wiki a1hmcpsbxl3htgp2i2xun0cp9tkks9e 2418004 2418003 2026-05-21T13:24:07Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290014|Enable AG on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2418004 wikitext text/x-wiki azora0tz8xwrhx6rusdhjbrm0bv2obm 2418005 2418004 2026-05-21T13:25:04Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418005 wikitext text/x-wiki 4lrhxa7i9rhi13jabjcucps2gr5mli9 2418006 2418005 2026-05-21T13:25:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2241.codfw.wmnet with OS trixie 2418006 wikitext text/x-wiki ezcqmeknfxyvx1g4ww7w9npqp5h9675 2418007 2418006 2026-05-21T13:25:14Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418007 wikitext text/x-wiki 2qjwzddt5rw9vst9i9znfdl52ycbyib 2418008 2418007 2026-05-21T13:26:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet 2418008 wikitext text/x-wiki chwg2u5jk4yzgzcxtsfij4aeykyc0mc 2418009 2418008 2026-05-21T13:26:49Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet 2418009 wikitext text/x-wiki sog7yvyz0z1vxmzj15oj762sdjbo6zx 2418010 2418009 2026-05-21T13:27:14Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2218: repool after maintenance 2418010 wikitext text/x-wiki d6rr5wgahwr9q11e4gprvd508kl33dj 2418011 2418010 2026-05-21T13:27:20Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2418011 wikitext text/x-wiki 85dhdtowdz446xmmqxorkt17dt7ccpm 2418012 2418011 2026-05-21T13:27:23Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet 2418012 wikitext text/x-wiki 85u9wu0uzmcf5oj8i0xiejj9zesvkrh 2418013 2418012 2026-05-21T13:28:28Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet 2418013 wikitext text/x-wiki okcld0y46xplp8qktd2kfzeuf5zhkfj 2418016 2418013 2026-05-21T13:29:29Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet 2418016 wikitext text/x-wiki bi6y4oe6nksjz7v6fxlnq1iew0cbc1b 2418017 2418016 2026-05-21T13:30:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 (T426633)', diff saved to https://phabricator.wikimedia.org/P92780 and previous config saved to /var/cache/conftool/dbconfig/20260521-133048-fceratto.json 2418017 wikitext text/x-wiki 5j2nkpz0bae6rywzjxv8sk6w9yud2nd 2418018 2418017 2026-05-21T13:31:10Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance 2418018 wikitext text/x-wiki igvqzuxmwny5dbwlmbjhdek4iif4hwp 2418019 2418018 2026-05-21T13:31:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 (T426633)', diff saved to https://phabricator.wikimedia.org/P92781 and previous config saved to /var/cache/conftool/dbconfig/20260521-133116-fceratto.json 2418019 wikitext text/x-wiki o5atuu58rc6d8itt6okg9p9ztybh8r6 2418020 2418019 2026-05-21T13:31:32Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290014|Enable AG on phase 2 wikis (T426871)]] (duration: 09m 11s) 2418020 wikitext text/x-wiki 42ob83b69ez5shbfv9twv5l911lwj11 2418021 2418020 2026-05-21T13:32:50Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet 2418021 wikitext text/x-wiki c0pln0lc0s8pcirvtnjxna2jr0bpv3v 2418022 2418021 2026-05-21T13:32:53Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet 2418022 wikitext text/x-wiki 53zt9pbm8qibj36vlhar0o6y1ngx7dv 2418023 2418022 2026-05-21T13:32:57Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet 2418023 wikitext text/x-wiki kzjxhmvgbegysjzg0k8uoov49ic8bau 2418024 2418023 2026-05-21T13:35:02Z Stashbot 7414 dbrant@deploy1003: Started scap sync-world: Backport for [[gerrit:1290035|docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] 2418024 wikitext text/x-wiki mzca2qlli8lagiqitoxkle8xfv9v3rp 2418025 2418024 2026-05-21T13:36:17Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet 2418025 wikitext text/x-wiki oj195f54dg0ii7gbbk4l9mc7sj9kf4t 2418026 2418025 2026-05-21T13:36:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet 2418026 wikitext text/x-wiki 5a44ozdy4gpfmtew08slnnhjgnwtggk 2418027 2418026 2026-05-21T13:36:51Z Stashbot 7414 dbrant@deploy1003: dbrant: Backport for [[gerrit:1290035|docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2418027 wikitext text/x-wiki ig2scy8izohym6kf1gmba0e14m0ic7g 2418028 2418027 2026-05-21T13:37:40Z Stashbot 7414 dbrant@deploy1003: dbrant: Continuing with deployment 2418028 wikitext text/x-wiki ia112yxvcyqwtdik0x1btjw9yotq23y 2418029 2418028 2026-05-21T13:37:57Z Stashbot 7414 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-serve-codfw: maintenance 2418029 wikitext text/x-wiki bd3a1n85thbuzlzsb2uqo8syvrctst0 2418030 2418029 2026-05-21T13:38:00Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet 2418030 wikitext text/x-wiki f1wkkl9kpk8h0n72uvhltr3926oulfp 2418031 2418030 2026-05-21T13:38:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 (T426633)', diff saved to https://phabricator.wikimedia.org/P92782 and previous config saved to /var/cache/conftool/dbconfig/20260521-133815-fceratto.json 2418031 wikitext text/x-wiki m5euvwnbmwe2km4zse6uss0oydn1oop 2418032 2418031 2026-05-21T13:38:44Z Stashbot 7414 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-serve-codfw: maintenance 2418032 wikitext text/x-wiki 0mlskrgtwtt41tkg2gch1l0l7nz9rah 2418033 2418032 2026-05-21T13:39:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet 2418033 wikitext text/x-wiki sph3vhlopopeonus4684c7okvm2tl33 2418034 2418033 2026-05-21T13:40:29Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2241.codfw.wmnet with reason: host reimage 2418034 wikitext text/x-wiki n6d6c6gyc9nx813cbqqjhq5jakjs6cb 2418036 2418034 2026-05-21T13:41:28Z Stashbot 7414 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply 2418036 wikitext text/x-wiki fyvh9wcpx9f7h1t9ypb7vl2xgcoazn1 2418037 2418036 2026-05-21T13:41:55Z Stashbot 7414 dbrant@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290035|docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] (duration: 06m 52s) 2418037 wikitext text/x-wiki 5i51zd42rejjmx3nsu5702p5u8suifx 2418038 2418037 2026-05-21T13:42:56Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet 2418038 wikitext text/x-wiki jr9xd52ou2v4f48nvingtlgq2do5bkh 2418039 2418038 2026-05-21T13:42:58Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet 2418039 wikitext text/x-wiki 6r0y6bhwcyl380ql7q7nvgbfiycl55r 2418040 2418039 2026-05-21T13:43:03Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet 2418040 wikitext text/x-wiki 1cdxzpn0chsfml5r190v2fgfawdwqvn 2418041 2418040 2026-05-21T13:43:17Z Stashbot 7414 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply 2418041 wikitext text/x-wiki 4jtm4xthw0j3xkudnuyx5gns3z5q46p 2418042 2418041 2026-05-21T13:43:51Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290743|composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347|Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342|fix: simplify to show only one icon type for password reveal (T419413)]] 2418042 wikitext text/x-wiki fvhldqf9rckw18knk6muy5omg6mbudm 2418043 2418042 2026-05-21T13:44:23Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply 2418043 wikitext text/x-wiki 6x4vzrnmf8tbtm27223wqzag1y74jv9 2418044 2418043 2026-05-21T13:44:30Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2241.codfw.wmnet with reason: host reimage 2418044 wikitext text/x-wiki 18a4z88d5vwnc249lqenigjbc6p930n 2418045 2418044 2026-05-21T13:45:34Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1290743|composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347|Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342|fix: simplify to show only one icon type for password reveal (T419413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes 2418045 wikitext text/x-wiki 2i5riuw27fhvc37f2q1ogit8nicmh0d 2418046 2418045 2026-05-21T13:45:45Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply 2418046 wikitext text/x-wiki cikn8wldhtcstfj48xz146ckf2au7vo 2418047 2418046 2026-05-21T13:46:59Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment 2418047 wikitext text/x-wiki eyvjxigw0a79ypm4bp4tw8fpiavqz1w 2418048 2418047 2026-05-21T13:47:54Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply 2418048 wikitext text/x-wiki owtd5p25gepgqjug4t7gipa1wgvty39 2418049 2418048 2026-05-21T13:48:07Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet 2418049 wikitext text/x-wiki g4mdc0yktzpmwzi5nzuxi90n1vw6zzm 2418050 2418049 2026-05-21T13:48:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92784 and previous config saved to /var/cache/conftool/dbconfig/20260521-134822-fceratto.json 2418050 wikitext text/x-wiki 59tklippcwfshjj4dtd9yzuitautd8r 2418051 2418050 2026-05-21T13:51:11Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290743|composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347|Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342|fix: simplify to show only one icon type for password reveal (T419413)]] (duration: 07m 20s) 2418051 wikitext text/x-wiki lzy8dc6p293gzi95qn1wxksvn22bb2v 2418052 2418051 2026-05-21T13:51:37Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2418052 wikitext text/x-wiki as20f76wb0xxhyv65jcsiww94r0rhto 2418053 2418052 2026-05-21T13:53:13Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet 2418053 wikitext text/x-wiki ghwxmsj6k6ywy49lcoomp34xaxtkj8k 2418054 2418053 2026-05-21T13:53:16Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet 2418054 wikitext text/x-wiki hp7n9ar1pw5mpd62tzktepv7ybx73em 2418055 2418054 2026-05-21T13:53:20Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet 2418055 wikitext text/x-wiki k18gqk97rgpydjktgu7u7kwwiebsjhe 2418056 2418055 2026-05-21T13:58:23Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet 2418056 wikitext text/x-wiki 05cqvfdmejdq38hnh4b1tttgfkzpgxe 2418057 2418056 2026-05-21T13:58:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92786 and previous config saved to /var/cache/conftool/dbconfig/20260521-135830-fceratto.json 2418057 wikitext text/x-wiki 61swcws8t5csin2zzkfz42s48y0cqhq 2418058 2418057 2026-05-21T13:59:03Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet 2418058 wikitext text/x-wiki isclh12uqqed9fapimc29tejm2w99pt 2418059 2418058 2026-05-21T13:59:53Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply 2418059 wikitext text/x-wiki lps70x6yah6qzxwg9thbf9m8aqxjj2z 2418061 2418059 2026-05-21T14:02:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2241.codfw.wmnet with OS trixie 2418061 wikitext text/x-wiki p7c8qls8sh1nzkodc04232rzg74ctxz 2418062 2418061 2026-05-21T14:03:18Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet 2418062 wikitext text/x-wiki eryce19nnrqla20hvdfulnzooti66s7 2418063 2418062 2026-05-21T14:03:20Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet 2418063 wikitext text/x-wiki fjtsurgqnu37sqwpdvmjv5e11q9h1va 2418064 2418063 2026-05-21T14:03:25Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet 2418064 wikitext text/x-wiki oqtqn3jue9cvhkotk4prxqpz4mc7rgb 2418065 2418064 2026-05-21T14:04:15Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Migration of db2241.codfw.wmnet completed 2418065 wikitext text/x-wiki hecvddueip31b75li89puj773frckf4 2418066 2418065 2026-05-21T14:07:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet 2418066 wikitext text/x-wiki hb9k3e02ipha8emdoj1ad7ilboq7s9i 2418067 2418066 2026-05-21T14:07:08Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet 2418067 wikitext text/x-wiki gxe72n4f0cr1ppk751mlww0cdys91ul 2418068 2418067 2026-05-21T14:08:00Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet 2418068 wikitext text/x-wiki 38p0zgazr2lhmqudg85gg4s4lffuzx0 2418069 2418068 2026-05-21T14:08:13Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2418069 wikitext text/x-wiki sl4vi4yyyherxxffi55c73b5q38hc0x 2418070 2418069 2026-05-21T14:08:28Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet 2418070 wikitext text/x-wiki bns8msu674ja31gu8jutg8z4vooqk9j 2418071 2418070 2026-05-21T14:08:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 (T426633)', diff saved to https://phabricator.wikimedia.org/P92788 and previous config saved to /var/cache/conftool/dbconfig/20260521-140837-fceratto.json 2418071 wikitext text/x-wiki teh04mlfhmknjmee0zgsr3zjdrhz2t8 2418072 2418071 2026-05-21T14:09:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2039.codfw.wmnet with reason: Maintenance 2418072 wikitext text/x-wiki fmzoaei4vcy0yc632gmse7g3syiyivp 2418073 2418072 2026-05-21T14:09:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2039 (T426633)', diff saved to https://phabricator.wikimedia.org/P92789 and previous config saved to /var/cache/conftool/dbconfig/20260521-140906-fceratto.json 2418073 wikitext text/x-wiki 9vaesz96hfuyq2lzomvp7n7holsfmt2 2418075 2418073 2026-05-21T14:11:16Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet 2418075 wikitext text/x-wiki 8pybeulteke66yg96elhwkasvk2id5q 2418076 2418075 2026-05-21T14:12:38Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: repool after maintenance 2418076 wikitext text/x-wiki os8xdwvbi9zugnhhpz8ytx51ya00xof 2418077 2418076 2026-05-21T14:12:44Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2418077 wikitext text/x-wiki tsp19pqnlvw6qzues771xox1mx4elne 2418078 2418077 2026-05-21T14:13:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet 2418078 wikitext text/x-wiki 6y5fmsfv7qw4y2hc6s4bo9giom9t0f0 2418079 2418078 2026-05-21T14:13:38Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2418079 wikitext text/x-wiki 8rvzcj21ss8w1f5wlry5jjsx3xygzey 2418080 2418079 2026-05-21T14:14:33Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet 2418080 wikitext text/x-wiki m0lwojg2a0yp1upfjntaqcdf7mj4n6l 2418081 2418080 2026-05-21T14:14:35Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet 2418081 wikitext text/x-wiki 3jpxi1leq9liyk2fi8awgt4ac1ipmrn 2418082 2418081 2026-05-21T14:14:39Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet 2418082 wikitext text/x-wiki ar66uv3dvraulkn1q5tvywakcu6gus8 2418084 2418082 2026-05-21T14:16:54Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet 2418084 wikitext text/x-wiki hkchjnzbds8xpnum4jw2eeb6rka1uyz 2418085 2418084 2026-05-21T14:17:02Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet 2418085 wikitext text/x-wiki banvpd2hb2h5fjarj8tsqnd7j2uxlsp 2418088 2418085 2026-05-21T14:17:27Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet 2418088 wikitext text/x-wiki mxn4aydax30oglz7rw2cygxmb4photl 2418089 2418088 2026-05-21T14:19:41Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418089 wikitext text/x-wiki 1vnsvsz2q38lf2bvpwofx310ge63wb9 2418090 2418089 2026-05-21T14:19:45Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418090 wikitext text/x-wiki nkkug0cvb2tewqidm2cbpo8trxz1tqw 2418091 2418090 2026-05-21T14:19:57Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet 2418091 wikitext text/x-wiki iibt9u7vai0k5n4xtldao1xnkm7pboz 2418092 2418091 2026-05-21T14:20:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 (T426633)', diff saved to https://phabricator.wikimedia.org/P92792 and previous config saved to /var/cache/conftool/dbconfig/20260521-142037-fceratto.json 2418092 wikitext text/x-wiki n6838ihn0hgb3ucdnmop1oteyylrpa3 2418093 2418092 2026-05-21T14:21:42Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet 2418093 wikitext text/x-wiki j0951feap97rb13lx3yvjq5jbp0oypk 2418094 2418093 2026-05-21T14:21:44Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418094 wikitext text/x-wiki ez3pc228z3xbd21r2wl8hsau11wt1h5 2418095 2418094 2026-05-21T14:21:50Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418095 wikitext text/x-wiki 9890kjiuw2illj7e67gak4f2eqgm06e 2418096 2418095 2026-05-21T14:24:43Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet 2418096 wikitext text/x-wiki ppyhl8pyaeux4omibreu1hj9pksomo3 2418097 2418096 2026-05-21T14:25:23Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet 2418097 wikitext text/x-wiki j0f76dz3dadj61ms3yicnnoh1hq0xcr 2418098 2418097 2026-05-21T14:25:31Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet 2418098 wikitext text/x-wiki j3gw0z87tb0ieqcqh09h1ilwf2tu6gn 2418099 2418098 2026-05-21T14:26:27Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418099 wikitext text/x-wiki 4z3s4f79pigtpw3h9z66dcmyu31bcdj 2418100 2418099 2026-05-21T14:26:32Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418100 wikitext text/x-wiki mj5e0750dz8ozpidhfgn9ykwhclkpko 2418101 2418100 2026-05-21T14:26:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet 2418101 wikitext text/x-wiki 7aeli6x5r2yb2u161sfi2qqmlkdbc46 2418102 2418101 2026-05-21T14:27:04Z Stashbot 7414 slyngshede@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-reboot (exit_code=1) rolling reboot on P{cp601[5-6].drmrs.wmnet} and A:cp 2418102 wikitext text/x-wiki 9bjg68udcwzvswouhk3cbnp785p5va7 2418105 2418102 2026-05-21T14:29:44Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet 2418105 wikitext text/x-wiki 1v2qfp7st4cljldv0eovds8dvcou9pv 2418106 2418105 2026-05-21T14:29:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki 2418106 wikitext text/x-wiki ln0yg0e0cdokmqlrh7owjdv5w2shksg 2418107 2418106 2026-05-21T14:30:22Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors 2418107 wikitext text/x-wiki g5xlgaa2mi2gagzqzgc8xlvfs29ysxp 2418108 2418107 2026-05-21T14:30:26Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors 2418108 wikitext text/x-wiki 1b6rgwliwve5hifawsg1xtv05e2x3sq 2418109 2418108 2026-05-21T14:30:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92793 and previous config saved to /var/cache/conftool/dbconfig/20260521-143045-fceratto.json 2418109 wikitext text/x-wiki 9ilt1nt4aqzmsvpiaxgfc80xtxemlve 2418110 2418109 2026-05-21T14:32:34Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet 2418110 wikitext text/x-wiki a0tl7tuyzorftt22gkys95xkaw7tf81 2418111 2418110 2026-05-21T14:32:36Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet 2418111 wikitext text/x-wiki rzq3nrkm8mkxgaa2rms6g526jmomd0t 2418112 2418111 2026-05-21T14:32:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet 2418112 wikitext text/x-wiki 007pn3zq4lxdoelqkccz2l5sm1xroqi 2418113 2418112 2026-05-21T14:32:41Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet 2418113 wikitext text/x-wiki f6baxhf7mj7jw2i60wf1ac8ayvgauek 2418114 2418113 2026-05-21T14:35:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet 2418114 wikitext text/x-wiki sff1q1kp2oqzome65gk4omdhwpan9mx 2418115 2418114 2026-05-21T14:35:44Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet 2418115 wikitext text/x-wiki of06khc7thq1j4xrqj785vx6t3ravxs 2418116 2418115 2026-05-21T14:37:47Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet 2418116 wikitext text/x-wiki 36gl2ojmpnn3vx40nrcntdjbvzuwmmg 2418117 2418116 2026-05-21T14:37:59Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors 2418117 wikitext text/x-wiki 3qsn70sbyrshuzv1tos89uw1g12g9ke 2418118 2418117 2026-05-21T14:38:03Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors 2418118 wikitext text/x-wiki eck77gatpby0abe4wo4qj47ccabrbg5 2418119 2418118 2026-05-21T14:39:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet 2418119 wikitext text/x-wiki cs4swscwjvuv0aj8uj6j9cszu449aoo 2418121 2418119 2026-05-21T14:40:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92795 and previous config saved to /var/cache/conftool/dbconfig/20260521-144055-fceratto.json 2418121 wikitext text/x-wiki idp7s21c6gm8k3yuvphoajh1b4od5a3 2418122 2418121 2026-05-21T14:41:18Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418122 wikitext text/x-wiki das1t2dde3p3zww7rog29mofg9oc6rb 2418123 2418122 2026-05-21T14:41:30Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418123 wikitext text/x-wiki aoc16t6hw6wamwmu42ptsm3d6dkej64 2418124 2418123 2026-05-21T14:42:09Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet 2418124 wikitext text/x-wiki m7w3tc0qgdyaydnmbqhr1rijrc1hyul 2418125 2418124 2026-05-21T14:42:12Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet 2418125 wikitext text/x-wiki 080pvlwu0r7xv1e1ulx513127lewzqr 2418126 2418125 2026-05-21T14:42:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad 2418126 wikitext text/x-wiki jp3bvczxieg73p2vurg1ufqzqeau9wa 2418127 2418126 2026-05-21T14:42:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet 2418127 wikitext text/x-wiki klrzwdgww7vkeuhdu4dbe289h0en7w2 2418128 2418127 2026-05-21T14:42:55Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve1001.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2418128 wikitext text/x-wiki mv5b2339clravzz0w7ddk3arggvugzt 2418129 2418128 2026-05-21T14:42:59Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet 2418129 wikitext text/x-wiki g1oopj6b1n5n8yqc267dnfwe1mzhl8r 2418130 2418129 2026-05-21T14:44:55Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418130 wikitext text/x-wiki bcn9qy0edzf2fm03kiw278a0gt29e48 2418131 2418130 2026-05-21T14:45:03Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418131 wikitext text/x-wiki 6l6q7vgpxtqcbwcm256ml8ts5heq6e3 2418132 2418131 2026-05-21T14:47:01Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet 2418132 wikitext text/x-wiki 4sp5psdsi6gd1tqq4sq0yjsel6n5tai 2418133 2418132 2026-05-21T14:47:05Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet 2418133 wikitext text/x-wiki 06gsejxl9wuufr72bfpb7t0unzeyg2o 2418134 2418133 2026-05-21T14:48:02Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet 2418134 wikitext text/x-wiki odnljcaaokkiyadkbrbzviz16pnuzw1 2418135 2418134 2026-05-21T14:49:44Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Migration of db2241.codfw.wmnet completed 2418135 wikitext text/x-wiki p8aj6k8i0m0xyix9zwzvy38ghsialt5 2418136 2418135 2026-05-21T14:49:46Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2418136 wikitext text/x-wiki tsktuuu7f8z5apx6bm2akikbk471hmz 2418137 2418136 2026-05-21T14:50:24Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet 2418137 wikitext text/x-wiki dr4uhkblnncrhk57yh5dramgc2957ep 2418138 2418137 2026-05-21T14:51:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 (T426633)', diff saved to https://phabricator.wikimedia.org/P92797 and previous config saved to /var/cache/conftool/dbconfig/20260521-145103-fceratto.json 2418138 wikitext text/x-wiki ai8zijjv73mhldarro5i03jxu1m2b2j 2418139 2418138 2026-05-21T14:51:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance 2418139 wikitext text/x-wiki 24s78lhyuulln5uhoua3uu9vr70ze4m 2418140 2418139 2026-05-21T14:51:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2040 (T426633)', diff saved to https://phabricator.wikimedia.org/P92798 and previous config saved to /var/cache/conftool/dbconfig/20260521-145132-fceratto.json 2418140 wikitext text/x-wiki ow0jznp856185fstw7d99o1z44zca9s 2418141 2418140 2026-05-21T14:53:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet 2418141 wikitext text/x-wiki 1ucr8j6hc7b58y13zv4llomq7lz7cym 2418142 2418141 2026-05-21T14:53:07Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet 2418142 wikitext text/x-wiki 2dcfzp55pbxosf8mr8g5qa1esjah89w 2418143 2418142 2026-05-21T14:53:09Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet 2418143 wikitext text/x-wiki i976r2z2abz7zyq25t008y7mr0t93k3 2418144 2418143 2026-05-21T14:53:11Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve1001.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2418144 wikitext text/x-wiki 9pf64l2k1a18kjzxvt0yu7htwvaf1no 2418145 2418144 2026-05-21T14:53:14Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet 2418145 wikitext text/x-wiki 2d6jk0ahtbmetzhcav31hr2ll424ro8 2418146 2418145 2026-05-21T14:53:19Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet 2418146 wikitext text/x-wiki 8wwxwucqinkojgxoc4gbxwei4y56dl1 2418147 2418146 2026-05-21T14:54:02Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet 2418147 wikitext text/x-wiki j4qq8n3r7uv2a4c738310x5eny53cgc 2418148 2418147 2026-05-21T14:54:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet 2418148 wikitext text/x-wiki r2idwyxbic9odmdtw6452lkq52x3zm9 2418149 2418148 2026-05-21T14:54:29Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290805|hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] 2418149 wikitext text/x-wiki atmz1ndjn1q4cx2xt2ie66ypbm4fhld 2418150 2418149 2026-05-21T14:55:00Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet 2418150 wikitext text/x-wiki fk1ijt9bumspkt8skh6w8it67c7mugq 2418151 2418150 2026-05-21T14:55:28Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet 2418151 wikitext text/x-wiki jynm46bb7sgnot48uunla89sh4488k6 2418152 2418151 2026-05-21T14:56:16Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290805|hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2418152 wikitext text/x-wiki hfaqwfu3ils6eqbxnzykdwg4nsmrp68 2418153 2418152 2026-05-21T14:57:22Z Stashbot 7414 claime: Disabling puppet on A:cp-text - T426323 2418153 wikitext text/x-wiki 9awg6u74k4qtj3u3x9oagy76h8s4l2y 2418154 2418153 2026-05-21T14:59:02Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki 2418154 wikitext text/x-wiki qoisbb011d2vs4rwvr1tknr21di1we4 2418155 2418154 2026-05-21T15:00:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet 2418155 wikitext text/x-wiki 9cgf3c57umdbvzqyryj3bpr5f8wdwvm 2418156 2418155 2026-05-21T15:00:23Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2418156 wikitext text/x-wiki ensvw4uq6vqzddnn0w3yqem94g0f5vw 2418157 2418156 2026-05-21T15:00:32Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master 2418157 wikitext text/x-wiki 4j83wv9l7ey7udu79kmy1pnzq5udw5q 2418159 2418157 2026-05-21T15:00:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors 2418159 wikitext text/x-wiki ifudg5egfel1avo2ax2plj7s73zoyd9 2418160 2418159 2026-05-21T15:01:00Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors 2418160 wikitext text/x-wiki 1lxo82ag9movd87yfe95g2e3ljf0mu1 2418162 2418160 2026-05-21T15:02:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet 2418162 wikitext text/x-wiki sfcqoy42sy88apvfcyx8hn49lkmmg94 2418163 2418162 2026-05-21T15:02:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet 2418163 wikitext text/x-wiki 1jo3dzqexn2ue76u7znycv35b53guvu 2418164 2418163 2026-05-21T15:03:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 (T426633)', diff saved to https://phabricator.wikimedia.org/P92799 and previous config saved to /var/cache/conftool/dbconfig/20260521-150308-fceratto.json 2418164 wikitext text/x-wiki h57lm1w6figlqyms0k0vyzwrq8fk873 2418165 2418164 2026-05-21T15:04:41Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290805|hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] (duration: 10m 11s) 2418165 wikitext text/x-wiki e7hk789844buw9njablm2shbjzqxi69 2418166 2418165 2026-05-21T15:05:18Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors 2418166 wikitext text/x-wiki rc5fu2y1qnzfraikdr47uavpaicav2p 2418167 2418166 2026-05-21T15:05:22Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors 2418167 wikitext text/x-wiki 6ps15g03e43jexoel0oq5tb92mrwl44 2418168 2418167 2026-05-21T15:06:12Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet 2418168 wikitext text/x-wiki 50g0opquiietnnq65svuq7tsl3d6vb9 2418169 2418168 2026-05-21T15:07:39Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master 2418169 wikitext text/x-wiki bi576178n4d674vv3m45o0hrp9koaes 2418170 2418169 2026-05-21T15:10:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet 2418170 wikitext text/x-wiki ezswwts4a2oq02m6pvnvq4sl6qy1r6w 2418171 2418170 2026-05-21T15:10:49Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet 2418171 wikitext text/x-wiki f27q12hd0o2a0lhjor8yoyprygbudfk 2418172 2418171 2026-05-21T15:10:57Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet 2418172 wikitext text/x-wiki 6wlljdih3axfehkljlsokmnlgtdbj7q 2418173 2418172 2026-05-21T15:11:00Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet 2418173 wikitext text/x-wiki 6cmph67s2yr1t6a6p55s4n61d4kyptf 2418174 2418173 2026-05-21T15:11:14Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet 2418174 wikitext text/x-wiki fsb77xg0f83kztbr161e79tsb3asjda 2418175 2418174 2026-05-21T15:11:30Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet 2418175 wikitext text/x-wiki 8bx3ulk44u7dslw8mywxdmds403xtli 2418176 2418175 2026-05-21T15:13:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92800 and previous config saved to /var/cache/conftool/dbconfig/20260521-151316-fceratto.json 2418176 wikitext text/x-wiki nv9sbugjwvqj7fdshrqjb2etf9baoem 2418178 2418176 2026-05-21T15:16:00Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet 2418178 wikitext text/x-wiki ksjnqcjr1rkoa8af8jqs8ajdnm3yaf2 2418180 2418178 2026-05-21T15:19:08Z Stashbot 7414 claime: Enabling puppet on A:cp-text - T426323 2418180 wikitext text/x-wiki b0dge7q4xc62a7j26sfe3soxkm6mron 2418181 2418180 2026-05-21T15:21:35Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet 2418181 wikitext text/x-wiki 05hgl7eab4todfs7cel8cnz4oc1mavk 2418182 2418181 2026-05-21T15:21:43Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet 2418182 wikitext text/x-wiki ihhjqpbi91r6dlocom0whay66a9qopl 2418183 2418182 2026-05-21T15:23:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92801 and previous config saved to /var/cache/conftool/dbconfig/20260521-152323-fceratto.json 2418183 wikitext text/x-wiki kliziuw1wwa9wacqhdsrdmg3m0nl6ea 2418184 2418183 2026-05-21T15:24:14Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2418184 wikitext text/x-wiki 2b9zxggcqrnlgcbq2uyscgjo52g4kbi 2418185 2418184 2026-05-21T15:24:41Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418185 wikitext text/x-wiki 7cw6rddazp5pdyp7qa4ynhk1f2xk3qy 2418186 2418185 2026-05-21T15:24:46Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2418186 wikitext text/x-wiki mn9troz4ear4g5chdgj1955qc3edrqg 2418187 2418186 2026-05-21T15:24:48Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2418187 wikitext text/x-wiki gpg7j8m5hbaaacehpn6dkjkgi417ezn 2418188 2418187 2026-05-21T15:25:15Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2418188 wikitext text/x-wiki 4crcl01xj9vdlu0fxqeu5242lym0o3f 2418189 2418188 2026-05-21T15:25:42Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418189 wikitext text/x-wiki k4ipe9833xmkfixzjt557ohsn9xrca5 2418190 2418189 2026-05-21T15:33:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 (T426633)', diff saved to https://phabricator.wikimedia.org/P92802 and previous config saved to /var/cache/conftool/dbconfig/20260521-153331-fceratto.json 2418190 wikitext text/x-wiki qpdbkonlnj7fhtwdtuayxm34rwdm9tt 2418191 2418190 2026-05-21T15:33:54Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2048.codfw.wmnet with reason: Maintenance 2418191 wikitext text/x-wiki 9lv50hqbymqcil9o0i7sq4ezb14uaus 2418192 2418191 2026-05-21T15:34:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2048 (T426633)', diff saved to https://phabricator.wikimedia.org/P92803 and previous config saved to /var/cache/conftool/dbconfig/20260521-153400-fceratto.json 2418192 wikitext text/x-wiki 1l53cu6f1ife75ig4o9a5u2q07ihjec 2418193 2418192 2026-05-21T15:34:52Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2418193 wikitext text/x-wiki crxw3hu16qt938jibgckdwup6u7wycw 2418194 2418193 2026-05-21T15:34:55Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2418194 wikitext text/x-wiki 7w0re236zj4m1wh5ct4qx0urkx5wq5r 2418195 2418194 2026-05-21T15:34:58Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2418195 wikitext text/x-wiki 55nwghtqq8e8sletwayqnae6icedhqn 2418196 2418195 2026-05-21T15:35:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2418196 wikitext text/x-wiki 26c1teah670eaq9lq2tznd2i4ixuqv0 2418197 2418196 2026-05-21T15:38:25Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418197 wikitext text/x-wiki 8ot2x3say2lufv3umbv025ulte6dnxr 2418198 2418197 2026-05-21T15:39:03Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418198 wikitext text/x-wiki 07iwki23i0rjz7nguo0j4l17elxqpf4 2418199 2418198 2026-05-21T15:41:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2048 (T426633)', diff saved to https://phabricator.wikimedia.org/P92804 and previous config saved to /var/cache/conftool/dbconfig/20260521-154108-fceratto.json 2418199 wikitext text/x-wiki nqmabpr46k3pf92s11fg3gek3ic4wow 2418200 2418199 2026-05-21T15:42:31Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es2048: Repooling 2418200 wikitext text/x-wiki 6xrlxoba6ivfye45vba46ti63ic13i5 2418202 2418200 2026-05-21T15:52:45Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2418202 wikitext text/x-wiki 1pr76147uge7owy42p1x9cdknnkln4a 2418203 2418202 2026-05-21T15:57:06Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418203 wikitext text/x-wiki mligmscimcrg1lt5ghc19yc7zsy39vq 2418204 2418203 2026-05-21T15:57:30Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2418204 wikitext text/x-wiki sxfiaw8c2no1xv7e9p8qtvd5obdfd4x 2418205 2418204 2026-05-21T15:58:37Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2418205 wikitext text/x-wiki 1b16zqr90hobtbehbgam61sqnfnaqu7 2418206 2418205 2026-05-21T15:58:40Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2418206 wikitext text/x-wiki 93j86s8k9papcty2qhhuw6fze0fkxpt 2418207 2418206 2026-05-21T15:58:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2418207 wikitext text/x-wiki jojxp5v355yre9r96vwko5xrt3qbxro 2418208 2418207 2026-05-21T15:58:45Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2418208 wikitext text/x-wiki 2k3cu7js4du3fpuwii2ncxwtfsp3dub 2418209 2418208 2026-05-21T16:02:47Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2418209 wikitext text/x-wiki krvuv7zrjpphea1jqwc9xjfbfzwb7go 2418210 2418209 2026-05-21T16:02:51Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2418210 wikitext text/x-wiki iln6har0bs3etr412uh54oarppbatap 2418211 2418210 2026-05-21T16:02:54Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2418211 wikitext text/x-wiki gdclo1a1aiwz01081jh1333ltq7rsg0 2418212 2418211 2026-05-21T16:02:56Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2418212 wikitext text/x-wiki m6mqte38arnrw1efwswf2qlozy8jjmw 2418213 2418212 2026-05-21T16:05:34Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2418213 wikitext text/x-wiki 2x0s7io4lr0kykmqtapm0hgzbznj66i 2418214 2418213 2026-05-21T16:05:37Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2418214 wikitext text/x-wiki n1qfb2c0hmnnroskrxsksbevsax3vk3 2418215 2418214 2026-05-21T16:05:39Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2418215 wikitext text/x-wiki qztdxv13nh8y2cpgakcku8jgtsbglx1 2418216 2418215 2026-05-21T16:05:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2418216 wikitext text/x-wiki l6h2ixcjday2sewh0v7gydo8kz6yeal 2418218 2418216 2026-05-21T16:15:40Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2418218 wikitext text/x-wiki ohid0g64cu9qnjb26bs4nnajg1uvyhb 2418219 2418218 2026-05-21T16:15:43Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2418219 wikitext text/x-wiki 0db6qr0mi8spoyqj2ibksqoe2lqtxpe 2418220 2418219 2026-05-21T16:15:45Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2418220 wikitext text/x-wiki olw4ybdrqks7oao7v5qvnsq6lp56rzn 2418221 2418220 2026-05-21T16:15:47Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2418221 wikitext text/x-wiki cy11kmsm0mu6romqcbnszm0vvnfbq66 2418222 2418221 2026-05-21T16:18:11Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool pc2 (T421705)', diff saved to https://phabricator.wikimedia.org/P92807 and previous config saved to /var/cache/conftool/dbconfig/20260521-161808-ladsgroup.json 2418222 wikitext text/x-wiki 9yg8zn7gur9fwq5xcx2msmyjq46n8k6 2418223 2418222 2026-05-21T16:18:23Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2048: Repooling 2418223 wikitext text/x-wiki e2ymysam3zqkk5aeapilojstd4vb4nn 2418227 2418223 2026-05-21T16:24:03Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc2022.codfw.wmnet with reason: Move to nftables 2418227 wikitext text/x-wiki inbft8x0xr6is7osgz34udiufveli3i 2418228 2418227 2026-05-21T16:24:33Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc1022.eqiad.wmnet with reason: Move to nftables 2418228 wikitext text/x-wiki bbxmmzxdm9ijcf1quomdlktpfiypk6p 2418229 2418228 2026-05-21T16:26:35Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2418229 wikitext text/x-wiki togi4dc0l4kp13ei3glmvkcwvf9weva 2418234 2418229 2026-05-21T16:33:55Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" 2418234 wikitext text/x-wiki qbbo5wybxbl40ujhd3kvq4la9kmgikj 2418235 2418234 2026-05-21T16:34:00Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" 2418235 wikitext text/x-wiki kbehiai9t08a7nbe8l2nu2z2piyb5o1 2418236 2418235 2026-05-21T16:34:02Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418236 wikitext text/x-wiki 5wpi81mgpeco8tjoz6tw6k7thvmj9fw 2418237 2418236 2026-05-21T16:34:48Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 2418237 wikitext text/x-wiki i8rzuy0q7atmy0k6kf3hmzcxdjdohgu 2418238 2418237 2026-05-21T16:35:38Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 2418238 wikitext text/x-wiki hrdxbf538ev10o55w6bnhzuw695ujsc 2418239 2418238 2026-05-21T16:35:40Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2030 2418239 wikitext text/x-wiki 8qe7i9we8qxboa5uiei7jhyumm3pa69 2418240 2418239 2026-05-21T16:35:42Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 2418240 wikitext text/x-wiki i2u8bbmxcahwz5hpvjle4np2i2abta3 2418241 2418240 2026-05-21T16:35:51Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2030 2418241 wikitext text/x-wiki og5s12yoyxg6pu7k04ybixnzelrm4s7 2418242 2418241 2026-05-21T16:35:54Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 2418242 wikitext text/x-wiki 1bhxy79wc4y8xkmvs5inkxtl81pbq25 2418243 2418242 2026-05-21T16:35:59Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 2418243 wikitext text/x-wiki ihh30bfg02uqrlosfq9ql5ex7ybgas9 2418244 2418243 2026-05-21T16:36:12Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 2418244 wikitext text/x-wiki nn4x2qigbcamuzqt7yqjp88h1xwwu4u 2418245 2418244 2026-05-21T16:37:04Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 2418245 wikitext text/x-wiki nf36h6xdp3ekunf875ww4l81nu49rcs 2418246 2418245 2026-05-21T16:37:14Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1014.eqiad.wmnet 2418246 wikitext text/x-wiki rmbqnlsxjwc5a4k55msvmanw32141df 2418247 2418246 2026-05-21T16:37:17Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 2418247 wikitext text/x-wiki 45kp2jsgkytfux8nx5tco69x83roekf 2418248 2418247 2026-05-21T16:40:44Z Stashbot 7414 papaul: rebooting msw-a1-codfw 2418248 wikitext text/x-wiki frp9olcqkdivskkp0id3glao1b2zf9b 2418249 2418248 2026-05-21T16:43:13Z Stashbot 7414 papaul: rebooting msw-b6-codfw 2418249 wikitext text/x-wiki n7gnf3ks3r88byc7og0zrhe6anod8rm 2418250 2418249 2026-05-21T16:45:03Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1014.eqiad.wmnet 2418250 wikitext text/x-wiki dzvh69vmdew1d6akyujs3tlno43haap 2418251 2418250 2026-05-21T16:48:03Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet 2418251 wikitext text/x-wiki 91d73n7e50hngac3lk8dibl0s3no4y6 2418252 2418251 2026-05-21T16:48:18Z Stashbot 7414 papaul: rebooting msw-b7-codfw 2418252 wikitext text/x-wiki 6o8pgeqy2b5xzzquz4kytknl235jc9p 2418253 2418252 2026-05-21T16:51:40Z Stashbot 7414 papaul: rebooting msw-c6-codfw 2418253 wikitext text/x-wiki 3zrnbixg0pkaexzxk20rum0l8wxjjqf 2418254 2418253 2026-05-21T16:52:28Z Stashbot 7414 papaul: rebooting msw-c7-codfw 2418254 wikitext text/x-wiki qxumt1xhkrp8p0d6ad6d0w31xp61cnv 2418255 2418254 2026-05-21T16:55:03Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 2418255 wikitext text/x-wiki so7hwpygyvc3qty3z59vthu3ti1u5ap 2418256 2418255 2026-05-21T16:55:54Z Stashbot 7414 papaul: rebooting msw-d3-codfw 2418256 wikitext text/x-wiki 46s9cnkog69bl5ly4ydr4dnj4qbis91 2418257 2418256 2026-05-21T16:57:24Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 2418257 wikitext text/x-wiki mp2uwy4soof61rzuq1rgxb344a8im2c 2418258 2418257 2026-05-21T16:58:07Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 2418258 wikitext text/x-wiki 7qv1l89lw3hqixhiw3mor7gjn0r01z1 2418259 2418258 2026-05-21T16:58:29Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 2418259 wikitext text/x-wiki dqey8gh5hlxvc1vj612p0bamv6vnusp 2418260 2418259 2026-05-21T16:58:39Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2031 2418260 wikitext text/x-wiki 53lsfiwqhox8fpcllwaphz3podwcaxk 2418261 2418260 2026-05-21T17:00:26Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 2418261 wikitext text/x-wiki olny8ejdyw0wdfnfqaxcqm4pybl0ziy 2418262 2418261 2026-05-21T17:03:16Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2418262 wikitext text/x-wiki 8yqgzjyjbrpzbcgdaesk9qobvpvn6xo 2418263 2418262 2026-05-21T17:03:18Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2418263 wikitext text/x-wiki 3tybao9ebhuuvdgaib673ohyjfp89qz 2418264 2418263 2026-05-21T17:03:20Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2418264 wikitext text/x-wiki igywano3nj09nwn8gex6sgziqhw3mly 2418265 2418264 2026-05-21T17:03:23Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2418265 wikitext text/x-wiki pdwhum8buwt8wqvtwt7qyzmew9n4wol 2418266 2418265 2026-05-21T17:06:44Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418266 wikitext text/x-wiki cz9it35d8gkeyte8ahe3wcvq8kfkcya 2418267 2418266 2026-05-21T17:07:04Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418267 wikitext text/x-wiki l889tl6extq01nf7ukvznp06pqgq59s 2418268 2418267 2026-05-21T17:07:18Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418268 wikitext text/x-wiki lnbxqu03olxpc2cz9wkavjothzzg061 2418269 2418268 2026-05-21T17:07:28Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2418269 wikitext text/x-wiki iet0fhqnd45pnj9qdyo8lcvedjmrj12 2418270 2418269 2026-05-21T17:07:40Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418270 wikitext text/x-wiki hsnn6wvcvhwrz3uo3ocw0crojr1osyh 2418271 2418270 2026-05-21T17:08:00Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2418271 wikitext text/x-wiki i8wkg4mhiafda3bspmiza7tstsp8j4h 2418272 2418271 2026-05-21T17:08:02Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2418272 wikitext text/x-wiki t6mtirrsw8dsjra1m2fyv7yi7ifo5uo 2418273 2418272 2026-05-21T17:08:14Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2418273 wikitext text/x-wiki 0ekuexumy3uj9lko5xz7aoz5rtkiwcu 2418274 2418273 2026-05-21T17:08:16Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2418274 wikitext text/x-wiki gq8fsar1jvahw8l6y8jtucg19es3m0y 2418275 2418274 2026-05-21T17:08:26Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repool pc2 (T421705)', diff saved to https://phabricator.wikimedia.org/P92810 and previous config saved to /var/cache/conftool/dbconfig/20260521-170823-ladsgroup.json 2418275 wikitext text/x-wiki g76ie9smu3dr91jfbfdj2ddxiorbcop 2418276 2418275 2026-05-21T17:08:29Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2418276 wikitext text/x-wiki kqu74h2azt4xdbeyznzoc2qa1rsksys 2418277 2418276 2026-05-21T17:08:31Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2418277 wikitext text/x-wiki jxuke06oi2ce8g5922ezbwxgodfi60o 2418278 2418277 2026-05-21T17:08:43Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2418278 wikitext text/x-wiki ej18obxt7yysc3kderxmg8rriyihu88 2418279 2418278 2026-05-21T17:08:46Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2418279 wikitext text/x-wiki 28mxox3v1u62ol2ftny8gulazd2ymu9 2418280 2418279 2026-05-21T17:09:01Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2418280 wikitext text/x-wiki c4vi3ze7ka4b6ysjvjystj9kh7i1u3l 2418281 2418280 2026-05-21T17:09:03Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2418281 wikitext text/x-wiki 8keuf1o6uppczcjunftjxr4qnnlag0q 2418282 2418281 2026-05-21T17:09:28Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2418282 wikitext text/x-wiki 3u58tbqt2wdroned2pu7axehus0l224 2418283 2418282 2026-05-21T17:11:23Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418283 wikitext text/x-wiki 011207gwvxxfh54z43efba69c6x1q96 2418284 2418283 2026-05-21T17:11:31Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418284 wikitext text/x-wiki 633xl30jk8y0mamvcvuho6iccny3160 2418286 2418284 2026-05-21T17:13:43Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1016.eqiad.wmnet 2418286 wikitext text/x-wiki 1cjhtjdhswxxnfvocwyzvnyo90th4pc 2418287 2418286 2026-05-21T17:14:41Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418287 wikitext text/x-wiki dw8p2mz7gv3f0qtusjmw0qxoeznf4jg 2418288 2418287 2026-05-21T17:15:00Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418288 wikitext text/x-wiki 7qg8k0fzhlghcx3wnbvmcolbz8ts2cg 2418289 2418288 2026-05-21T17:22:22Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2418289 wikitext text/x-wiki fp899ae4nozz7y9mb20zo9cwx1vvohb 2418290 2418289 2026-05-21T17:22:54Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1016.eqiad.wmnet 2418290 wikitext text/x-wiki 6lc9stegxfpgtwdetnlb5dwuu8lkhg9 2418291 2418290 2026-05-21T17:23:29Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2418291 wikitext text/x-wiki 7538buqvkl353tbsfbcbq860tmni2ak 2418292 2418291 2026-05-21T17:24:24Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2418292 wikitext text/x-wiki ffgtl070bhjb5ncg1y6q52w0fphdfo3 2418293 2418292 2026-05-21T17:25:06Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2418293 wikitext text/x-wiki 66cioigt25bjvl9bwjbdprskvga7s0m 2418294 2418293 2026-05-21T17:25:38Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2418294 wikitext text/x-wiki qs0u2x82u9jv0cazomuqwq59zkm74u6 2418298 2418294 2026-05-21T17:30:53Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418298 wikitext text/x-wiki ikc0gnwuncj9ee6oradju30y2ajsfjd 2418300 2418298 2026-05-21T17:36:02Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2418300 wikitext text/x-wiki kvb80z5om1eke1ilai4x16gwnlj8dqg 2418301 2418300 2026-05-21T17:36:58Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2418301 wikitext text/x-wiki g605trzq9rwn0moc4gtk8dghlol92ie 2418302 2418301 2026-05-21T17:37:28Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 2418302 wikitext text/x-wiki 4zq593kme1mt66h8k2km9t56e2j8qf7 2418303 2418302 2026-05-21T17:38:41Z Stashbot 7414 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down 2418303 wikitext text/x-wiki b1ab9vs9sqi1zlv6tf65y4wcjpo4ceq 2418304 2418303 2026-05-21T17:39:17Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2418304 wikitext text/x-wiki 88xodz3ilo6wvvuixu05rwk0wc9qul0 2418305 2418304 2026-05-21T17:39:45Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 2418305 wikitext text/x-wiki 1657mzxm0yb75jy637v1v9tbgh1zknw 2418306 2418305 2026-05-21T17:40:09Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2418306 wikitext text/x-wiki pvtt846r1jilixpqx6659b5jxnjn0nr 2418307 2418306 2026-05-21T17:40:39Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2418307 wikitext text/x-wiki 6pj0fzj1ahssqnskazl3rgceklnapwr 2418308 2418307 2026-05-21T17:40:59Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2418308 wikitext text/x-wiki 9xl4mthyje0d18x7fg5r3gvmrilndub 2418310 2418308 2026-05-21T17:41:20Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2418310 wikitext text/x-wiki iyqofxkauaczib620n7xmmzp50wtw7u 2418311 2418310 2026-05-21T17:41:23Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2418311 wikitext text/x-wiki ipi3nytguhtb7z3qs83bwkoull9x9p7 2418312 2418311 2026-05-21T17:41:32Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2418312 wikitext text/x-wiki eu4klt9pt5yvipo3z3fomux68w3sfkn 2418313 2418312 2026-05-21T17:41:36Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2418313 wikitext text/x-wiki l6vl835tf9w84muprec111pq614h2zu 2418314 2418313 2026-05-21T17:41:40Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2418314 wikitext text/x-wiki 96z5zrpqpv3cgsc3yy4cqblg40wdym6 2418315 2418314 2026-05-21T17:41:42Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2418315 wikitext text/x-wiki oezo37vrg25tuvb22hnbu7jsa3nae7t 2418316 2418315 2026-05-21T17:41:54Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2418316 wikitext text/x-wiki e1horpksxlmk9vncr8l0sw8pc0ogdml 2418317 2418316 2026-05-21T17:42:25Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2418317 wikitext text/x-wiki g96b5zxre61r97scupyqtddgvkypzk8 2418318 2418317 2026-05-21T17:42:35Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2418318 wikitext text/x-wiki h650z0l09sicuug3dl1u3suaq8o65x7 2418319 2418318 2026-05-21T17:43:26Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2418319 wikitext text/x-wiki hz7xq5asox1ocjccmr57effmapwpqyo 2418320 2418319 2026-05-21T17:43:45Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2418320 wikitext text/x-wiki dlmu8tlkzxpl6pi5dq8hdfcoejmjaub 2418321 2418320 2026-05-21T17:43:49Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2418321 wikitext text/x-wiki kaezeezf1st48j5tvoewi5b2g0n5fok 2418322 2418321 2026-05-21T17:46:29Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2418322 wikitext text/x-wiki 8ptqor2exh9dytxk6er8t6iybxjmvjx 2418323 2418322 2026-05-21T17:46:50Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2418323 wikitext text/x-wiki 0vuf075d0sift4u089ptu5fqxq8s52o 2418324 2418323 2026-05-21T17:48:36Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply 2418324 wikitext text/x-wiki 6m2g7x8ohsn1icw5g5aygdwnzb4pa0v 2418325 2418324 2026-05-21T17:49:24Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply 2418325 wikitext text/x-wiki bomswcniiqiocbgj1rhwipf16od61d7 2418326 2418325 2026-05-21T17:49:55Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply 2418326 wikitext text/x-wiki 31kj60mv66hjetql8rzyc49xrmle8m9 2418327 2418326 2026-05-21T17:50:31Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply 2418327 wikitext text/x-wiki eus0p7q44vxdmgcusw3hza5x4kslpl7 2418328 2418327 2026-05-21T17:51:02Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply 2418328 wikitext text/x-wiki 4brc1aenh427pwlwzzb7pnkbnxdbel0 2418329 2418328 2026-05-21T17:51:20Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply 2418329 wikitext text/x-wiki m3n0pro9htd2elqfbt2h873mar1ibb5 2418330 2418329 2026-05-21T17:51:51Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply 2418330 wikitext text/x-wiki bujxk39d2rfaepbpo6v79ksi2nfvbmm 2418331 2418330 2026-05-21T17:52:11Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2418331 wikitext text/x-wiki t60wgrjdc1imwxv4fnwn4t08bbjxqev 2418332 2418331 2026-05-21T17:52:30Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2418332 wikitext text/x-wiki esot7fx2nlghufennz0i4wlp2rtrr20 2418333 2418332 2026-05-21T17:52:42Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2418333 wikitext text/x-wiki kfm115qz1otuf0u6gpby26wvhgfo462 2418334 2418333 2026-05-21T17:52:56Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2418334 wikitext text/x-wiki qh5tk8twgtehqgjafiltjobkjtl67ym 2418335 2418334 2026-05-21T17:53:10Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2418335 wikitext text/x-wiki orfg73by5ratiud46ucl9rql3l4ztou 2418336 2418335 2026-05-21T17:53:41Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2418336 wikitext text/x-wiki cz50y8521kzeeseim8xukps3axf8pry 2418337 2418336 2026-05-21T17:54:43Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2418337 wikitext text/x-wiki 5720sgdb4gtp7qfbkoklkqsr6j3bnou 2418342 2418337 2026-05-21T18:39:50Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418342 wikitext text/x-wiki 2q7nps042yjmrbuob4k9j38ua499516 2418343 2418342 2026-05-21T18:50:09Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418343 wikitext text/x-wiki 4xly27wz6zs9phl6e161yejsfostlpl 2418345 2418343 2026-05-21T18:53:09Z Stashbot 7414 papaul: rebooting msw1-codfw 2418345 wikitext text/x-wiki h9mwwihkpat0zrefpbnnoxsnjmzp17k 2418347 2418345 2026-05-21T18:59:34Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418347 wikitext text/x-wiki j2ks643k0nr12ubk82e6j3xpgvtud4y 2418348 2418347 2026-05-21T19:10:01Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418348 wikitext text/x-wiki jkf3fk97ispfnnojzh31aqpqcklt4ni 2418351 2418348 2026-05-21T19:22:07Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:restbase 2418351 wikitext text/x-wiki k47qxdntbkc7hc46bd4gfgs2z49kvx6 2418352 2418351 2026-05-21T20:16:02Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418352 wikitext text/x-wiki 14i8shl8c79fy8zohyfomwqhcdsfa6k 2418353 2418352 2026-05-21T20:26:11Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418353 wikitext text/x-wiki jws5intnmka9ep5bh97zht95ian78ki 2418359 2418353 2026-05-21T21:19:47Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418359 wikitext text/x-wiki ekwn2c91opysb2dy0ve46wlniscgpk0 2418360 2418359 2026-05-21T21:20:10Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418360 wikitext text/x-wiki 9pl8dyu0qdonlz494zw2tcjhtibg6h5 2418361 2418360 2026-05-21T21:25:19Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418361 wikitext text/x-wiki d4qxfl5ze4zc0p2ug203zls19c7uoqr 2418362 2418361 2026-05-21T21:25:50Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418362 wikitext text/x-wiki 18c9oq5zjaqs8rxve37bbd3soe78m29 2418364 2418362 2026-05-21T21:44:08Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host zuul2002.codfw.wmnet with OS trixie 2418364 wikitext text/x-wiki 4rq577qngornhz0141hy3xbhe76iu4y 2418368 2418364 2026-05-21T21:49:20Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2418368 wikitext text/x-wiki hih4xackpgq2qbxrji7w01ozs6dbnre 2418369 2418368 2026-05-21T21:49:33Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2418369 wikitext text/x-wiki 1jjdpf8k69649213qr2gkou2pac708r 2418372 2418369 2026-05-21T22:02:19Z Stashbot 7414 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: T426560 - bking@cumin2002 2418372 wikitext text/x-wiki fkl4aecdm2ztbdrkuvidbbkf0zuueso 2418373 2418372 2026-05-21T22:03:22Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage 2418373 wikitext text/x-wiki 2mhw7vrkhetqkan2cou5vo0yz1xnt1m 2418375 2418373 2026-05-21T22:08:55Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage 2418375 wikitext text/x-wiki mfuzz9qu0whunt1k7o8qqds4f0hf7q8 2418377 2418375 2026-05-21T22:26:40Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul2002.codfw.wmnet with OS trixie 2418377 wikitext text/x-wiki 5evj9ticrts2xjwuseya8v7ysip2dxl 2418380 2418377 2026-05-21T23:36:27Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290954|Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957|Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958|Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960|Drop unused wpReportIncidentLocalLinks]] 2418380 wikitext text/x-wiki mb26si0o57mgy3z40bcepmjcwvpvkpt 2418381 2418380 2026-05-21T23:38:09Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290954|Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957|Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958|Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960|Drop unused wpReportIncidentLocalLinks]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified 2418381 wikitext text/x-wiki sr0zogyqoldbjltt6zj3d68o7e2dzfj 2418382 2418381 2026-05-21T23:38:59Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2418382 wikitext text/x-wiki 8c9d171osqe39m7r0k0sh7u6vmqh0eu 2418384 2418382 2026-05-21T23:43:09Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290954|Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957|Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958|Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960|Drop unused wpReportIncidentLocalLinks]] (duration: 06m 42s) 2418384 wikitext text/x-wiki 9zvdc41jcz9vc4fnwmz098zc9czegca 2418387 2418384 2026-05-22T04:56:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1012.eqiad.wmnet 2418387 wikitext text/x-wiki ez2fghxvdztja3y3v72gs64eitifs9f 2418388 2418387 2026-05-22T05:03:10Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2418388 wikitext text/x-wiki g40gzag05m4izseiqasns46fax1e33s 2418389 2418388 2026-05-22T05:06:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2418389 wikitext text/x-wiki 53pfjj4f23sqmmpni6z9jaxem8oesgf 2418390 2418389 2026-05-22T05:11:55Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2418390 wikitext text/x-wiki jka9r8ln2ojwtnez49o7n3fntp3al2b 2418391 2418390 2026-05-22T05:11:57Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418391 wikitext text/x-wiki a30qk2jgvxlmwty71u1orxmg1kcvddr 2418392 2418391 2026-05-22T05:11:59Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1012.eqiad.wmnet 2418392 wikitext text/x-wiki c7108wgdkqr9fsdermcyspqxzvgjsl0 2418393 2418392 2026-05-22T05:17:27Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy2005.codfw.wmnet with reason: Reboot 2418393 wikitext text/x-wiki jj6tp39836cb6y62ug2i9yb80wjd9d1 2418394 2418393 2026-05-22T05:19:42Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1028.eqiad.wmnet with reason: Reboot 2418394 wikitext text/x-wiki mj9w7bdfhtihywnfqkhcispv3v7wv8f 2418395 2418394 2026-05-22T05:23:14Z Stashbot 7414 marostegui: Failover m5-master T426633 2418395 wikitext text/x-wiki bowanqd5b1pidwuqq778ngal2d0xnv1 2418396 2418395 2026-05-22T05:24:08Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2418396 wikitext text/x-wiki joipbnmvomi2fkn7cqr10u57ztdy7au 2418397 2418396 2026-05-22T05:25:45Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2418397 wikitext text/x-wiki 6562ozphf8g9gnahaygs5632su2nokb 2418398 2418397 2026-05-22T06:01:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org 2418398 wikitext text/x-wiki lmb5j2rgfryocsp3i6uvmi3k5llxqz2 2418399 2418398 2026-05-22T06:07:23Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org 2418399 wikitext text/x-wiki cz17vewvryngobze5t1t7qy8fodmubg 2418400 2418399 2026-05-22T06:08:18Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet 2418400 wikitext text/x-wiki ke0n0i6cwasg8wi739sookvpk4d1e80 2418401 2418400 2026-05-22T06:13:10Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet 2418401 wikitext text/x-wiki 9x5qd7u28r4cyvfzft3449z52u0whtw 2418402 2418401 2026-05-22T06:46:34Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp3067.esams.wmnet} and A:cp 2418402 wikitext text/x-wiki posopql7s1ok0j225occ1xbm47b22il 2418403 2418402 2026-05-22T06:56:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet 2418403 wikitext text/x-wiki 3tfr5pq4inx29eziuz64x2guivjyull 2418404 2418403 2026-05-22T06:58:25Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet 2418404 wikitext text/x-wiki tfgrv4w0u9bzxquw36mgnjw9enf5ygi 2418405 2418404 2026-05-22T06:58:28Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp3067.esams.wmnet} and A:cp 2418405 wikitext text/x-wiki lii2387oy7110e90uw41u7iwryd8mi2 2418406 2418405 2026-05-22T06:58:42Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp3075.esams.wmnet} and A:cp 2418406 wikitext text/x-wiki oimo8sew4cxoj094oot8zwrcgdgv99w 2418407 2418406 2026-05-22T07:01:03Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1057 2418407 wikitext text/x-wiki 99r8elhk7qbq3qecekmzstzbfd0qjgc 2418408 2418407 2026-05-22T07:02:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1057 2418408 wikitext text/x-wiki jn0vpbs6h3ka0r0kx8xypj2pd20jj5x 2418409 2418408 2026-05-22T07:04:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet 2418409 wikitext text/x-wiki fse2avokqk3bjoqdv4muej4dfqb75go 2418410 2418409 2026-05-22T07:06:31Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A 2418410 wikitext text/x-wiki ivh4gb5l3jtxgk2rupcjy48d0mii1p8 2418411 2418410 2026-05-22T07:10:26Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet 2418411 wikitext text/x-wiki i688jfx00ja2l9z4vl309li2k21rn85 2418412 2418411 2026-05-22T07:10:29Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp3075.esams.wmnet} and A:cp 2418412 wikitext text/x-wiki s647bkogmfayzx3xt56b0zxvobyr9of 2418413 2418412 2026-05-22T07:11:38Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A 2418413 wikitext text/x-wiki f1rukf6yhxn3q2jks6ys4xg7hq706x5 2418414 2418413 2026-05-22T07:14:15Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp306[8-9].esams.wmnet} and A:cp 2418414 wikitext text/x-wiki n6lnjh25miig7meyokq70rbxzqkoitu 2418415 2418414 2026-05-22T07:26:05Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet 2418415 wikitext text/x-wiki 7qte59h4wrm88wepbbctl59247tvkkl 2418416 2418415 2026-05-22T07:32:01Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet 2418416 wikitext text/x-wiki 4pyyxf2vfh1izh8k0tqq54ioj2g6ka6 2418418 2418416 2026-05-22T08:05:08Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2418418 wikitext text/x-wiki 4re44iq8msujdjuybjz7c16v4up7vdg 2418419 2418418 2026-05-22T08:05:52Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2418419 wikitext text/x-wiki qypumy5vwsqmkxpi9z9h1rrvlvk259z 2418420 2418419 2026-05-22T08:07:49Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet 2418420 wikitext text/x-wiki 18fmi862e91e1cn72s3vu20kiqjenfl 2418421 2418420 2026-05-22T08:07:52Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp306[8-9].esams.wmnet} and A:cp 2418421 wikitext text/x-wiki 2dbodcvhjzae6ogims5iel7qha227wm 2418426 2418421 2026-05-22T08:09:57Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2418426 wikitext text/x-wiki 7rm8s7whxe0wnbra2wl8udpq5si2xzs 2418428 2418426 2026-05-22T08:15:31Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" 2418428 wikitext text/x-wiki hivln8rwbpfdc2a3ixbkd867q653am2 2418429 2418428 2026-05-22T08:15:34Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti1058.eqiad.wmnet on all recursors 2418429 wikitext text/x-wiki 0uodbzchu0kdojwkhn1m1zq130gwisv 2418430 2418429 2026-05-22T08:15:36Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" 2418430 wikitext text/x-wiki t5r3biw0wgpbmzstni4t29z8oj4mvjp 2418431 2418430 2026-05-22T08:15:39Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418431 wikitext text/x-wiki bj8r20iu5kf7fb8acugzet2w2t2zg1c 2418432 2418431 2026-05-22T08:15:41Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti1058.eqiad.wmnet on all recursors 2418432 wikitext text/x-wiki 9w16x3vb7etj5wj0a6bvbilnykjas15 2418433 2418432 2026-05-22T08:18:30Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp307[6-7].esams.wmnet} and A:cp 2418433 wikitext text/x-wiki a3sdgve9bqtunz8kmnui24g2e2jqkoc 2418434 2418433 2026-05-22T08:30:05Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet 2418434 wikitext text/x-wiki gitc83kfp7nwunwtgii920o2lajpbfj 2418435 2418434 2026-05-22T08:33:25Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2418435 wikitext text/x-wiki 2lh2769dd0dyszm0udh3fn4go584u7q 2418436 2418435 2026-05-22T08:33:56Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2418436 wikitext text/x-wiki iou9gt7a7hiw7f9fdh9hmbce5h785ul 2418437 2418436 2026-05-22T08:40:04Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2418437 wikitext text/x-wiki mazhr78s4p6lnwuoeffh4o02ivwg0j9 2418438 2418437 2026-05-22T08:46:52Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2418438 wikitext text/x-wiki hp55f51t8h6jul0yr2ppwip12slccr5 2418439 2418438 2026-05-22T08:47:39Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2418439 wikitext text/x-wiki nhgjlgnax3zdr94jk82xlfmyvcwn8xy 2418440 2418439 2026-05-22T09:03:21Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie 2418440 wikitext text/x-wiki 5on6nrgqpqc9mfsufzc7m1uekq5z4nt 2418441 2418440 2026-05-22T09:04:23Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2418441 wikitext text/x-wiki m462qma07zk6x91kgs738dhwjhpjfho 2418442 2418441 2026-05-22T09:11:50Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet 2418442 wikitext text/x-wiki p0i79qbwmfzz343ifx9jdpw6cs5aswr 2418443 2418442 2026-05-22T09:11:53Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp307[6-7].esams.wmnet} and A:cp 2418443 wikitext text/x-wiki 4peks2ba25fajb4i5ttxkjctnyc9hav 2418444 2418443 2026-05-22T09:14:27Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp307[0-1].esams.wmnet} and A:cp 2418444 wikitext text/x-wiki rvfps4rfyfdscjjod40i3not0p8b22h 2418445 2418444 2026-05-22T09:16:50Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie 2418445 wikitext text/x-wiki h96o29gw46018iwxnbj24vndupwtisg 2418446 2418445 2026-05-22T09:21:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2418446 wikitext text/x-wiki 42e0f9t3q3w84p4homlewdfd1h1gsaf 2418447 2418446 2026-05-22T09:26:06Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet 2418447 wikitext text/x-wiki qciff2u1ylz0k01xgkx45ulk16hu0fk 2418448 2418447 2026-05-22T09:26:14Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418448 wikitext text/x-wiki n67p6mgaxrrjg9w20s2eh7vsdu3bunw 2418449 2418448 2026-05-22T09:26:24Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418449 wikitext text/x-wiki 0kr242c8mhdo3e3s7xvzdasp14dq1sq 2418450 2418449 2026-05-22T09:33:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie 2418450 wikitext text/x-wiki 4tr6t8xchdiwu3grisl5ojsf97ybu76 2418451 2418450 2026-05-22T09:34:07Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS trixie 2418451 wikitext text/x-wiki 8l3db6f3ovywl1ekx80xy1nvyxcky1i 2418452 2418451 2026-05-22T09:34:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2418452 wikitext text/x-wiki tmnvmjrg577rqefvae07aneglxke3qr 2418453 2418452 2026-05-22T09:34:55Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1172: Upgrading db1172.eqiad.wmnet 2418453 wikitext text/x-wiki mfuv2oxpwtcvfzf55o8oq3hhvd4buwk 2418454 2418453 2026-05-22T09:35:44Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Upgrading db1172.eqiad.wmnet 2418454 wikitext text/x-wiki n0onq83zrzpouv465ez5j6a2skuyenz 2418455 2418454 2026-05-22T09:38:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1172.eqiad.wmnet with OS trixie 2418455 wikitext text/x-wiki ng4tzbxrv0kej7ygqtowkvn8st1tm7c 2418456 2418455 2026-05-22T09:38:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2418456 wikitext text/x-wiki 8o8cc9kc06rd5bk0obvl7cleeibrsyq 2418457 2418456 2026-05-22T09:39:06Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2154: Upgrading db2154.codfw.wmnet 2418457 wikitext text/x-wiki d8fjc47m396jcdxuo17yauaka4ljc9q 2418458 2418457 2026-05-22T09:39:36Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2154: Upgrading db2154.codfw.wmnet 2418458 wikitext text/x-wiki sjagemeajrhwvhqfr6wzq4krxj4fsb5 2418459 2418458 2026-05-22T09:51:35Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage 2418459 wikitext text/x-wiki flhts58q25f049c1gmvft1k23dew3vg 2418460 2418459 2026-05-22T09:53:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage 2418460 wikitext text/x-wiki n468eduxkhxrjguc19xuvrvu7llr2uz 2418461 2418460 2026-05-22T09:55:30Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage 2418461 wikitext text/x-wiki 3nfjsawzrojn3vgmxtqre7ubug6qame 2418462 2418461 2026-05-22T09:56:22Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2154.codfw.wmnet with OS trixie 2418462 wikitext text/x-wiki 3r0b0xmzmkkjqkqi3gqoz768ssvsa5x 2418463 2418462 2026-05-22T09:59:34Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage 2418463 wikitext text/x-wiki sd2inrdzdvj3ca5ykqtr7hbzg6om2qy 2418464 2418463 2026-05-22T10:06:09Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet 2418464 wikitext text/x-wiki hmkr4djmr0fjziwfohgoqbgpqodlivm 2418465 2418464 2026-05-22T10:06:12Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp307[0-1].esams.wmnet} and A:cp 2418465 wikitext text/x-wiki b6ihjjoruhwf9cf30c0ljfbywpcn1l6 2418466 2418465 2026-05-22T10:07:53Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp307[8-9].esams.wmnet} and A:cp 2418466 wikitext text/x-wiki ozkqv8bfe8cqgg2k29dnr7qct14x8a7 2418467 2418466 2026-05-22T10:13:10Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2154.codfw.wmnet with reason: host reimage 2418467 wikitext text/x-wiki fo5cj97afi6yg2i9b7cozmhpthq5ffj 2418468 2418467 2026-05-22T10:15:38Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1017.eqiad.wmnet 2418468 wikitext text/x-wiki g9l6wbhkemvkhhcp1l41ls222gsw9yz 2418469 2418468 2026-05-22T10:16:07Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1172.eqiad.wmnet with OS trixie 2418469 wikitext text/x-wiki 40yld953jte9vsxoms6nen99g1mq11s 2418470 2418469 2026-05-22T10:18:24Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: host reimage 2418470 wikitext text/x-wiki l6l2rn9s7y68327qb7oz5qccco86wii 2418471 2418470 2026-05-22T10:19:38Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet 2418471 wikitext text/x-wiki pyss7ofk4oadi5nywuz5hch65x7cont 2418472 2418471 2026-05-22T10:24:18Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Migration of db1172.eqiad.wmnet completed 2418472 wikitext text/x-wiki ef05rm7fegpixcqt6nilajcgqi7p4oh 2418473 2418472 2026-05-22T10:26:59Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet 2418473 wikitext text/x-wiki 4qaeyhsahbfvlzwc462xymffgxrpsm1 2418474 2418473 2026-05-22T10:31:13Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie 2418474 wikitext text/x-wiki flw6t66cwluqg7xmj51a4klhg88f56b 2418475 2418474 2026-05-22T10:35:32Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2154.codfw.wmnet with OS trixie 2418475 wikitext text/x-wiki 2l08qu9ps8qvgmna485rzbjpmpy9xel 2418476 2418475 2026-05-22T10:37:34Z Stashbot 7414 moritzm: remove ganeti1024 foom eqiad Ganeti cluster T424680 2418476 wikitext text/x-wiki hd3k1et2qwpvrebdhb23jgs2h25sf0l 2418477 2418476 2026-05-22T10:41:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet 2418477 wikitext text/x-wiki 4wd96hkknf63rdrucsasswv8leinmex 2418478 2418477 2026-05-22T10:42:12Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2418478 wikitext text/x-wiki rv6t1ehckwk0ywo2bpk6afszz0lhzeg 2418479 2418478 2026-05-22T10:42:47Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2154: Migration of db2154.codfw.wmnet completed 2418479 wikitext text/x-wiki amrih7c97vz3s80akuhzjy94loev1ts 2418480 2418479 2026-05-22T10:42:50Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2418480 wikitext text/x-wiki 4k67uhwd53ly00s2wbstx8ghcydhcmh 2418481 2418480 2026-05-22T10:42:59Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2418481 wikitext text/x-wiki 9oubsn2ep8w0g9jio52oqb7ft6misz8 2418482 2418481 2026-05-22T10:43:16Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2418482 wikitext text/x-wiki jarv3reotkylmbjvx3e5u9bnwbg3kxj 2418483 2418482 2026-05-22T10:43:20Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2418483 wikitext text/x-wiki 9z78tvepipy1mu8nf9r9w6rb0ngvr14 2418484 2418483 2026-05-22T10:43:38Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2418484 wikitext text/x-wiki 3g3xgkgzno6ku2hoae5btcbs5el4cyw 2418485 2418484 2026-05-22T10:47:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet 2418485 wikitext text/x-wiki t01byqvccho9rax6l0dds4ca1bnltey 2418486 2418485 2026-05-22T10:47:52Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418486 wikitext text/x-wiki 3bc8l4igccn4ya8v0u6frgbc00fefxj 2418487 2418486 2026-05-22T10:48:13Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2418487 wikitext text/x-wiki asi76k00k4akkna2dvljklob7d77h0x 2418488 2418487 2026-05-22T10:55:15Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C 2418488 wikitext text/x-wiki 1486jmtb1hnxahqhaykctm9jekaftgz 2418489 2418488 2026-05-22T10:55:21Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C 2418489 wikitext text/x-wiki 5bdqcxahp3ccgo9kjjt31xbekpfpul6 2418490 2418489 2026-05-22T10:56:05Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet 2418490 wikitext text/x-wiki d9e6ra48uacb8xj8coplzbq8gb8suml 2418493 2418490 2026-05-22T11:01:08Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet 2418493 wikitext text/x-wiki mxtxmdz2vyw5d3nhzni4dgkwt7ubbqk 2418494 2418493 2026-05-22T11:01:11Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp307[8-9].esams.wmnet} and A:cp 2418494 wikitext text/x-wiki f8pirzi1htyunwpr1wo9t7i7i94523l 2418496 2418494 2026-05-22T11:03:50Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet 2418496 wikitext text/x-wiki ngu68tdnovmr1vjafmlugbrq5ycocyy 2418497 2418496 2026-05-22T11:07:12Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp307[2-3].esams.wmnet} and A:cp 2418497 wikitext text/x-wiki fr4hozy37e6d30h4gk53kd2ea01s8kw 2418500 2418497 2026-05-22T11:09:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Migration of db1172.eqiad.wmnet completed 2418500 wikitext text/x-wiki 83iys29f1soaka3ttu98jp6vopx73ez 2418501 2418500 2026-05-22T11:09:51Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2418501 wikitext text/x-wiki 4mstw3uag5y2eyf2hs8s4dimm8qhx4d 2418502 2418501 2026-05-22T11:11:09Z Stashbot 7414 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1017.eqiad.wmnet with reason: Rebooting clouddb1017 2418502 wikitext text/x-wiki froxi9m2e9ud637xzv51i6r6btvlvxc 2418503 2418502 2026-05-22T11:15:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C 2418503 wikitext text/x-wiki jvrx7p78t8izd2fmju9d3kv5886rvz2 2418504 2418503 2026-05-22T11:19:22Z Stashbot 7414 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet 2418504 wikitext text/x-wiki fgk4jlmjueql3oqhiosd7ti0nmyzfjn 2418505 2418504 2026-05-22T11:28:20Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Migration of db2154.codfw.wmnet completed 2418505 wikitext text/x-wiki dyp66xk6o2h0lpjaz3w79q9zpk9ecgl 2418506 2418505 2026-05-22T11:28:22Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2418506 wikitext text/x-wiki il9eh3za51x2gyrno09saygdtn60bbr 0 17290 2418232 2416128 2026-05-21T16:33:15Z Stashbot 7414 hashar: Reloaded Zuul to enable Node24 CI job for `labs/tools/wdaudiolex-fe` # T426366 2418232 wikitext text/x-wiki begpdv5ct9zk3x41m7w7h47hdsq97nl 498 30942 2417844 2417839 2026-05-21T11:59:55Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=0) on hosts matched by 'D{cloudlb2002-dev.codfw.wmnet}' 2417844 wikitext text/x-wiki jdmmn8990pcs62w9j08v49nq825356d 2418014 2417844 2026-05-21T13:28:43Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons (T426563) 2418014 wikitext text/x-wiki pfwcb1do0e0uvumgdqr2cucxq72j3cn 2418035 2418014 2026-05-21T13:41:11Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) (T426563) 2418035 wikitext text/x-wiki i4sxcml7kmro285i6gl7nnqvdtxhoj1 2418083 2418035 2026-05-21T14:16:48Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2418083 wikitext text/x-wiki 7zbrkes5ulg2q12vq2bvsivamlykt43 2418086 2418083 2026-05-21T14:17:06Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=97) 2418086 wikitext text/x-wiki ljmmlxyau75cygi4o1u7new5tapa11o 2418087 2418086 2026-05-21T14:17:24Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2418087 wikitext text/x-wiki jgxojlnfv6dc4vpp0koz502n2b4k879 2418103 2418087 2026-05-21T14:28:15Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) 2418103 wikitext text/x-wiki 938m55clzpav8fbg3key3gk29uag5iw 2418104 2418103 2026-05-21T14:29:07Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2418104 wikitext text/x-wiki q4qu896psm8pcdlsg6uo8mtlrfed39w 2418120 2418104 2026-05-21T14:39:58Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) 2418120 wikitext text/x-wiki ab7dchv375dp5499pgd1ft365yre1s5 2418297 2418120 2026-05-21T17:30:49Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2418297 wikitext text/x-wiki e6w1e8lo2235d367abds9wpm3stppif 2418309 2418297 2026-05-21T17:41:17Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) 2418309 wikitext text/x-wiki co7b99y6fd6q58lbt77j49wdbs3iusp 2418338 2418309 2026-05-21T18:06:25Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2418338 wikitext text/x-wiki gz6mljplxk8f6u064dwfg2v16loyk1i 2418339 2418338 2026-05-21T18:17:35Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) 2418339 wikitext text/x-wiki kk3kpkboavqkbz5ico8du4jsof8dcoq 2418340 2418339 2026-05-21T18:27:51Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons 2418340 wikitext text/x-wiki 2i5bczj8ys8dxv9l7t3lwzeirrsjyv4 2418341 2418340 2026-05-21T18:38:37Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) 2418341 wikitext text/x-wiki ajyrtlevhuobe7j37rfp4hwzg6mp555 498 443628 2418354 2415181 2026-05-21T20:39:24Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627543 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056) 2418354 wikitext text/x-wiki m45dbj3invjbgu3w531h9l7z1sri8z5 498 443946 2418349 2417356 2026-05-21T19:18:12Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed f0d7740038 (l10n updates: ko, ta, vi) 2418349 wikitext text/x-wiki 09c8598nght73z8vbdzu4rxh7qqs6bw 2418366 2418349 2026-05-21T21:45:27Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 97d0866c92 (treat outdated [OAuth 1] access tokens more robustly) 2418366 wikitext text/x-wiki gxmy8kwqvj4wvzut14nnm3w0fb05s0b 14 445064 2418425 1855109 2026-05-22T08:09:53Z Koavf 1382 new key for [[Category:Maps outages, 2018 and later]]: "2018" using [[w:Wikipedia:HotCat|HotCat]] 2418425 wikitext text/x-wiki m9dz7ezs0mwmc2i3z4rz15eeo4mp4dm 14 445065 2418427 1855112 2026-05-22T08:10:00Z Koavf 1382 new key for [[Category:Maps outages, 2018 and later]]: "2019" using [[w:Wikipedia:HotCat|HotCat]] 2418427 wikitext text/x-wiki q6rpuhqtsvyzg5jx120eafnj86y8evq 14 445068 2418424 1886961 2026-05-22T08:09:43Z Koavf 1382 new key for [[Category:Maps outages, 2018 and later]]: "2020" using [[w:Wikipedia:HotCat|HotCat]] 2418424 wikitext text/x-wiki 2lfz2nr7v0id2hmfuhr8udeo4maf58m 0 445656 2417916 2298554 2026-05-21T12:46:06Z Taavi-WMF 41365 /* Operations */ run as gitpuppet 2417916 wikitext text/x-wiki f9605ybxu7haa12b86nzb6m5uhsijvz 0 446037 2418158 2411046 2026-05-21T15:00:46Z MLechvien-WMF 49197 Adding Incident Lifecycle section and how to use the states during the incident 2418158 wikitext text/x-wiki t53ptm0nb2qa71y2ovx3g4ledb2cyip 2418161 2418158 2026-05-21T15:01:45Z MLechvien-WMF 49197 Adding links to subsections 2418161 wikitext text/x-wiki nwhe7x28t9isq9u4g3kmjvkccyw8ue7 2418356 2418161 2026-05-21T20:49:56Z A smart kitten 38172 /* To resolve the incident and stop being IC */ link the text representing the Phab tags to the Phab tags themselves (using [[phab:tag/...]] links); replace underscores in the display-text with spaces (now that the separation of the tag-names are [hopefully] clear from the linked pieces of text themselves). feel free to amend as desired :] 2418356 wikitext text/x-wiki pp4qd4t3rkpb5qr3x8cuut5rg0rhszj 0 446912 2418498 2355433 2026-05-22T11:08:23Z JMeybohm (WMF) 16709 Remove outdated information, add precision to some areas, add links to upstream policies (where available) 2418498 wikitext text/x-wiki s3mnyt1vymhnmv61ttn1o99a4z796l3 498 447034 2418350 2413849 2026-05-21T19:19:07Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 5a0564cc1a (l10n updates: tr) 2418350 wikitext text/x-wiki b77slv5u6zapmq2hyciqsmut83icpos 14 448849 2418422 1936909 2026-05-22T08:09:05Z Koavf 1382 added [[Category:SLOs]] using [[w:Wikipedia:HotCat|HotCat]] 2418422 wikitext text/x-wiki p8vp4q6w4qj6dn3rkzn17jhfoa36rnf 0 449160 2417889 2417728 2026-05-21T12:30:34Z Dexbot 30554 Bot: Updating the report 2417889 wikitext text/x-wiki mrcda8pfgvfoo5wxaf6fcffgo8d7xn3 2417940 2417889 2026-05-21T12:57:39Z Dexbot 30554 Bot: Updating the report 2417940 wikitext text/x-wiki dl6os59q8e48z27a3hamg46v1uxlkt5 2418074 2417940 2026-05-21T14:09:51Z Dexbot 30554 Bot: Updating the report 2418074 wikitext text/x-wiki dvu3r1vlg6xzdfb1dkm5q1pglcq32hh 2418224 2418074 2026-05-21T16:19:11Z Dexbot 30554 Bot: Updating the report 2418224 wikitext text/x-wiki 86v8wq4pbcc7th9irpfvb0m7g6uxev1 2418385 2418224 2026-05-22T00:02:09Z Dexbot 30554 Bot: Updating the report 2418385 wikitext text/x-wiki tks7aid1i3yg9xgvle0mix90pv7m4ev 0 452153 2418231 2414270 2026-05-21T16:32:37Z FNegri-WMF 32595 Add some missing Toolforge services 2418231 wikitext text/x-wiki qzb5qv0exeds4n2qicmc4hjo4hsd3mb 2418233 2418231 2026-05-21T16:33:55Z FNegri-WMF 32595 Add Help page link for Toolforge Redis 2418233 wikitext text/x-wiki mm08madhqxdf4f2cphjhclqbhn29x22 2418370 2418233 2026-05-21T21:58:13Z BLiviero-WMF 53603 Added owner column 2418370 wikitext text/x-wiki euges619jf517lw9i9i3vrlyypni6vr 2418371 2418370 2026-05-21T22:01:01Z BLiviero-WMF 53603 2418371 wikitext text/x-wiki jnqf50ec3i95yp3yypde6ezo1unhgjs 498 452890 2418355 2415182 2026-05-21T20:40:01Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627549 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056) 2418355 wikitext text/x-wiki gg6r9msu04ysxaojm968onspqmcrbou 5 453462 2418177 2398285 2026-05-21T15:15:08Z Valerio Bozzolan 10109 /* Open Source License */ Reply 2418177 wikitext text/x-wiki phkvxa1luurs3e3w71wd2fo8l13mvoy 2418179 2418177 2026-05-21T15:16:02Z Valerio Bozzolan 10109 /* Open Source License */ big typo 2418179 wikitext text/x-wiki f4nvq3bdrls2jihz3u7u2129l6hd5cj 0 453476 2418015 2111331 2026-05-21T13:29:27Z JMonton-WMF 47887 fix link 2418015 wikitext text/x-wiki ngpnqezorlkvaw5bgwgy9wtl3lrk90x 116 453906 2418217 2416335 2026-05-21T16:06:24Z Gitlabaccountapprovalbot 37332 @gkm563 was approved. 2418217 wikitext text/x-wiki by8xaad2gb3exa9f8x3lve2v4si1kml 2418230 2418217 2026-05-21T16:27:26Z Gitlabaccountapprovalbot 37332 @renovatebot was approved. 2418230 wikitext text/x-wiki naf623qdcssbux4e0hgxuvkmvnv979h 2418344 2418230 2026-05-21T18:51:16Z Gitlabaccountapprovalbot 37332 kartikeyg0104 was rejected. 2418344 wikitext text/x-wiki h2nmlkz1stynj17ucxnq988w3c5e2p3 2418386 2418344 2026-05-22T01:48:19Z Gitlabaccountapprovalbot 37332 50323233 was rejected. 2418386 wikitext text/x-wiki 5teuxe9zwlj3vfu8of7binex6zmv1kh 0 454457 2418299 2383076 2026-05-21T17:32:45Z MPopov (WMF) 5891 /* Naming and data type conventions */ Added user auth status labels 2418299 wikitext text/x-wiki jqd6uzd0yv7ihpytzhtmlb5wt3fve3d 14 455908 2418423 2233830 2026-05-22T08:09:08Z Koavf 1382 added [[Category:SLOs]] using [[w:Wikipedia:HotCat|HotCat]] 2418423 wikitext text/x-wiki 295sg7sls3n6ai7nvz8psz9iszoyn1n 0 455989 2418374 2410683 2026-05-21T22:07:46Z SFaci-WMF 35637 /* Experiment design: user traffic per wiki */ 2418374 wikitext text/x-wiki 8z44w5p2vnhxjwguqlwtt7w1ah176jo 2418376 2418374 2026-05-21T22:13:50Z SFaci-WMF 35637 /* Experiment design: user traffic per wiki */ 2418376 wikitext text/x-wiki 6uo86toza8wslmpzhvzyuuk07utz0bs 2418383 2418376 2026-05-21T23:40:23Z CMing (WMF) 23277 /* Experiment design: user traffic per wiki */ 2418383 wikitext text/x-wiki 4z9biqmh25np9qpsf7ailfe7jp2w1md 0 459160 2418491 2417170 2026-05-22T10:59:14Z DAdedoyin (WMF) 39150 Added image of donor portal architecture 2418491 wikitext text/x-wiki 9rx3kig22q37hyps5swc5fygmbjqwj3 2418492 2418491 2026-05-22T11:00:37Z DAdedoyin (WMF) 39150 /* Donor Portal */ 2418492 wikitext text/x-wiki dvnba462m4zxkrkr9c2jak41ujbr22o 2418495 2418492 2026-05-22T11:02:47Z DAdedoyin (WMF) 39150 /* Contact Details */ 2418495 wikitext text/x-wiki jpd5lhyuupq4kp6y4z7r0eydacoui81 498 459243 2418365 2415209 2026-05-21T21:44:34Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254787328 (https://github.com/cluebotng/component-configs/commits/80d9748516b3523cecfc76bd85d662fe91dde0c4) 2418365 wikitext text/x-wiki f4e64cedfpvfciu3ptstj0kl57b0r01 2418367 2418365 2026-05-21T21:48:38Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254813158 (https://github.com/cluebotng/component-configs/commits/ebf46d18f22eee769feeeffd6f4603ec4f320bbb) 2418367 wikitext text/x-wiki 2dnd8htn6qwypdx9wj4zd13hz5bwrhi 2418378 2418367 2026-05-21T22:33:34Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256861316 (https://github.com/cluebotng/component-configs/commits/e4c68154f62c8846320103abb0e97af63b20f9b1) 2418378 wikitext text/x-wiki 54qvrm3r4e9spdta1ezqprlpgrfn0y7 2418379 2418378 2026-05-21T22:37:19Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256956868 (https://github.com/cluebotng/component-configs/commits/95f311a49172258bfd4d1c00d74a918a7aceab68) 2418379 wikitext text/x-wiki ngqd764x825vczjn0saghob9xfiyg12 498 459244 2418285 2415195 2026-05-21T17:11:39Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26241143565 (https://github.com/cluebotng/component-configs/commits/76187b1d77a1454297932d833d668fafc6490c10) 2418285 wikitext text/x-wiki h5qyug3h7ma9bg1shqlmvqleww56tpo 498 460213 2418225 2026-05-21T16:20:27Z Labslogbot 55 Auto update of instance info. 2418225 wikitext text/x-wiki 3w4p8f7dkjv8jiiyd1ylccprgcg377r 498 460214 2418226 2026-05-21T16:21:21Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project wiki-polis-backend in eqiad1 2418226 wikitext text/x-wiki b9indjcyc6ft33wti6xoyo9tbz4jyzx 0 460215 2418346 2026-05-21T18:55:33Z MPopov (WMF) 5891 Started drafting decision record 2418346 wikitext text/x-wiki be9kmr1yxsblboeztzalx7kq3gy6okt 2418363 2418346 2026-05-21T21:33:14Z MPopov (WMF) 5891 Finalize the decision 2418363 wikitext text/x-wiki 9xgax4i68ppx7t7nwn0wwsvjqdkk6om