Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.6 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 0 4108 2424757 2424728 2026-06-09T12:17:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1298654]] to Tuesday, June 09 UTC afternoon backport window 2424757 wikitext text/x-wiki qbr4qz2nnrzkw5xbwcfycv5gaeh3rqv 2424775 2424757 2026-06-09T12:25:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1299477]] to Tuesday, June 09 UTC afternoon backport window 2424775 wikitext text/x-wiki bm7risx8ofv1h1ae84bb5v297ol9waa 2424801 2424775 2026-06-09T12:43:25Z Dmitry Brant 38163 already done 2424801 wikitext text/x-wiki 49fh10joc4j1by15yw6rrb59sb0aze0 2424809 2424801 2026-06-09T12:49:44Z Revi C. 1919 /* {{Deployment_day|date=2026-06-09}} */ title 2424809 wikitext text/x-wiki q75wi2cz4u0rwupc9qe0kiwtvckw877 2424825 2424809 2026-06-09T13:05:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1299497]] to Tuesday, June 09 UTC afternoon backport window 2424825 wikitext text/x-wiki h7vpss6i7b83c6989i72q8rkhtfhzrs 2424862 2424825 2026-06-09T13:38:04Z ScheduleDeploymentBot 37566 Add [[gerrit:1299495]] to Tuesday, June 09 UTC afternoon backport window 2424862 wikitext text/x-wiki 2zp3zrdbdvulhe35z2t33bu4a4zmxei 2424863 2424862 2026-06-09T13:38:30Z ScheduleDeploymentBot 37566 Add [[gerrit:1299502]] to Tuesday, June 09 UTC afternoon backport window 2424863 wikitext text/x-wiki rysfvkmucirw0bovks67v8i3lnc57lw 2424954 2424863 2026-06-09T15:50:25Z ScheduleDeploymentBot 37566 Add [[gerrit:1299454]] to Tuesday, June 09 UTC late backport window 2424954 wikitext text/x-wiki adr6m4ra7urd4swx5r9was08ttsp2yn 2424981 2424954 2026-06-09T16:18:17Z ScheduleDeploymentBot 37566 Add [[gerrit:1299556]] to Wednesday, June 10 UTC morning backport window 2424981 wikitext text/x-wiki gflms3tuo7iumq36n0p73t2cdze9pq1 2424982 2424981 2026-06-09T16:18:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1299556]] to Wednesday, June 10 UTC morning backport window 2424982 wikitext text/x-wiki fyn6bfhtg9fs8zuf6xtam22qu2dcnsw 2424985 2424982 2026-06-09T16:19:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1299561]] to Wednesday, June 10 UTC morning backport window 2424985 wikitext text/x-wiki qdo82iqfgpeua7hntwliwvwntrhjpiw 2424987 2424985 2026-06-09T16:20:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1299529]] to Wednesday, June 10 UTC morning backport window 2424987 wikitext text/x-wiki lpj9a9y2uxbf1jtar08le6tk57l4kux 2424990 2424987 2026-06-09T16:23:45Z Atsuko-WMF 54082 removing duplicate 2424990 wikitext text/x-wiki i6das5pkxhalh3sk9pyflugfod8tj1c 2425053 2424990 2026-06-09T18:26:44Z ScheduleDeploymentBot 37566 Add [[gerrit:1299589]] to Tuesday, June 09 UTC late backport window 2425053 wikitext text/x-wiki j1pg354awt7dsxh94df39avydda2rks 2425054 2425053 2026-06-09T18:27:13Z ScheduleDeploymentBot 37566 Add [[gerrit:1299588]] to Tuesday, June 09 UTC late backport window 2425054 wikitext text/x-wiki tf5s0ib0ybh07boibpte7uptbnbal6t 2425255 2425054 2026-06-10T05:18:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1299676]] to Wednesday, June 10 UTC afternoon backport window 2425255 wikitext text/x-wiki lttcnp97x80zyurva4uq2ull7urgm1b 0 6479 2425007 2424667 2026-06-09T16:40:28Z Effie Mouzeli (WMF) 12880 scrollbot update 2425007 wikitext text/x-wiki nw5e875oxb2sxd1winys2jqa4mldbb7 0 7919 2424745 2424743 2026-06-09T12:00:40Z Stashbot 7414 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dmaza out of all services on: 2435 hosts 2424745 wikitext text/x-wiki moqbvg7pwm0qk2lnuus1no54b0rvcyl 2424746 2424745 2026-06-09T12:05:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage 2424746 wikitext text/x-wiki 3zxwqij6d5xhuvb3f08dyt9p2qvpa7q 2424747 2424746 2026-06-09T12:07:29Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298829|wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] 2424747 wikitext text/x-wiki qqisy5sq6suq0sxp2dau52zv8apng56 2424748 2424747 2026-06-09T12:10:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage 2424748 wikitext text/x-wiki l3sscvu3swhicl5a01qo729fboyfnuc 2424751 2424748 2026-06-09T12:13:30Z Stashbot 7414 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1298829|wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2424751 wikitext text/x-wiki 0pifc3wh2hou4p6mbllfdic2driu47k 2424752 2424751 2026-06-09T12:14:02Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 2424752 wikitext text/x-wiki hcmt6vsicuxypogxv8d20yjs37pgmpx 2424753 2424752 2026-06-09T12:15:29Z Stashbot 7414 topranks: drain traffic on ssw1-a1-codfw - add gshut community in evpn underlay - T427357 2424753 wikitext text/x-wiki 5979baw61rpuuchzdd6mandrw5vz29o 2424754 2424753 2026-06-09T12:16:59Z Stashbot 7414 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment 2424754 wikitext text/x-wiki 0b0eweqvla9ith901zimkc56nl0dc7t 2424755 2424754 2026-06-09T12:17:01Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool 2424755 wikitext text/x-wiki mr7tn453v1ghjy5tke78n7yws2th0jj 2424756 2424755 2026-06-09T12:17:20Z Stashbot 7414 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 2424756 wikitext text/x-wiki 3b2isqh3rsfpbcse5qezbguvmtcik89 2424758 2424756 2026-06-09T12:19:17Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1015.eqiad.wmnet with OS trixie 2424758 wikitext text/x-wiki 6gtsiph413e2bkiwt4mmdij1zytls4h 2424759 2424758 2026-06-09T12:19:32Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1016.eqiad.wmnet with OS trixie 2424759 wikitext text/x-wiki bce7fdcnz3vy3bte5mkfb7wqn7yo4qb 2424760 2424759 2026-06-09T12:19:35Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1015 2424760 wikitext text/x-wiki 2xakbg2t99xj13mrbpfm0xpl8xx2xg2 2424761 2424760 2026-06-09T12:19:37Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1015 2424761 wikitext text/x-wiki 7j4806z6oc22ntv5a6801vz09cdgytk 2424762 2424761 2026-06-09T12:19:52Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1016 2424762 wikitext text/x-wiki t8qmfu8c9nshkyytsuml4gat18wh74y 2424763 2424762 2026-06-09T12:19:55Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1016 2424763 wikitext text/x-wiki r8wqi4vc0113mb81l5bprejivouie9z 2424764 2424763 2026-06-09T12:20:02Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2241: rack depool 2424764 wikitext text/x-wiki 0ecmpr8u4ow44jnpq1ojwonot53l7t3 2424765 2424764 2026-06-09T12:20:13Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2424765 wikitext text/x-wiki moexu04ceg4iowf6extelnphz663s1r 2424766 2424765 2026-06-09T12:20:22Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool 2424766 wikitext text/x-wiki jjq3yjm8ikqq5maubp7e06axh94y1o0 2424767 2424766 2026-06-09T12:20:38Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2424767 wikitext text/x-wiki rviptizfoq1ebpdbo1ot1ifufa8198h 2424768 2424767 2026-06-09T12:20:47Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet 2424768 wikitext text/x-wiki 6awcdab5df6ljpt67ahdfzdigldu17g 2424769 2424768 2026-06-09T12:22:01Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet 2424769 wikitext text/x-wiki h96kujp16zm07uelasan92f9u4m56b4 2424770 2424769 2026-06-09T12:22:33Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet 2424770 wikitext text/x-wiki 24alupjnk6wi5tcvbc8bzv0q3v0ewys 2424771 2424770 2026-06-09T12:23:28Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet 2424771 wikitext text/x-wiki hwae7ci1noeg4gxm3ue818iw1e1k4tm 2424772 2424771 2026-06-09T12:23:34Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298829|wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] (duration: 16m 04s) 2424772 wikitext text/x-wiki hh8c5z8iyjqc8d7mq3q3gh11h0z8ti6 2424773 2424772 2026-06-09T12:24:05Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet 2424773 wikitext text/x-wiki hqg1iizatwfs08jfz6xcyhgfge0229f 2424774 2424773 2026-06-09T12:24:25Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2046: repool after maintenance 2424774 wikitext text/x-wiki 5ves0nkxw44po2r91pgp1a0vjl57699 2424776 2424774 2026-06-09T12:27:08Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet 2424776 wikitext text/x-wiki mj3tqyq5xdakfxvzp05huukoarioti4 2424777 2424776 2026-06-09T12:29:05Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2424777 wikitext text/x-wiki au4bf0j4ijo6w1i3ezc1towiucx8sn0 2424778 2424777 2026-06-09T12:29:15Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2424778 wikitext text/x-wiki owf92dz9xjxvhco1yxn8co7v022uz07 2424779 2424778 2026-06-09T12:32:06Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage 2424779 wikitext text/x-wiki 0zs1kjhtbj1min3f0ovtg3f4dhggwsi 2424780 2424779 2026-06-09T12:32:22Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage 2424780 wikitext text/x-wiki s7k44kvayzhgqumpftlq148ubsz8xq2 2424781 2424780 2026-06-09T12:32:55Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1237.eqiad.wmnet with OS trixie 2424781 wikitext text/x-wiki qwetg5jwrps3evglckt3iisx3pqnvpk 2424782 2424781 2026-06-09T12:33:16Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2424782 wikitext text/x-wiki 0yypu93qf7uyby1cqxu8t28ylgnseus 2424783 2424782 2026-06-09T12:33:36Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet 2424783 wikitext text/x-wiki notgxlbfi5l8vimcxlbt7svmtlr6z3u 2424784 2424783 2026-06-09T12:33:55Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet 2424784 wikitext text/x-wiki h2k7gqkxp243dmv2n57j0f3xwl9s9ge 2424785 2424784 2026-06-09T12:34:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2424785 wikitext text/x-wiki lh89xrucb6unz4i0bet4v0ssiyn9l68 2424786 2424785 2026-06-09T12:34:32Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet 2424786 wikitext text/x-wiki 43qmf29jo7kzieiirpubzro1c65r9o0 2424787 2424786 2026-06-09T12:34:40Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299478|STVFormatter: Cast strings to float before passing to round (T428584)]] 2424787 wikitext text/x-wiki 32kdm0pn8bw81i2csp3kg90gv5zvovy 2424788 2424787 2026-06-09T12:34:52Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1237: repool after maintenance db1237 2424788 wikitext text/x-wiki o0xt0254wabsjw8f4fvoq3avol62v8h 2424789 2424788 2026-06-09T12:34:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet 2424789 wikitext text/x-wiki bgxgg80u6humaarsc41r8baosp19qoi 2424791 2424789 2026-06-09T12:36:41Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299478|STVFormatter: Cast strings to float before passing to round (T428584)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2424791 wikitext text/x-wiki 38iv73low48cm9wwi8w1sw4t5b45efm 2424792 2424791 2026-06-09T12:37:21Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2424792 wikitext text/x-wiki npnw51wx9a5s2dyn1fbqp3yojph8e38 2424793 2424792 2026-06-09T12:38:30Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS trixie 2424793 wikitext text/x-wiki or44findy752p99i8tyjil0j8igz42d 2424796 2424793 2026-06-09T12:40:15Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2153.codfw.wmnet with OS trixie 2424796 wikitext text/x-wiki 8cgjthlvj15vcuyglkq8f3llg2dtuqu 2424797 2424796 2026-06-09T12:40:31Z Stashbot 7414 moritzm: installing wireshark security updates 2424797 wikitext text/x-wiki 3ybr5y32yjvb1r47rildeamacdvkbpr 2424798 2424797 2026-06-09T12:40:33Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage 2424798 wikitext text/x-wiki fsmrdhbfy03kxw1wfzh0xlch6vp4p74 2424799 2424798 2026-06-09T12:41:42Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299478|STVFormatter: Cast strings to float before passing to round (T428584)]] (duration: 07m 02s) 2424799 wikitext text/x-wiki ske3wjmd3jqgdja9s48hhy1nu55a5m4 2424800 2424799 2026-06-09T12:42:46Z Stashbot 7414 topranks: increase OSPF cost on ssw1-a1-codfw link to lsw1-a4-codfw to force traffic via alternate spine T427357 2424800 wikitext text/x-wiki 8d7yrdug15fhvch7b8r3eb0xm813lar 2424802 2424800 2026-06-09T12:43:28Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1299477|hCaptcha: Roll out to all wikis for api account creation. (T426050)]] 2424802 wikitext text/x-wiki faq47mbg2k5f301ff8n65zilidfm5ae 2424803 2424802 2026-06-09T12:45:10Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 2424803 wikitext text/x-wiki pjxmdetmennkibmdcs4dse6ai9zwyyw 2424804 2424803 2026-06-09T12:45:21Z Stashbot 7414 topranks: shut sub-interfaces for row A/B legacy vlans on cr1-codfw T427357 2424804 wikitext text/x-wiki nc0ps5e3jgu2tv0vmhbw0p2xzwms1lp 2424805 2424804 2026-06-09T12:45:31Z Stashbot 7414 kharlan@deploy1003: kharlan, dbrant: Backport for [[gerrit:1299477|hCaptcha: Roll out to all wikis for api account creation. (T426050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2424805 wikitext text/x-wiki 2u5okciijuyp73mdci0p7ux86trho0f 2424806 2424805 2026-06-09T12:45:46Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage 2424806 wikitext text/x-wiki gipe5t2ao64d7m3gt455fdrkqml03sb 2424807 2424806 2026-06-09T12:46:10Z Stashbot 7414 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 2424807 wikitext text/x-wiki 3ojvwc133dclmnq5m3i1zm61u1k9cfg 2424808 2424807 2026-06-09T12:46:29Z Stashbot 7414 kharlan@deploy1003: kharlan, dbrant: Continuing with deployment 2424808 wikitext text/x-wiki hnwz4v8mfszbh1q7j8k64pn7itt9em2 2424810 2424808 2026-06-09T12:50:50Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299477|hCaptcha: Roll out to all wikis for api account creation. (T426050)]] (duration: 07m 21s) 2424810 wikitext text/x-wiki 6x3vzfsmzrzdhskhjjj893uufol1682 2424812 2424810 2026-06-09T12:53:42Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage 2424812 wikitext text/x-wiki q8dhzj6qh2x0tq9or85nbxnz3x1uw8t 2424813 2424812 2026-06-09T12:55:41Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2424813 wikitext text/x-wiki 7ue4l4o3wk7hvrywflmu4r8u61uufhr 2424814 2424813 2026-06-09T12:56:49Z Stashbot 7414 XioNoX: lsw1-a4-codfw> request system reboot - T427357 2424814 wikitext text/x-wiki papmmct3i7ua2ux1vhl3elzp31cq1ts 2424815 2424814 2026-06-09T12:57:10Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 2424815 wikitext text/x-wiki g4wvieztgwtvqgodhjdphi9bladouvs 2424816 2424815 2026-06-09T12:57:34Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1016.eqiad.wmnet with OS trixie 2424816 wikitext text/x-wiki lr0dkfu686yk4t5dhpfefpcbinvslxn 2424817 2424816 2026-06-09T12:58:38Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2153.codfw.wmnet with reason: host reimage 2424817 wikitext text/x-wiki hajemi5hmzwkmajic2033ukwm4rdjmu 2424818 2424817 2026-06-09T12:59:46Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage 2424818 wikitext text/x-wiki 9klydzh3jyd7agkf77hvqzyyerluhsx 2424819 2424818 2026-06-09T13:03:03Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1015.eqiad.wmnet with OS trixie 2424819 wikitext text/x-wiki ajbrogdn1cv5gqrvpu6wnq9yfocutdr 2424820 2424819 2026-06-09T13:04:15Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2424820 wikitext text/x-wiki lsokdqrc20k6tkxa2gznhepfn0l0nuw 2424821 2424820 2026-06-09T13:04:19Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2424821 wikitext text/x-wiki kwlry11rsdxw9lue1s2out5ag9jk8zy 2424822 2424821 2026-06-09T13:04:21Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2153.codfw.wmnet with reason: host reimage 2424822 wikitext text/x-wiki mfya3nk2uu3f5wz6o944bm8m3fy1tb7 2424823 2424822 2026-06-09T13:04:44Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2424823 wikitext text/x-wiki rproiewxakdkgij2glnt40prl5pm5jz 2424824 2424823 2026-06-09T13:04:52Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2424824 wikitext text/x-wiki g3zhty9h08l18nzda5abm7023scl58t 2424826 2424824 2026-06-09T13:09:21Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298654|Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] 2424826 wikitext text/x-wiki nwu14fdkt48cvysni1t1wp4fpheymqq 2424827 2424826 2026-06-09T13:11:30Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Backport for [[gerrit:1298654|Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2424827 wikitext text/x-wiki sgfhuiv9x2w7z8xrkxgvgzylio4ky2j 2424828 2424827 2026-06-09T13:11:54Z Stashbot 7414 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 2424828 wikitext text/x-wiki 3tp8pjbmgmlptwu8hv2nt2mlf4d4cuv 2424829 2424828 2026-06-09T13:14:39Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Continuing with deployment 2424829 wikitext text/x-wiki 5polubfqn4og0q1rv4zsx1p0j7k2b2m 2424830 2424829 2026-06-09T13:15:57Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS trixie 2424830 wikitext text/x-wiki bo0dclns3iphtumsmc103yxz0la1dc6 2424831 2424830 2026-06-09T13:16:29Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet 2424831 wikitext text/x-wiki av6tmwuw3vi8s8mafv2xxzt1bdx97g7 2424832 2424831 2026-06-09T13:16:32Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet 2424832 wikitext text/x-wiki 6y14bd2gbcq5qadc64scqzb6i0ranlr 2424833 2424832 2026-06-09T13:16:49Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet 2424833 wikitext text/x-wiki qrpvw1zgt6jxggxpsudxyzbxuow4un8 2424834 2424833 2026-06-09T13:16:52Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet 2424834 wikitext text/x-wiki 0qagii3lik0l94u2z60wycw5ayfdbqk 2424835 2424834 2026-06-09T13:17:05Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet 2424835 wikitext text/x-wiki 2niumo9lh1tqnuq4072ep9k0i5x0w84 2424836 2424835 2026-06-09T13:17:09Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet 2424836 wikitext text/x-wiki mxd2yd668xaqh4anezpfpp4etqbn4g6 2424837 2424836 2026-06-09T13:19:02Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298654|Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] (duration: 09m 40s) 2424837 wikitext text/x-wiki 1nvqme59icsdmfoxy1kaiyu4ujc4tsg 2424838 2424837 2026-06-09T13:20:17Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1237: repool after maintenance db1237 2424838 wikitext text/x-wiki gpoir0cepbf8dphdcxtqw88lohhv9mj 2424839 2424838 2026-06-09T13:20:20Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2241: rack depool 2424839 wikitext text/x-wiki ojpylupvj7na9x0a4gvl96dsiwkrzhi 2424841 2424839 2026-06-09T13:20:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2153.codfw.wmnet with OS trixie 2424841 wikitext text/x-wiki ejrc00zv9qdopr6rqnbhwzv8cq7t21h 2424842 2424841 2026-06-09T13:21:19Z Stashbot 7414 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply 2424842 wikitext text/x-wiki 2u72hr32g4hqsxqy3yet8slxwfgx249 2424843 2424842 2026-06-09T13:21:58Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2424843 wikitext text/x-wiki 4l48el9cs6c0rrjfqj075hyt6xwkdfh 2424844 2424843 2026-06-09T13:23:23Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2424844 wikitext text/x-wiki okuigwc6h18z3exj359sbepibzsmyxi 2424845 2424844 2026-06-09T13:24:09Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2424845 wikitext text/x-wiki 8swgr3j2cqxib8ikhxc563la4p3ajxg 2424846 2424845 2026-06-09T13:25:21Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2424846 wikitext text/x-wiki oegeui9a8yosm3bx33wvrgra0ap1fx6 2424847 2424846 2026-06-09T13:25:31Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298834|config: Disable EmailConfirmationBanner on all wikis (T428291)]] 2424847 wikitext text/x-wiki 4bvy8499hvx3pz501qzjgby52a5bof9 2424848 2424847 2026-06-09T13:26:46Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1184: Migration of db1184.eqiad.wmnet completed 2424848 wikitext text/x-wiki 2yb6wzka4hikalrgxrrn9k4ez92kwdm 2424849 2424848 2026-06-09T13:27:32Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Backport for [[gerrit:1298834|config: Disable EmailConfirmationBanner on all wikis (T428291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2424849 wikitext text/x-wiki 7s9rzgrti0l11c5s70zqh3gsangfrq0 2424850 2424849 2026-06-09T13:28:12Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Continuing with deployment 2424850 wikitext text/x-wiki 2ph1zh4hykd84eabedsac13bwwek657 2424851 2424850 2026-06-09T13:28:19Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2424851 wikitext text/x-wiki 87fzbpov4rjs5ietjpjzf7bzqromwdx 2424852 2424851 2026-06-09T13:28:24Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2424852 wikitext text/x-wiki pz87vlweh1k2ulkzr2kw2p9yqy38ixi 2424853 2424852 2026-06-09T13:28:29Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2424853 wikitext text/x-wiki bo69ed1czq06zumyqpglfn64ol8yc86 2424854 2424853 2026-06-09T13:28:36Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2424854 wikitext text/x-wiki j4u4zuvfhjwu1pffjxxlm6xm2th6jsv 2424855 2424854 2026-06-09T13:30:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Migration of db2153.codfw.wmnet completed 2424855 wikitext text/x-wiki aydjljrt392smsclvbo8chy3moo2qv0 2424856 2424855 2026-06-09T13:32:32Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298834|config: Disable EmailConfirmationBanner on all wikis (T428291)]] (duration: 07m 01s) 2424856 wikitext text/x-wiki nzwtw7zw3nyciyojwom5gbui3ff5lj9 2424857 2424856 2026-06-09T13:32:59Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2424857 wikitext text/x-wiki lugmp26w23hjh2ph0nnd1u7q4z44gu6 2424858 2424857 2026-06-09T13:33:55Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2424858 wikitext text/x-wiki sevzk5ijm5bxnoqe0s6rbee4w1zcfx7 2424859 2424858 2026-06-09T13:35:19Z Stashbot 7414 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply 2424859 wikitext text/x-wiki gpjm5uwdangtlvmbqh6hmwi84ruw8c2 2424861 2424859 2026-06-09T13:37:57Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply 2424861 wikitext text/x-wiki rp3k5sho7wsn6thgdrpxs5etkktxnoi 2424864 2424861 2026-06-09T13:39:08Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply 2424864 wikitext text/x-wiki iw7546qe472ikx9s5l0occwxuh4fqkd 2424865 2424864 2026-06-09T13:40:04Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw 2424865 wikitext text/x-wiki bwb96un1pesb19g6xy1e6jglg4vyqxt 2424866 2424865 2026-06-09T13:40:16Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1298929|Simplify fragment processing (T423700)]], [[gerrit:1298926|Move ::getFragmentsToTransform() to Content{Text,DOM}TransformStage]], [[gerrit:1298927|OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925|Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497| 2424866 wikitext text/x-wiki 3fjd5atgsx4d6x1bqrpwhhtvdtv46z0 2424867 2424866 2026-06-09T13:41:25Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply 2424867 wikitext text/x-wiki s0agxokno2tjuvxma9qtqd3kkjde2ul 2424868 2424867 2026-06-09T13:42:17Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1298929|Simplify fragment processing (T423700)]], [[gerrit:1298926|Move ::getFragmentsToTransform() to Content{Text,DOM}TransformStage]], [[gerrit:1298927|OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925|Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497|Store indicators 2424868 wikitext text/x-wiki brbbgz1vqibpi2lzt0xbua19ioxaeeh 2424869 2424868 2026-06-09T13:44:23Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply 2424869 wikitext text/x-wiki mmxeefmlszlxsg47gxu32lqisnyx8qc 2424870 2424869 2026-06-09T13:45:38Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2424870 wikitext text/x-wiki 6qtq6c611f7zq7vo8cw55tjm7y2qn22 2424871 2424870 2026-06-09T13:46:09Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2424871 wikitext text/x-wiki gmt6y1yng17dxlrzg2p9cvxjg6g2x54 2424872 2424871 2026-06-09T13:47:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A 2424872 wikitext text/x-wiki jwmcn8rqfphv4gtg2p5fjmz544itlug 2424873 2424872 2026-06-09T13:48:40Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2027.codfw.wmnet to cluster codfw and group A 2424873 wikitext text/x-wiki llvbnd2fj1wf16k3vlrvo97v7110xc3 2424874 2424873 2026-06-09T13:48:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2045.codfw.wmnet to cluster codfw and group A 2424874 wikitext text/x-wiki iyjm6jrby20l1aab37lrzmb9jc34ql2 2424875 2424874 2026-06-09T13:50:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2045.codfw.wmnet to cluster codfw and group A 2424875 wikitext text/x-wiki 0qcm952v9eko4tgpa6pu280whlasnfo 2424876 2424875 2026-06-09T13:50:47Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2424876 wikitext text/x-wiki mxlo8rzqatvc4gk9m6awjemmphx035e 2424877 2424876 2026-06-09T13:51:06Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to drbd 2424877 wikitext text/x-wiki mu4yosaaqtagb771n4ybv61i63mrel6 2424878 2424877 2026-06-09T13:52:23Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply 2424878 wikitext text/x-wiki f0gynfb97clgw29h2yhle5z7l9ixwc8 2424879 2424878 2026-06-09T13:52:28Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply 2424879 wikitext text/x-wiki jsimhu87hxy6ty6uumur0u7gvnjyudr 2424880 2424879 2026-06-09T13:55:06Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298929|Simplify fragment processing (T423700)]], [[gerrit:1298926|Move ::getFragmentsToTransform() to Content{Text,DOM}TransformStage]], [[gerrit:1298927|OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925|Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497 2424880 wikitext text/x-wiki qothcuoevn2x50n1hojuxmmwricaylu 2424881 2424880 2026-06-09T13:55:19Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply 2424881 wikitext text/x-wiki oq8ykecv43lnipipihkj3bs0bay3d33 2424882 2424881 2026-06-09T13:55:26Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply 2424882 wikitext text/x-wiki hec363t9hdo6sojljxy65935zp2rb6z 2424883 2424882 2026-06-09T13:56:03Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2424883 wikitext text/x-wiki 2a32p3cl96fjdby67uhcqfjs4as0lgg 2424884 2424883 2026-06-09T13:56:28Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2424884 wikitext text/x-wiki l8dmsq10ktv3htl37zi4ojr00kva29y 2424885 2424884 2026-06-09T13:56:53Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2424885 wikitext text/x-wiki gzxbss1bmw43cd6u2rr0ijkcmhotqpt 2424886 2424885 2026-06-09T13:56:57Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2424886 wikitext text/x-wiki 6d39j17si7kiptbqoj44lievrkwfeaq 2424887 2424886 2026-06-09T13:58:40Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2424887 wikitext text/x-wiki 3q8v2ij91oo74snmaopeyrvemo5x2ly 2424888 2424887 2026-06-09T13:58:47Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2424888 wikitext text/x-wiki se07o68p5ntu23xoegj8uro1i7f089l 2424889 2424888 2026-06-09T13:59:59Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299495|STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502|SecurePollLogPager: Cast user IDs to ints before use (T428599)]] 2424889 wikitext text/x-wiki b59vwtcpu23oe3jr1wbpsdyn3y321wn 2424891 2424889 2026-06-09T14:02:10Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299495|STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502|SecurePollLogPager: Cast user IDs to ints before use (T428599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2424891 wikitext text/x-wiki 6dkzm53ua9a4bflgy3deycsnbpy7kgu 2424892 2424891 2026-06-09T14:02:32Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2424892 wikitext text/x-wiki s6a90m637uotg812elp92aaqm6xyqn2 2424893 2424892 2026-06-09T14:02:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to drbd 2424893 wikitext text/x-wiki a7pgqbc34zgj5waipikkhuzr7jw4fll 2424895 2424893 2026-06-09T14:03:46Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to drbd 2424895 wikitext text/x-wiki 6hepd9uzggpttf06d78xzi3e8wilba2 2424896 2424895 2026-06-09T14:06:08Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool 2424896 wikitext text/x-wiki m1z64t3rwv243x5xeme99ph981orn6d 2424897 2424896 2026-06-09T14:06:31Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2424897 wikitext text/x-wiki 6me96lur6ixau1hesm71b2uj0fs4tor 2424898 2424897 2026-06-09T14:06:53Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299495|STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502|SecurePollLogPager: Cast user IDs to ints before use (T428599)]] (duration: 06m 53s) 2424898 wikitext text/x-wiki eq6l223a4yl1yupmcw1lyzgrrt6us8u 2424899 2424898 2026-06-09T14:07:03Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2424899 wikitext text/x-wiki rht43jgftz41ma2zac70fzbbq17bkit 2424900 2424899 2026-06-09T14:07:24Z Stashbot 7414 Dreamy_Jazz: Afternoon UTC backport window done 2424900 wikitext text/x-wiki emb8is8c4z4x1f5bevpmd9zw7vxy9gr 2424901 2424900 2026-06-09T14:10:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host parsoidtest1001.eqiad.wmnet 2424901 wikitext text/x-wiki 5bo6hjbdv3812a78ivk54u33qmlg7bg 2424902 2424901 2026-06-09T14:12:15Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1184: Migration of db1184.eqiad.wmnet completed 2424902 wikitext text/x-wiki 6pd8l5n7dlxe8ycz55j6iqb3kdjvf6i 2424903 2424902 2026-06-09T14:12:18Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2424903 wikitext text/x-wiki bwqkzj3er2tr76knb4qss3i6awae7sm 2424904 2424903 2026-06-09T14:14:43Z Stashbot 7414 moritzm: imported routinator 0.15.2-1bookworm to thirdparty/routinator for bookworm-wikimedia T428456 2424904 wikitext text/x-wiki t3ibbzbonbxcnfwx92ptgh1hcoy1rx2 2424905 2424904 2026-06-09T14:16:08Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to drbd 2424905 wikitext text/x-wiki gaur0ymil0dwswm292q0zx7xcqpt344 2424906 2424905 2026-06-09T14:16:12Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Migration of db2153.codfw.wmnet completed 2424906 wikitext text/x-wiki hu53b6ojok2xmyrkb9xjitx2rgzuc8g 2424907 2424906 2026-06-09T14:16:14Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2424907 wikitext text/x-wiki cmgptzzip1fju3bv3r59fl1g13x51i8 2424908 2424907 2026-06-09T14:16:31Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host parsoidtest1001.eqiad.wmnet 2424908 wikitext text/x-wiki a4jh3hcpumy6wte5kek0kkx6j6k9qhm 2424910 2424908 2026-06-09T14:20:04Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw 2424910 wikitext text/x-wiki 72vy3tu5cjjjzbpu9bys8rla4a9rjd4 2424913 2424910 2026-06-09T14:26:02Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad 2424913 wikitext text/x-wiki t8l4q1wa7cjlwmnlwruk06m15c4pszl 2424914 2424913 2026-06-09T14:26:19Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-eqiad 2424914 wikitext text/x-wiki m9et5u1f7agyi6k078q2emb94y8gj1d 2424915 2424914 2026-06-09T14:33:30Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2001.codfw.wmnet with OS trixie 2424915 wikitext text/x-wiki lqja0d6n83b2mvpzlqicitvqam7eu2n 2424916 2424915 2026-06-09T14:35:02Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad 2424916 wikitext text/x-wiki o0mmg249n0whcsuvpci4tp01vcgblbm 2424918 2424916 2026-06-09T14:40:40Z Stashbot 7414 moritzm: upgrade routinator in codfw to 0.15.2 T428456 2424918 wikitext text/x-wiki svn8ovfxskok278bsti1yqbdbxblxzk 2424920 2424918 2026-06-09T14:46:08Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply 2424920 wikitext text/x-wiki 6h1vfuzqih1t1h2xdck5vyxararvbzc 2424921 2424920 2026-06-09T14:47:18Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply 2424921 wikitext text/x-wiki 67ilrvz0jwplacl6kqr7l1shtc76rcv 2424922 2424921 2026-06-09T14:52:03Z Stashbot 7414 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab[2002-2003].codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: T410849 2424922 wikitext text/x-wiki iglwxdpc3czuu01e75ldrfaoa2ky8w3 2424923 2424922 2026-06-09T14:52:08Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage 2424923 wikitext text/x-wiki prhmjwimtu2otv6dp7yzim3hursu4ss 2424924 2424923 2026-06-09T14:58:34Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage 2424924 wikitext text/x-wiki tae5m2e2kambxltxfsjvxuygf6msx2j 2424925 2424924 2026-06-09T15:01:22Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab2002 for T410849 2424925 wikitext text/x-wiki qci1bw4luq6zyzvpf3hgrdv0539ihxn 2424926 2424925 2026-06-09T15:02:07Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab2002 for T410849 (duration: 00m 45s) 2424926 wikitext text/x-wiki fqjyyrpotw00n14p3l1vup9ahm65yf9 2424927 2424926 2026-06-09T15:02:38Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab1004 for T410849 2424927 wikitext text/x-wiki 8q6gdlj6ihhb3iwsn7oqf1ikhbv4egk 2424928 2424927 2026-06-09T15:03:23Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab1004 for T410849 (duration: 00m 42s) 2424928 wikitext text/x-wiki 4eghs4gtuthag9i8edhj9s2bj6vxnit 2424930 2424928 2026-06-09T15:16:24Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2001.codfw.wmnet with OS trixie 2424930 wikitext text/x-wiki icee1rt1gia7tm7ryq7sda20bzyzl2v 2424931 2424930 2026-06-09T15:21:43Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. 2424931 wikitext text/x-wiki 79r692qhncj7iddfs8gtxf6u7ev8t7p 2424932 2424931 2026-06-09T15:21:58Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. 2424932 wikitext text/x-wiki ecdb1f80ta56aj8d1derau94brjurth 2424933 2424932 2026-06-09T15:22:48Z Stashbot 7414 urbanecm: Remove `migrateMentorStatusAwayToCommunityConfiguration` from updatelog on all wikis (T409170; the script was only ever run as a dry-run) 2424933 wikitext text/x-wiki l8c9szbcwadupqwdhne6sez2itdw4d1 2424935 2424933 2026-06-09T15:25:30Z Stashbot 7414 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1299468|ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] 2424935 wikitext text/x-wiki sq5h3jglopcp7t0aytpe7u4h2dzv8e2 2424936 2424935 2026-06-09T15:26:50Z Stashbot 7414 jiji@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2424936 wikitext text/x-wiki nkidujlyj0wivdd5rohls37nmvyper5 2424937 2424936 2026-06-09T15:27:12Z Stashbot 7414 jiji@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2424937 wikitext text/x-wiki 2loipccm9ic2wczfpfsn9hteznco4h0 2424938 2424937 2026-06-09T15:27:30Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2002.codfw.wmnet with OS trixie 2424938 wikitext text/x-wiki rhwsilt83vbawpsewyma4r50wdr9ldy 2424939 2424938 2026-06-09T15:28:57Z Stashbot 7414 jiji@deploy1003: Rolling back deployment 2424939 wikitext text/x-wiki idt2omqcqsvvfi60pkzhypj90ymm488 2424941 2424939 2026-06-09T15:32:46Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab2002 for T410849 (followup for robots.txt) 2424941 wikitext text/x-wiki tfrbdnyozy282ibr6vns70lfn19gg8e 2424942 2424941 2026-06-09T15:32:52Z Stashbot 7414 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299468|ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] (duration: 07m 21s) 2424942 wikitext text/x-wiki 5yl8dpnvoahnxocgxf1t1q7w933cqyj 2424944 2424942 2026-06-09T15:33:32Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab2002 for T410849 (followup for robots.txt) (duration: 00m 45s) 2424944 wikitext text/x-wiki 5n24eb2rd6hve6drlwn6ke0ize8to8a 2424945 2424944 2026-06-09T15:33:55Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab1004 for T410849 (followup for robots.txt) 2424945 wikitext text/x-wiki pwvu660snqnw7wzd8g09zq5z5q92f66 2424946 2424945 2026-06-09T15:34:33Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab1004 for T410849 (followup for robots.txt) (duration: 00m 40s) 2424946 wikitext text/x-wiki b2rn7gilqbc9xop8owa3eegtn196s4f 2424948 2424946 2026-06-09T15:43:48Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad 2424948 wikitext text/x-wiki abqr883lubr7x4tywdb4kbh5a6yk5d3 2424949 2424948 2026-06-09T15:45:29Z Stashbot 7414 jiji@deploy1003: Started scap sync-world: redeploy 1299468 2424949 wikitext text/x-wiki 2wtlggdakcskhjmqxl8bxm3ney95zbf 2424950 2424949 2026-06-09T15:46:04Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage 2424950 wikitext text/x-wiki ol71wvxy8yklnhacbdk80pmvx6rcbmh 2424951 2424950 2026-06-09T15:46:43Z Stashbot 7414 jiji@deploy1003: jiji: redeploy 1299468 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2424951 wikitext text/x-wiki 3xt11xzt6w8nys7dph7jo3h201rcgg5 2424952 2424951 2026-06-09T15:47:15Z Stashbot 7414 jiji@deploy1003: jiji: Continuing with deployment 2424952 wikitext text/x-wiki f8kfv5deq6eesqw3lnowvmnlxm4rby8 2424953 2424952 2026-06-09T15:49:57Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage 2424953 wikitext text/x-wiki fu3txa0eg3ryk18m59ysx6la5e6spin 2424955 2424953 2026-06-09T15:51:35Z Stashbot 7414 jiji@deploy1003: Finished scap sync-world: redeploy 1299468 (duration: 07m 23s) 2424955 wikitext text/x-wiki ca2bdzyvleefmsf4klja6shg9nt17ml 2424956 2424955 2026-06-09T15:53:50Z Stashbot 7414 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. 2424956 wikitext text/x-wiki 7qc3t0eic136ojwvc4z09781y8077c4 2424957 2424956 2026-06-09T15:54:31Z Stashbot 7414 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. 2424957 wikitext text/x-wiki n6rzl6u6j8bdi732tvbq0bp0dgxw41l 2424958 2424957 2026-06-09T15:56:42Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply 2424958 wikitext text/x-wiki gi18c09zjyti9or35k28ekke8aynieo 2424959 2424958 2026-06-09T15:57:06Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply 2424959 wikitext text/x-wiki lphalqfhy9lo0v9p27uzb6wwswce92h 2424960 2424959 2026-06-09T15:57:15Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. 2424960 wikitext text/x-wiki 4mxh1lzte7hjpbzxoxayu2k0aombljm 2424961 2424960 2026-06-09T15:57:49Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. 2424961 wikitext text/x-wiki s9dwl2g3lef4xbtb3sdhfvb6r5phmxf 2424962 2424961 2026-06-09T15:58:17Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply 2424962 wikitext text/x-wiki bot0q4d8vq278tks62bxyfr4mfv9utg 2424963 2424962 2026-06-09T15:59:00Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply 2424963 wikitext text/x-wiki dbsbctpv3r4orjsxf9k339lpjugyrlo 2424964 2424963 2026-06-09T15:59:09Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply 2424964 wikitext text/x-wiki 4oeemts77jj6dpaucuwe8rwz1e3o79a 2424965 2424964 2026-06-09T15:59:11Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. 2424965 wikitext text/x-wiki mjvy0o06u51tnp6e5w4qj8gs6rd6qda 2424966 2424965 2026-06-09T15:59:30Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2424966 wikitext text/x-wiki gvmqxfjd3ohboukwpuztwvm8l1r6rwj 2424967 2424966 2026-06-09T15:59:45Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. 2424967 wikitext text/x-wiki qq7t3hbqqgmz0c82yuvghxb5q1aswue 2424968 2424967 2026-06-09T15:59:51Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply 2424968 wikitext text/x-wiki lgaof2qvmx86378otavg1inyj8tno2x 2424969 2424968 2026-06-09T16:00:28Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. 2424969 wikitext text/x-wiki 36i36e9pjg6gcxfpemg9onurse8vt3m 2424970 2424969 2026-06-09T16:02:26Z Stashbot 7414 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. 2424970 wikitext text/x-wiki 4t66vdthgrlirdwi4ughuidvnzvog3u 2424971 2424970 2026-06-09T16:02:38Z Stashbot 7414 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2424971 wikitext text/x-wiki qyo3cfhug7f4z8fjc4ucfxlc78v6ex4 2424973 2424971 2026-06-09T16:07:40Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2002.codfw.wmnet with OS trixie 2424973 wikitext text/x-wiki mj3ah22m1z6iw7y4g9uvv3uekar55g9 2424974 2424973 2026-06-09T16:09:34Z Stashbot 7414 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. 2424974 wikitext text/x-wiki 801u01gm8jue2ozhfqdckqysdh5mdrs 2424975 2424974 2026-06-09T16:10:12Z Stashbot 7414 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. 2424975 wikitext text/x-wiki rsd7jij4io4u6vz5swyx2vxfdurntss 2424976 2424975 2026-06-09T16:12:06Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1001.eqiad.wmnet with OS trixie 2424976 wikitext text/x-wiki bz6ukw12atnsu0aq08c3vuejo6yn0s3 2424977 2424976 2026-06-09T16:13:08Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2424977 wikitext text/x-wiki pict8s5fp6anu0v2caersbtoc0vh1yr 2424978 2424977 2026-06-09T16:13:13Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2424978 wikitext text/x-wiki o1h9n95tya2yaib3w6tldnidmpqfmvh 2424979 2424978 2026-06-09T16:15:45Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2424979 wikitext text/x-wiki nnasbldt58diacw2su2i77afw21r3nu 2424980 2424979 2026-06-09T16:16:08Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2424980 wikitext text/x-wiki tb4bbnht3nuoofqkpgc9ovu4le631pv 2424983 2424980 2026-06-09T16:19:34Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2424983 wikitext text/x-wiki furwb5bc1fmpmk8hnsoceg72po21x68 2424984 2424983 2026-06-09T16:19:38Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2424984 wikitext text/x-wiki 8ibr2ignplbj5ew8n9u0xmseh8hx3ru 2424986 2424984 2026-06-09T16:20:16Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2424986 wikitext text/x-wiki 7cdglf9kvjei7bx8on4pvko3aeqxsvq 2424988 2424986 2026-06-09T16:22:42Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2424988 wikitext text/x-wiki tpz876sd9nzhdju79fhto6vgyo433fd 2424989 2424988 2026-06-09T16:23:18Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2424989 wikitext text/x-wiki 7gd7gh0s0zglb83l4unl1yyaenqqjsy 2424991 2424989 2026-06-09T16:26:07Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage 2424991 wikitext text/x-wiki 746adnys9cthsd1tq1g2dcreym58x8o 2424994 2424991 2026-06-09T16:28:39Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2424994 wikitext text/x-wiki 25xbns73rqiewil9waafpwrybdrvjr3 2424996 2424994 2026-06-09T16:29:12Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2424996 wikitext text/x-wiki 0ivioninaibze5gh8gcdfzgcmdsyiri 2424997 2424996 2026-06-09T16:30:26Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage 2424997 wikitext text/x-wiki 360bmhvuiaf555boyci0ni8q097n6si 2424998 2424997 2026-06-09T16:30:40Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply 2424998 wikitext text/x-wiki qop1tpyqexhlz2asy6tfuif90gktsua 2425000 2424998 2026-06-09T16:31:16Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply 2425000 wikitext text/x-wiki crje7m3z8l1akue5gcu3d38hk5znkek 2425001 2425000 2026-06-09T16:31:18Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425001 wikitext text/x-wiki fk4oo5tcfatoelsjh80l9fejfuv8tke 2425002 2425001 2026-06-09T16:31:21Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425002 wikitext text/x-wiki sge1c8h9hc0k7szql8unh86wzekpsjy 2425003 2425002 2026-06-09T16:31:24Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425003 wikitext text/x-wiki 0808do15sqdwtsmiuzmvn4e1uk6r2qj 2425004 2425003 2026-06-09T16:34:44Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie 2425004 wikitext text/x-wiki 3qr69d1liguct1gphai50c20q7nno8a 2425005 2425004 2026-06-09T16:35:35Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2008 2425005 wikitext text/x-wiki 05qjbf18z9meojkopitve2shdqaq4iy 2425008 2425005 2026-06-09T16:41:14Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply 2425008 wikitext text/x-wiki ckbmzi2z88jxengvsf0e1tn9d73jf0t 2425009 2425008 2026-06-09T16:41:36Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply 2425009 wikitext text/x-wiki sz1oo7u7ilrtttoi46on6kg55bk8l32 2425010 2425009 2026-06-09T16:47:26Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2425010 wikitext text/x-wiki bqem1ft4laghupj227kc80sgzhxbsem 2425011 2425010 2026-06-09T16:47:29Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply 2425011 wikitext text/x-wiki oj4bjvljj8hty5fbaklt4lz4a5z2y6m 2425012 2425011 2026-06-09T16:47:44Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply 2425012 wikitext text/x-wiki 5e7cmhc4vx88tim7usjpgcslcv8t9ol 2425013 2425012 2026-06-09T16:47:57Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1001.eqiad.wmnet with OS trixie 2425013 wikitext text/x-wiki rlk4jnnyo2xal4737j6tp3mnopbuzvg 2425014 2425013 2026-06-09T16:48:16Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2425014 wikitext text/x-wiki azrae8il3vkgg6iadc36dygyl3qyhwl 2425015 2425014 2026-06-09T16:50:24Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS trixie 2425015 wikitext text/x-wiki mbp0tufclp7uh5dhl0ahyan92ws1m6d 2425016 2425015 2026-06-09T16:52:36Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2425016 wikitext text/x-wiki j91s42t7xbdy9ktthdit9s2uxozfv16 2425017 2425016 2026-06-09T16:53:20Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2425017 wikitext text/x-wiki 610l52wobhith0i3i6miuysv6bbp129 2425018 2425017 2026-06-09T16:57:03Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2425018 wikitext text/x-wiki 9twngmhbryjof97yow16tltbos90qvn 2425019 2425018 2026-06-09T16:57:08Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2425019 wikitext text/x-wiki ct8oc6ihlfcgwkmsajohobhflughiw9 2425020 2425019 2026-06-09T16:57:13Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.netbox 2425020 wikitext text/x-wiki sifqesqasn97yp75s7qsy0vl8ncwd1c 2425021 2425020 2026-06-09T16:58:41Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2425021 wikitext text/x-wiki tnjwyxwgfbj3zwt16pf6cs11ytbmb07 2425022 2425021 2026-06-09T16:58:57Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2425022 wikitext text/x-wiki 78bp6ja8zlfx03397bo86djaynfxwpa 2425024 2425022 2026-06-09T17:03:44Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5018.eqsin.wmnet with OS trixie 2425024 wikitext text/x-wiki fwgk0q2tkgmoaqfl4bcojxbzgf2zxyk 2425025 2425024 2026-06-09T17:04:09Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" 2425025 wikitext text/x-wiki kz77wuaduajn2dsnato75f9dswnuaus 2425026 2425025 2026-06-09T17:04:16Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5018 2425026 wikitext text/x-wiki 3yq6uzaws2mxr0xe1y8lplrukjjhjdk 2425027 2425026 2026-06-09T17:04:18Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" 2425027 wikitext text/x-wiki bd1gpwdwnsj7rdbeslh4st0dlqx9qwv 2425028 2425027 2026-06-09T17:04:20Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2425028 wikitext text/x-wiki 6dsve26gtg7e731v7jxi0pbzm5zk0b7 2425029 2425028 2026-06-09T17:04:23Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2425029 wikitext text/x-wiki 2dor8w67vbk4ur572guq4cnce0nskuf 2425030 2425029 2026-06-09T17:04:25Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2425030 wikitext text/x-wiki f42lswrbx0jsbijnsc1iybvv4zepsuf 2425031 2425030 2026-06-09T17:04:27Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2425031 wikitext text/x-wiki sdsezus0j9ql07ozepo3gt5ue3lmpo1 2425032 2425031 2026-06-09T17:04:29Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2008 2425032 wikitext text/x-wiki b0vat976wdy88syilnjzo1x0mruo8pl 2425033 2425032 2026-06-09T17:04:54Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2425033 wikitext text/x-wiki g5afmko0nt9f2ludxvq45jvtgj7wreu 2425035 2425033 2026-06-09T17:05:27Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2008 2425035 wikitext text/x-wiki 36uu43wx2bidk7nciifqcpn4tpkr6ae 2425036 2425035 2026-06-09T17:05:29Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2008 2425036 wikitext text/x-wiki rbvjnc48pcxhli34gdvrm6xs9t6hd7s 2425037 2425036 2026-06-09T17:06:50Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage 2425037 wikitext text/x-wiki biy7on7ia0fdnli88u7b38wlff3j1t9 2425038 2425037 2026-06-09T17:14:15Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage 2425038 wikitext text/x-wiki qivc4pw4bplkh3jv9rcujxj0q17djck 2425039 2425038 2026-06-09T17:32:13Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1002.eqiad.wmnet with OS trixie 2425039 wikitext text/x-wiki 2tylrreb0mqp0ofnorkc0o2wghmk886 2425041 2425039 2026-06-09T17:43:40Z Stashbot 7414 jayme: kafka-main2008 is down due to hardware failure T428654 2425041 wikitext text/x-wiki 796qlc9cpiy8r64pmbvgjq2kn7uexpy 2425042 2425041 2026-06-09T17:46:07Z Stashbot 7414 sukhe: sudo cumin 'A:hcaptcha-proxy' 'run-puppet-agent': rolling out CR 1299427 T428539 2425042 wikitext text/x-wiki hfx58v3h1j3d7nt65wyqafs4gvaz1ai 2425043 2425042 2026-06-09T17:47:06Z Stashbot 7414 cmooney@dns2005: START - running authdns-update 2425043 wikitext text/x-wiki q8jxgmt0sq46ukle4cfk6cbmc3vppw4 2425044 2425043 2026-06-09T17:47:46Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: reimage 2425044 wikitext text/x-wiki 1s6gjq7ga0x8nm32tdep775j03akzmz 2425045 2425044 2026-06-09T17:48:27Z Stashbot 7414 cmooney@dns2005: END - running authdns-update 2425045 wikitext text/x-wiki 9h3ke4a3qnhr8777daxlv0jl3qy5thk 2425046 2425045 2026-06-09T17:48:54Z Stashbot 7414 mutante: https://releases.wikimedia.org | https://releases-jenkins.wikimedia.org - down for maintenance T418299 2425046 wikitext text/x-wiki cbehw4v7ol8yhvrb7z4v0abs0s3mdsq 2425052 2425046 2026-06-09T18:26:11Z Stashbot 7414 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2008.codfw.wmnet with OS trixie 2425052 wikitext text/x-wiki s2sqxietcfzl3tfnms8a4tj25j9cgx3 2425055 2425052 2026-06-09T18:29:24Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie 2425055 wikitext text/x-wiki 43nx6rjwb8m5t392zghmvnfw18wraks 2425056 2425055 2026-06-09T18:31:02Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.6 refs T423915 2425056 wikitext text/x-wiki cg0rpaceojpzdu73ymd520xoejpjreo 2425057 2425056 2026-06-09T18:42:22Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425057 wikitext text/x-wiki fzjwa29r5ciqjga11l68uypnln06fjl 2425058 2425057 2026-06-09T18:42:47Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425058 wikitext text/x-wiki 28ra21ei8rurs0ppuqq689424we3kv0 2425059 2425058 2026-06-09T18:43:14Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425059 wikitext text/x-wiki jp8lnrfkm07pw6ge11b5850jejed1ka 2425060 2425059 2026-06-09T18:43:38Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425060 wikitext text/x-wiki nz7c0clf5w3cgwu49kyo6nmi0t589ey 2425061 2425060 2026-06-09T18:46:29Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425061 wikitext text/x-wiki 1j8epz65k92aevwaqtz9hxtek63bk89 2425062 2425061 2026-06-09T18:46:52Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425062 wikitext text/x-wiki rp8rabwdtlr7njygpef75lylp4ga7vt 2425063 2425062 2026-06-09T18:47:10Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425063 wikitext text/x-wiki lg3y4xj88p48zhyi7m253b5l4v8sad4 2425064 2425063 2026-06-09T18:47:27Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425064 wikitext text/x-wiki k7vdqxl8nphfm1ofr414na26hlt3mma 2425065 2425064 2026-06-09T18:47:54Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2425065 wikitext text/x-wiki chmuk5amtpfcvt30p8xwvjf676j7i4z 2425066 2425065 2026-06-09T18:50:37Z Stashbot 7414 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2425066 wikitext text/x-wiki lz538fhprqk141gapomr6b1r52j3rj3 2425067 2425066 2026-06-09T18:50:59Z Stashbot 7414 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2425067 wikitext text/x-wiki d1oxu5kyacin5a33xns80vwsne1ehw5 2425068 2425067 2026-06-09T18:51:07Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2425068 wikitext text/x-wiki 0apl5xb5b4btbso5hcptkdjjdghphol 2425069 2425068 2026-06-09T18:51:48Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2425069 wikitext text/x-wiki 5hpupkxi2p0ebdrrltqh9p1tfjdi3e1 2425070 2425069 2026-06-09T18:51:56Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2425070 wikitext text/x-wiki 4diu9v96hlwoe57lucj6eeeswd47u12 2425071 2425070 2026-06-09T18:51:58Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage 2425071 wikitext text/x-wiki n3a0hgbav9pm36glg4op0jv1ussbs4j 2425072 2425071 2026-06-09T18:52:23Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2425072 wikitext text/x-wiki 3c8z7m8xjgz66kr5rc4s0eo3u4k2i3s 2425073 2425072 2026-06-09T18:52:32Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2425073 wikitext text/x-wiki 6oeca9vxoo49c4yv9cc7hchq6ei0ule 2425074 2425073 2026-06-09T18:52:37Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" 2425074 wikitext text/x-wiki ittun4xqndomfimn2cxgco07u3evg4o 2425075 2425074 2026-06-09T18:52:43Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" 2425075 wikitext text/x-wiki ag7uncr543baysd6uie9l7jkozsqtih 2425076 2425075 2026-06-09T18:52:45Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2425076 wikitext text/x-wiki lup196fzfd9jh1bdkfyqmucdrhksexz 2425077 2425076 2026-06-09T18:53:20Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2425077 wikitext text/x-wiki 9deha0g31u6flnphlolhnh5ntygcbop 2425078 2425077 2026-06-09T18:53:28Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2425078 wikitext text/x-wiki m5l3et5eqncru0p0pe1egii74u1y2go 2425079 2425078 2026-06-09T18:53:56Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2425079 wikitext text/x-wiki jbj0l0d4zou3stolmmc2iu3nu3zkgqp 2425080 2425079 2026-06-09T18:54:04Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2425080 wikitext text/x-wiki c2pz154gz0zrcvswi1uvq8u9hhc01d5 2425081 2425080 2026-06-09T18:54:36Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425081 wikitext text/x-wiki 24vqnp3daw56t5r913yvcj9p2cfj1ib 2425082 2425081 2026-06-09T18:54:55Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2425082 wikitext text/x-wiki t9a175tqwtx9mx3j0i8v9vngyialn9o 2425083 2425082 2026-06-09T18:55:02Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2425083 wikitext text/x-wiki i01f6wzogd03r2fd1dyfrshwcd9fcua 2425084 2425083 2026-06-09T18:55:51Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2425084 wikitext text/x-wiki 3pu51q12peqbbihkhcf4eg5asw9wmhv 2425085 2425084 2026-06-09T18:55:59Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2425085 wikitext text/x-wiki fdgr8lewduj36yai2qqc3awkp5lzrsk 2425086 2425085 2026-06-09T18:56:28Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425086 wikitext text/x-wiki dneu4ft5x7ylkehz2lszpr54jsu6p40 2425087 2425086 2026-06-09T18:56:36Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2425087 wikitext text/x-wiki 07er27os1ie5s3vvjk6mnqox88u5we8 2425088 2425087 2026-06-09T18:57:26Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2425088 wikitext text/x-wiki og1bdfkc3holt1jju0uf20hfosbzn0t 2425089 2425088 2026-06-09T18:57:32Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2425089 wikitext text/x-wiki bl6zwtmupfj44icpfdae1tv9utkdbka 2425090 2425089 2026-06-09T18:58:03Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425090 wikitext text/x-wiki 37rhjun3y15aatrpb03kh1hl34vrhmh 2425091 2425090 2026-06-09T18:58:09Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2425091 wikitext text/x-wiki k6003jax1s6blqm0gnpl5h6p9p85ykp 2425092 2425091 2026-06-09T18:58:42Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage 2425092 wikitext text/x-wiki ehxwp94pfxgn1324iblq3dlmp04l4x3 2425093 2425092 2026-06-09T18:59:00Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2425093 wikitext text/x-wiki 3lya6fwuhcos1xmzacpn5pv82rsoy2i 2425094 2425093 2026-06-09T19:00:10Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425094 wikitext text/x-wiki nohjlu8s0zvyy75kffnbd4vb1as8hxd 2425095 2425094 2026-06-09T19:12:24Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync 2425095 wikitext text/x-wiki cn2dcml5w253aho44j3nsrs8qazey9s 2425096 2425095 2026-06-09T19:12:46Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync 2425096 wikitext text/x-wiki sen4ynksvougoh02gm0njcghpnpi90q 2425097 2425096 2026-06-09T19:15:08Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1015.eqiad.wmnet 2425097 wikitext text/x-wiki 7m7eo26yu7qpz35kxwpnewl4gnrkfk4 2425098 2425097 2026-06-09T19:15:51Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2008.codfw.wmnet with OS trixie 2425098 wikitext text/x-wiki 0gab5ph7o29zevblbhyet2jzuc8ek3d 2425099 2425098 2026-06-09T19:20:38Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.dns.netbox 2425099 wikitext text/x-wiki 5y1mudzj014loxf112marq2h63jxx88 2425100 2425099 2026-06-09T19:27:54Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" 2425100 wikitext text/x-wiki nf3y58f3hw3z496li0xol57xi7gmkaa 2425101 2425100 2026-06-09T19:28:27Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" 2425101 wikitext text/x-wiki likx9fy23jcnoydb58e9zccoj8stzl9 2425102 2425101 2026-06-09T19:28:29Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2425102 wikitext text/x-wiki q4hayameatb7ue4o5duom5kjsofi79z 2425103 2425102 2026-06-09T19:28:32Z Stashbot 7414 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wdqs1015.eqiad.wmnet 2425103 wikitext text/x-wiki 8zqnjin2nu5yvk2y8kh6ktn3ul070fz 2425104 2425103 2026-06-09T19:45:41Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425104 wikitext text/x-wiki 8y5cfk2ob9bimoivkjsyb64qedb14si 2425105 2425104 2026-06-09T19:45:58Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425105 wikitext text/x-wiki 6bmhm5ztg8czi2w19uh7cn7bgxv1045 2425106 2425105 2026-06-09T19:46:41Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425106 wikitext text/x-wiki 3qjg3cy2n9glu0cn7g4ebbttl36jrpv 2425107 2425106 2026-06-09T19:46:59Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425107 wikitext text/x-wiki nqbxeej2zageunkn2a97i405mhgkjd6 2425108 2425107 2026-06-09T19:47:19Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425108 wikitext text/x-wiki hzzvcy2btjjn9mfeeymq4b1ykqt0jl9 2425109 2425108 2026-06-09T19:47:44Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425109 wikitext text/x-wiki gngy1uey1q8pfrw3ebitr9eorhfb0kh 2425110 2425109 2026-06-09T19:48:15Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425110 wikitext text/x-wiki e11qracqpk5jno23nmzhqt2rr76mwiu 2425111 2425110 2026-06-09T19:53:34Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425111 wikitext text/x-wiki i1z2194dreo8z8eljgn07oe02s5tea3 2425112 2425111 2026-06-09T19:54:10Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425112 wikitext text/x-wiki m6qu1mv87gnsl9bote6q4x8ds08ykf7 2425113 2425112 2026-06-09T19:59:30Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425113 wikitext text/x-wiki 5dag5c5tu631qbzxgxlf6cxywd8967a 2425114 2425113 2026-06-09T20:04:12Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425114 wikitext text/x-wiki p2j03r4okfmffklbdymfa718yf66xhx 2425115 2425114 2026-06-09T20:09:39Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425115 wikitext text/x-wiki a6ig0uh6l9m7hhycx3hoa9y2m3lv9fv 2425116 2425115 2026-06-09T20:10:15Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299454|wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] 2425116 wikitext text/x-wiki ih6yzvn4rfbfw62rachlqllf7tnuc0t 2425117 2425116 2026-06-09T20:12:19Z Stashbot 7414 cscott@deploy1003: cscott, gkyziridis: Backport for [[gerrit:1299454|wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425117 wikitext text/x-wiki 0463uv0gvzyy83za1678fkbvp49iyv9 2425118 2425117 2026-06-09T20:12:44Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425118 wikitext text/x-wiki gda9wtdru0qfc03rrmnd45rl18iao6z 2425119 2425118 2026-06-09T20:13:39Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 2425119 wikitext text/x-wiki is8pek1zkl01o0cbzz2owkg78lzk9be 2425120 2425119 2026-06-09T20:13:48Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 2425120 wikitext text/x-wiki haylyaftk6g1ldaaa901n3l3yokvj3p 2425121 2425120 2026-06-09T20:13:51Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 2425121 wikitext text/x-wiki 5iym3wej7p0vwv5twnxkvaamebd4ge1 2425122 2425121 2026-06-09T20:14:01Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 2425122 wikitext text/x-wiki nvgyo0twdx0im7kosc5llqi94ai5zm9 2425123 2425122 2026-06-09T20:14:05Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 2425123 wikitext text/x-wiki g4d4w5pq9ytto6kj3lv7nnj8p52r3yp 2425124 2425123 2026-06-09T20:14:15Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 2425124 wikitext text/x-wiki c3sa25hwydwjjmsx2dq9lop2f0j3un7 2425125 2425124 2026-06-09T20:14:19Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 2425125 wikitext text/x-wiki ez8dhjjq7bxm5kes4gyywfr6c96lcs1 2425126 2425125 2026-06-09T20:14:28Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 2425126 wikitext text/x-wiki oba9koiab0qzu8cirou7jiapd7d6o2n 2425127 2425126 2026-06-09T20:24:53Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425127 wikitext text/x-wiki ean94az1mfyhb4exdxy6srqmtqi5xo4 2425128 2425127 2026-06-09T20:28:03Z Stashbot 7414 cscott@deploy1003: cscott, gkyziridis: Continuing with deployment 2425128 wikitext text/x-wiki bwi647jool7yr2fd4j9iwwysv399xll 2425129 2425128 2026-06-09T20:28:37Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425129 wikitext text/x-wiki twg61ke4jwecp7ym1xc15g3zjs0rmzp 2425130 2425129 2026-06-09T20:32:23Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299454|wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] (duration: 22m 08s) 2425130 wikitext text/x-wiki 21yupk8px0hoiz0pfu8owpvsg4yngct 2425131 2425130 2026-06-09T20:33:13Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425131 wikitext text/x-wiki dqnrd01s0mbevs4ubl4oo6zq0zn70zn 2425132 2425131 2026-06-09T20:33:29Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425132 wikitext text/x-wiki hq3ee3et9vo9i4fc89sve2h7qxt2mjk 2425133 2425132 2026-06-09T20:38:14Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425133 wikitext text/x-wiki qpe7rfuzbfkddbrtm2llyx0nu2vchxp 2425134 2425133 2026-06-09T20:38:39Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425134 wikitext text/x-wiki f2np24v2iuhlag5m8pah14u9cz6s3mo 2425135 2425134 2026-06-09T20:39:32Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299588|Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589|Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] 2425135 wikitext text/x-wiki 1g1vv12nt58l5i8ewmxlg914no0ncor 2425136 2425135 2026-06-09T20:41:28Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1299588|Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589|Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425136 wikitext text/x-wiki 9ddl1mpdtnc4bofuq03wrxble34uf36 2425137 2425136 2026-06-09T20:42:45Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425137 wikitext text/x-wiki f7f9jrefzhwuxw8kz7oefn9iys9skqb 2425138 2425137 2026-06-09T20:43:02Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2425138 wikitext text/x-wiki 0iwct7pbkpfp71iamdd1pboooe6n7i1 2425139 2425138 2026-06-09T20:43:43Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie 2425139 wikitext text/x-wiki gvwn21z7wuulk32cnsdq58pmmm4ebam 2425140 2425139 2026-06-09T20:46:14Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2425140 wikitext text/x-wiki 39elkzm7nqric8xfblsrhjvh0h2ngm9 2425141 2425140 2026-06-09T20:50:40Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299588|Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589|Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] (duration: 11m 13s) 2425141 wikitext text/x-wiki rb1lu31sedfhc1q4ier5t781oek06fd 2425142 2425141 2026-06-09T20:50:52Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie 2425142 wikitext text/x-wiki mh2nspgs48uijte1233ujvay6y3ag9b 2425143 2425142 2026-06-09T21:06:46Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart 2425143 wikitext text/x-wiki gyk3c2mhzi52ygkk77ixvo2cqpx520z 2425146 2425143 2026-06-09T21:15:55Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart 2425146 wikitext text/x-wiki m86wp5fdo5g9jiow2cgx2my2f06tmeb 2425148 2425146 2026-06-09T21:24:59Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) 2425148 wikitext text/x-wiki fh51pfxtlr4o1f220pzavsg5b26h6hl 2425149 2425148 2026-06-09T21:27:06Z Stashbot 7414 maryum: Deployed security fix for T428324 2425149 wikitext text/x-wiki nsajt6sdrgrese3k8vn9031v91n58wh 2425151 2425149 2026-06-09T21:34:53Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: debug 2425151 wikitext text/x-wiki 8o6mmtmo77ci9aekyp979e6f8kk32ta 2425153 2425151 2026-06-09T21:41:32Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299602|Revert "Create VectorComponentPageToolbar component" (T428649)]] 2425153 wikitext text/x-wiki kr9ok3hvbvgsde57e9ujvcnpi6cltta 2425154 2425153 2026-06-09T21:43:31Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299602|Revert "Create VectorComponentPageToolbar component" (T428649)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425154 wikitext text/x-wiki k9vajaemb8sh8uljdz1uatwkyrsjeg5 2425155 2425154 2026-06-09T21:45:33Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2425155 wikitext text/x-wiki 8qwofoy5ag9tfpc0gpui0jv76v08p14 2425156 2425155 2026-06-09T21:48:21Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) 2425156 wikitext text/x-wiki 5qiobsxqwzmx5ztgvdu8qgl1zq72e6q 2425157 2425156 2026-06-09T21:49:48Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299602|Revert "Create VectorComponentPageToolbar component" (T428649)]] (duration: 08m 16s) 2425157 wikitext text/x-wiki csef4y85kl7dmsfp1fldpr06226gjpf 2425158 2425157 2026-06-09T21:52:19Z Stashbot 7414 ryankemper: T428241 removed retired wdqs2009 full-graph journal dump (446G x2, ~892G) from clouddumps100[1-2]:/srv/dumps/xmldatadumps/public/other/wdqs 2425158 wikitext text/x-wiki 2pkqqdoaj9g3x9jj1i3l2m4eqfdj9mo 2425159 2425158 2026-06-09T21:54:53Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299639|[Bug] Donor Badge: Remove client prefs for control group (T428501)]] 2425159 wikitext text/x-wiki 1zepl7sppyqg6lo2iy5jhl1vyv9rr2y 2425160 2425159 2026-06-09T21:56:56Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299639|[Bug] Donor Badge: Remove client prefs for control group (T428501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425160 wikitext text/x-wiki 5m4dxye52up76xp6y583pj8t5886p7k 2425161 2425160 2026-06-09T22:06:06Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: debug 2425161 wikitext text/x-wiki 0bbz3m9noprkeza3xjvu1kaox7q4b6b 2425162 2425161 2026-06-09T22:07:11Z Stashbot 7414 mutante: gerrit - apache httpd log file location moved to /srv/gerrit/site_path/review_site/logs/ T425667 2425162 wikitext text/x-wiki g5xz0exhtn3sghsb9ubrx2qwgbuh5mb 2425163 2425162 2026-06-09T22:11:29Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2425163 wikitext text/x-wiki pdqiu13bu9w15ugsjdfmgkm6q7uvenb 2425164 2425163 2026-06-09T22:15:50Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299639|[Bug] Donor Badge: Remove client prefs for control group (T428501)]] (duration: 20m 57s) 2425164 wikitext text/x-wiki tnl31hgfhabav297osl3rr1u4d839mw 2425165 2425164 2026-06-09T22:41:17Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299640|HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] 2425165 wikitext text/x-wiki pnj72hqaxv9jpolb54gytim7c1ftoez 2425166 2425165 2026-06-09T22:43:18Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1299640|HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425166 wikitext text/x-wiki 36rr65mj0wsbovn0sf9x2p6rfmm1pqe 2425167 2425166 2026-06-09T22:45:56Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2425167 wikitext text/x-wiki orec8twyo9av513dblcnf36ccir11bh 2425168 2425167 2026-06-09T22:50:16Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299640|HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] (duration: 08m 59s) 2425168 wikitext text/x-wiki itkhynhnybyrkihg91c3osu36vciumm 2425170 2425168 2026-06-10T00:03:21Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2009.codfw.wmnet with OS trixie 2425170 wikitext text/x-wiki tn70pp249sk7r615vjgz4nesd17219t 2425171 2425170 2026-06-10T00:03:52Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2009 2425171 wikitext text/x-wiki orj6hhnl3lrpgxjf0dbe8fephrut52u 2425172 2425171 2026-06-10T00:10:50Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.netbox 2425172 wikitext text/x-wiki mj0isp75jdrkhxqzan0kidkw0csit2q 2425173 2425172 2026-06-10T00:15:12Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" 2425173 wikitext text/x-wiki 6w2nd8ex6qew3n6g8vmhkpbfjq9g47d 2425174 2425173 2026-06-10T00:15:18Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" 2425174 wikitext text/x-wiki f43av0dorveyp8sie539lygemu7wz8q 2425175 2425174 2026-06-10T00:15:20Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2425175 wikitext text/x-wiki oatiib077l7gbdrfb2az2mz31qtxw9j 2425176 2425175 2026-06-10T00:15:23Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2425176 wikitext text/x-wiki 0llzm68zimbg422d04qgj3f822qg5ez 2425177 2425176 2026-06-10T00:15:25Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2425177 wikitext text/x-wiki dop46b982uem5p9syj6n21jaz612qup 2425178 2425177 2026-06-10T00:15:27Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2009 2425178 wikitext text/x-wiki doqpispgqti5ywwcm3qqn1ac37d8b9t 2425179 2425178 2026-06-10T00:15:41Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2009 2425179 wikitext text/x-wiki 6v7jsfgmgad41qwe6yh3k4lqnfehktn 2425180 2425179 2026-06-10T00:15:43Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2009 2425180 wikitext text/x-wiki evkgxrca7cc249bjww6dd1txf6d9971 2425181 2425180 2026-06-10T00:32:28Z Stashbot 7414 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2425181 wikitext text/x-wiki gc96856zvwnrcsh3coywyksm68r9eyp 2425182 2425181 2026-06-10T00:32:50Z Stashbot 7414 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2425182 wikitext text/x-wiki e698psehe4h5aiv8l6sqver89zy7uvg 2425183 2425182 2026-06-10T00:32:58Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2425183 wikitext text/x-wiki np2h5bn1rmymxnbzmfi5h2ze7v03nqc 2425184 2425183 2026-06-10T00:33:39Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2425184 wikitext text/x-wiki nx7snbz06i2q2yktwha6txf8szreknl 2425185 2425184 2026-06-10T00:33:47Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2425185 wikitext text/x-wiki e3yo2z6u664t00kz13rid8hta415hqn 2425186 2425185 2026-06-10T00:34:17Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2425186 wikitext text/x-wiki 74kvyxq8qzdfshw0d35u6eicme6fsvy 2425187 2425186 2026-06-10T00:34:24Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2425187 wikitext text/x-wiki tvox0ypx748qefkrzg4jk3x2olb8yxo 2425188 2425187 2026-06-10T00:35:12Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2425188 wikitext text/x-wiki dt2rhwbfnpriqobhzgqaxpw074o1j11 2425189 2425188 2026-06-10T00:35:20Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2425189 wikitext text/x-wiki tpiiy0jbuuarei040lawv2ioq5m5ht7 2425190 2425189 2026-06-10T00:35:48Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2425190 wikitext text/x-wiki 9e7ks9di7x45u6ndn5rdqeby8b0in39 2425191 2425190 2026-06-10T00:35:57Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2425191 wikitext text/x-wiki hn9jva351luu98xxys0g0933be0923i 2425192 2425191 2026-06-10T00:36:46Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2425192 wikitext text/x-wiki f3g30wc067kf2r2tscbkymqjl38cdgy 2425193 2425192 2026-06-10T00:36:53Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2425193 wikitext text/x-wiki dm7814zm1hnzu9rpqfju826h6bn55pu 2425194 2425193 2026-06-10T00:37:43Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2425194 wikitext text/x-wiki doy9u34mjgup6efzsacmvo43ya3ligk 2425195 2425194 2026-06-10T00:37:51Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2425195 wikitext text/x-wiki sl3xvr0phf4uup67hh7hov5f197qgsj 2425196 2425195 2026-06-10T00:38:18Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425196 wikitext text/x-wiki 6z8ioawgjyv7ops8gaez12jvr6nzfre 2425197 2425196 2026-06-10T00:38:25Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2425197 wikitext text/x-wiki q5jq5i65gsb3ldu0xktsuls71m6rru7 2425198 2425197 2026-06-10T00:38:47Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage 2425198 wikitext text/x-wiki bmh385yvo4lq7vk9wvy9og1em62p4ks 2425199 2425198 2026-06-10T00:39:14Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2425199 wikitext text/x-wiki 8uflmef3djotf3of9zwhmkcacu17i6q 2425200 2425199 2026-06-10T00:39:20Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2425200 wikitext text/x-wiki hhcyhotr28z37xzmh65efjnt6b71xph 2425201 2425200 2026-06-10T00:39:49Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425201 wikitext text/x-wiki 3ttc7ied1z0rkrdkgijmt9135oj8w86 2425202 2425201 2026-06-10T00:39:56Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2425202 wikitext text/x-wiki 9g4d0a7f90twr6lyrn750iezt08vdff 2425203 2425202 2026-06-10T00:40:46Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2425203 wikitext text/x-wiki ef3xcdomh8llf70yx9meikkzjurob1i 2425204 2425203 2026-06-10T00:43:23Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage 2425204 wikitext text/x-wiki lbae1w3n59qp0dahdhtti9bdk40cken 2425206 2425204 2026-06-10T00:53:54Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync 2425206 wikitext text/x-wiki 1lmmxt21zy3vvjhqykywvsc6behf7lp 2425207 2425206 2026-06-10T00:54:16Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync 2425207 wikitext text/x-wiki 38yytf3ncmlqe9ej6qhq2on17yx3rrb 2425208 2425207 2026-06-10T01:00:32Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2009.codfw.wmnet with OS trixie 2425208 wikitext text/x-wiki n3feimu62bquaco6nsdzjg05ccbatb4 2425209 2425208 2026-06-10T01:11:31Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS trixie 2425209 wikitext text/x-wiki a5adlhazj4d9wnaempj5haprhwxedgf 2425210 2425209 2026-06-10T01:12:02Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1008 2425210 wikitext text/x-wiki 6vsrsko29gydreqzhtg2fc47x5jibho 2425211 2425210 2026-06-10T01:19:41Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.netbox 2425211 wikitext text/x-wiki d6bhmwzty5c1hoikl71aliz4djneot5 2425212 2425211 2026-06-10T01:23:50Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" 2425212 wikitext text/x-wiki kkqzhxtob24ymsmxbp683gh0rzjgdaf 2425213 2425212 2026-06-10T01:23:56Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" 2425213 wikitext text/x-wiki ppq1inhk76qzsnr526wxbzyfa4td7rc 2425214 2425213 2026-06-10T01:23:58Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2425214 wikitext text/x-wiki esqz4h86ko5vwz6ag1kn9hcfx56gqpe 2425215 2425214 2026-06-10T01:24:01Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2425215 wikitext text/x-wiki dyi5licihuesnlbvd1hn186db71joka 2425216 2425215 2026-06-10T01:24:03Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2425216 wikitext text/x-wiki kxcrw8l2rhovdvoouz4v139c2x468fc 2425217 2425216 2026-06-10T01:24:05Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1008 2425217 wikitext text/x-wiki g0ug0vhj5gdcvmylmc1m3eg26v9g71w 2425218 2425217 2026-06-10T01:24:48Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1008 2425218 wikitext text/x-wiki jsikeqlwkolv4jjyk6ky5e8ya15a8xa 2425219 2425218 2026-06-10T01:25:03Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1008 2425219 wikitext text/x-wiki das4x8k0ki8v73olkjccfyyzqxpo6jy 2425220 2425219 2026-06-10T01:43:06Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage 2425220 wikitext text/x-wiki qjlwo25ymn8960s37e4fcgdqgd28ccr 2425221 2425220 2026-06-10T01:43:54Z Stashbot 7414 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2425221 wikitext text/x-wiki tq2noet0bbrzzammjhvg3ot47i4qk0n 2425222 2425221 2026-06-10T01:44:16Z Stashbot 7414 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2425222 wikitext text/x-wiki l68c2jvraxindlk88afbfk88usz1ot3 2425223 2425222 2026-06-10T01:44:24Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2425223 wikitext text/x-wiki olb3tbtoz4fb4h64i9czxlenm08x8fw 2425224 2425223 2026-06-10T01:45:06Z Stashbot 7414 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2425224 wikitext text/x-wiki llmptsx7e7b7lyfw0g48ddyh3s0ysc2 2425225 2425224 2026-06-10T01:45:14Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2425225 wikitext text/x-wiki lx9k3g2nemyumaxmfp4698omw4c4k1b 2425226 2425225 2026-06-10T01:45:41Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2425226 wikitext text/x-wiki 1qf9clzwmutahvsmqkudygm46i774vi 2425227 2425226 2026-06-10T01:45:49Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2425227 wikitext text/x-wiki pqt4m76ezkp72lc12gg7frlkp453u4e 2425228 2425227 2026-06-10T01:46:37Z Stashbot 7414 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2425228 wikitext text/x-wiki 2tbm8lkid8h8004ylv62xhgxwwqj6k7 2425229 2425228 2026-06-10T01:46:44Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2425229 wikitext text/x-wiki m784ml8b8rj8ooo0n7fq5dbik9xldyl 2425230 2425229 2026-06-10T01:47:10Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2425230 wikitext text/x-wiki kjayivu1kz6285yoawkbajah3aszl5i 2425231 2425230 2026-06-10T01:47:18Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2425231 wikitext text/x-wiki hvg2vwktrqgyktgqt90j3nz96d8fd5g 2425232 2425231 2026-06-10T01:48:07Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2425232 wikitext text/x-wiki 6x1atbatiq6as2b0oy0th3sj3usq6cf 2425233 2425232 2026-06-10T01:48:15Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2425233 wikitext text/x-wiki tjh6gg4lxqswlcebaevvtimra7p150h 2425234 2425233 2026-06-10T01:49:05Z Stashbot 7414 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2425234 wikitext text/x-wiki o2nki9628g11wurxv62rkx16kyhsdgq 2425235 2425234 2026-06-10T01:49:13Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2425235 wikitext text/x-wiki 9v5y824rdhkowvdgrsnc83q6uz8viec 2425236 2425235 2026-06-10T01:49:41Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425236 wikitext text/x-wiki bh55phdg2nnwudw2hsx3pr0w2mrxza6 2425237 2425236 2026-06-10T01:49:48Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2425237 wikitext text/x-wiki dfsdbsljrjf9xxr7bubzbngn9rlox3p 2425238 2425237 2026-06-10T01:49:50Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage 2425238 wikitext text/x-wiki fw63x0u0ysn9h1sivqltcf2m6j90jfd 2425239 2425238 2026-06-10T01:50:37Z Stashbot 7414 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2425239 wikitext text/x-wiki qwjwq5enzfjkw0fxo5bpcc4cm297bi4 2425240 2425239 2026-06-10T01:50:43Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2425240 wikitext text/x-wiki 0ipu10357q9c074hqejhflzu53m6a7q 2425241 2425240 2026-06-10T01:51:12Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425241 wikitext text/x-wiki 2zsm3q9co0rx7foa4gd0brijiao7n8i 2425242 2425241 2026-06-10T01:51:19Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2425242 wikitext text/x-wiki thb83l2d8nol8baqmwbksxfb865xl6w 2425243 2425242 2026-06-10T01:52:09Z Stashbot 7414 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2425243 wikitext text/x-wiki 0q552mt0dlk68fmprru6g1uy66ioy50 2425247 2425243 2026-06-10T02:00:52Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2425247 wikitext text/x-wiki 6fzhjz8lv4oatvgtxfqxmkzctwpyxtv 2425248 2425247 2026-06-10T02:02:47Z Stashbot 7414 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync 2425248 wikitext text/x-wiki cgb0upmnex26zsii8o3yykdiwijfr1w 2425249 2425248 2026-06-10T02:03:08Z Stashbot 7414 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync 2425249 wikitext text/x-wiki nuyx8z7d54yz76tbhle0z6w0hmgfatl 2425251 2425249 2026-06-10T02:07:34Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1008.eqiad.wmnet with OS trixie 2425251 wikitext text/x-wiki hc2adr5ubzx85pg710auvrszkvmywoe 2425252 2425251 2026-06-10T02:07:39Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) 2425252 wikitext text/x-wiki pctgtr1k45hdfy5xx3i6b0k2wuhckrt 2425256 2425252 2026-06-10T05:41:25Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet 2425256 wikitext text/x-wiki 3imzvzw6ufp9dbu3jsqghh9tjumt34a 2425257 2425256 2026-06-10T05:42:53Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet 2425257 wikitext text/x-wiki g34hpgowo3x0c5prbfbvwvxxe4hn25h 2425258 2425257 2026-06-10T05:43:04Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet 2425258 wikitext text/x-wiki 2kwpymg7f7170jy4cfr9en9urz2jtc0 2425259 2425258 2026-06-10T05:43:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet 2425259 wikitext text/x-wiki 1klag3tw5bhux6sbkcd32rvjmwlx519 2425260 2425259 2026-06-10T06:45:17Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet 2425260 wikitext text/x-wiki ritht54bvx40ua795hql16ajg4jbb8p 2425261 2425260 2026-06-10T06:45:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet 2425261 wikitext text/x-wiki 2k8oabv3ysnxs9b34x3n862pxilbgej 2425264 2425261 2026-06-10T07:11:17Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425264 wikitext text/x-wiki nzhnuprcnfw84q2924jm1ifdqr161l9 2425265 2425264 2026-06-10T07:11:37Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425265 wikitext text/x-wiki 00z6yj5zg8zsv4u07w9yxyijmhahx3n 2425266 2425265 2026-06-10T07:12:02Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425266 wikitext text/x-wiki cwizwoq47trbbqxmsv4krhp08v1qfhq 2425267 2425266 2026-06-10T07:12:09Z Stashbot 7414 atsukoito: backporting extensions/Translate to wmf/1.47.0-wmf.5 and applying the config 2425267 wikitext text/x-wiki mh1a0s6gh8p27371p39b7fxitms6c9l 2425268 2425267 2026-06-10T07:14:49Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425268 wikitext text/x-wiki 9bwt5rw10p5cm2k8617ojfwhon6tjia 2425269 2425268 2026-06-10T07:15:35Z Stashbot 7414 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1299556|ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561|ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529|translate: adding separate read/write endpoints (T425377)]] 2425269 wikitext text/x-wiki 3emo8orisccb4b40lucvd6z35kzrsox 2425270 2425269 2026-06-10T07:16:38Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425270 wikitext text/x-wiki ikmn4zifqlzclphzclkot920rlfzrkz 2425271 2425270 2026-06-10T07:17:48Z Stashbot 7414 atsuko@deploy1003: atsuko: Backport for [[gerrit:1299556|ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561|ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529|translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be veri 2425271 wikitext text/x-wiki 9pf4zzlzo0fraksjvg4pi2jh05uwgrb 2425272 2425271 2026-06-10T07:20:30Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade 2425272 wikitext text/x-wiki bsx9ab8zja6ktjie2qh2p9wp920ry4u 2425273 2425272 2026-06-10T07:20:32Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2425273 wikitext text/x-wiki sc5ataet12esirijritruja82m2c54h 2425274 2425273 2026-06-10T07:21:18Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425274 wikitext text/x-wiki 73ncsnxmzq7244qgoy0f61mq6gsg55c 2425275 2425274 2026-06-10T07:21:29Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade 2425275 wikitext text/x-wiki 5if8fpftm6fz4s1m1d4u12cqqon2c78 2425276 2425275 2026-06-10T07:22:31Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1215.eqiad.wmnet with reason: Reimage 2425276 wikitext text/x-wiki lt7k4x6c0mqvrwerdglqaf5qtpoghi3 2425277 2425276 2026-06-10T07:23:40Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425277 wikitext text/x-wiki 7xnjbohawt8eq943uwnzyngx3jqu405 2425278 2425277 2026-06-10T07:23:49Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1215.eqiad.wmnet with OS trixie 2425278 wikitext text/x-wiki o9jvu6n583y7fb06d3mic2hwwigloub 2425279 2425278 2026-06-10T07:25:11Z Stashbot 7414 atsuko@deploy1003: atsuko: Continuing with deployment 2425279 wikitext text/x-wiki ayp11gcj6b2651kxhmmcjm9c30rbd5b 2425280 2425279 2026-06-10T07:29:39Z Stashbot 7414 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299556|ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561|ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529|translate: adding separate read/write endpoints (T425377)]] (duration: 14m 03s) 2425280 wikitext text/x-wiki 0gim73grv4qbe4xx64jrk5rqx7sh121 2425281 2425280 2026-06-10T07:33:44Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425281 wikitext text/x-wiki 7ccm3hyawwelcv1x7zi3a31rn2zn6vh 2425282 2425281 2026-06-10T07:37:45Z Stashbot 7414 javiermonton@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply 2425282 wikitext text/x-wiki 1j25oqcbzw8epu7bxjmm03h2tg6af4n 2425283 2425282 2026-06-10T07:38:12Z Stashbot 7414 javiermonton@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2425283 wikitext text/x-wiki 6rmo4tf3ez7wjjmlyf0w8ncgy0756rt 2425284 2425283 2026-06-10T07:39:18Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage 2425284 wikitext text/x-wiki b1erwkv3exqcamzhfbt1lmxas88zip7 2425285 2425284 2026-06-10T07:40:36Z Stashbot 7414 moritzm: installing openssl security updates 2425285 wikitext text/x-wiki mzgl2a4swausuutgce84eylumdd4g0f 2425286 2425285 2026-06-10T07:40:42Z Stashbot 7414 javiermonton@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply 2425286 wikitext text/x-wiki otak7pgtbokbq8umlw09abvn6q9a1gj 2425287 2425286 2026-06-10T07:41:03Z Stashbot 7414 javiermonton@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2425287 wikitext text/x-wiki bvok9iocup30rymf8uwwu4ub90ip1ir 2425288 2425287 2026-06-10T07:44:11Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage 2425288 wikitext text/x-wiki hl783jjmbdphwfw054hm9rvsea9py1x 2425289 2425288 2026-06-10T07:48:31Z Stashbot 7414 javiermonton@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply 2425289 wikitext text/x-wiki kbxiwm5gm62cmgun35olbsix1r9lhxm 2425290 2425289 2026-06-10T07:48:50Z Stashbot 7414 javiermonton@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2425290 wikitext text/x-wiki 7wa3cy2fbgbqfl84qnhdunvvgu78kun 2425291 2425290 2026-06-10T07:50:01Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425291 wikitext text/x-wiki 4wtpk7a383f8ihhpufpddg5fds6iim1 2425293 2425291 2026-06-10T07:51:00Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425293 wikitext text/x-wiki 4lrdtmai92562g63jlaymdv2fzqkex8 2425294 2425293 2026-06-10T07:51:42Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425294 wikitext text/x-wiki bzezxjkr60i183tpiyxvibe9pkkyg6s 2425295 2425294 2026-06-10T07:52:29Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425295 wikitext text/x-wiki 596rtlcnaaukbfcfuruoa7br9yjghnx 2425296 2425295 2026-06-10T07:52:45Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425296 wikitext text/x-wiki 952arcqxkaobc9m71076sueqrkgro0j 2425297 2425296 2026-06-10T07:53:41Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425297 wikitext text/x-wiki p3qhwknu6m8ezkijzbm6n77ifwploij 2425298 2425297 2026-06-10T07:55:53Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425298 wikitext text/x-wiki d7lsdryor0btl8d0783hsrycbgbxcpo 2425299 2425298 2026-06-10T07:56:19Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425299 wikitext text/x-wiki lt3cl0xxpc9mbbmsfkamyccnn1z26jt 2425300 2425299 2026-06-10T07:57:29Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425300 wikitext text/x-wiki 6iipndsmwmvppqfuqhblo07ra6xat6e 2425301 2425300 2026-06-10T07:57:49Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425301 wikitext text/x-wiki mrwuv47rkhfj64x0cakfy8r5xz6tp5o 2425303 2425301 2026-06-10T08:01:24Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1215.eqiad.wmnet with OS trixie 2425303 wikitext text/x-wiki omcjk6m4s2zj36rtsuit1o0dfodhorq 2425307 2425303 2026-06-10T08:04:58Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425307 wikitext text/x-wiki rgf4fadsbg24p1b4wxwzbw534uebuku 2425308 2425307 2026-06-10T08:05:30Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425308 wikitext text/x-wiki h6v7oapocwew2r2kju9vf6s9l38ledv 2425309 2425308 2026-06-10T08:06:09Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425309 wikitext text/x-wiki ogltt30au753sndq7l5syyy7muomoat 2425312 2425309 2026-06-10T08:07:52Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2425312 wikitext text/x-wiki dtsrgfqhj36l157z9urkkzf86fh6ozi 2425313 2425312 2026-06-10T08:08:37Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425313 wikitext text/x-wiki fwunlww0ucebza8x7jhytsyv6ytwrzq 2425314 2425313 2026-06-10T08:08:47Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2425314 wikitext text/x-wiki gse5faemrn4zuv8c4996nl1cjvu5jr6 2425315 2425314 2026-06-10T08:08:50Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425315 wikitext text/x-wiki 5bkc2suzirxft3cus2faz1kbe9932fc 2425316 2425315 2026-06-10T08:09:10Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425316 wikitext text/x-wiki k3l8z9n8j84pr2ba639ul7ku0l6ug83 2425318 2425316 2026-06-10T08:09:27Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425318 wikitext text/x-wiki 84zgr1n282gu8q8o43o89qjx6x1jy7q 2425319 2425318 2026-06-10T08:11:49Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425319 wikitext text/x-wiki dbsin2q11b02gkqggs6xkrvi0ifhgk2 2425320 2425319 2026-06-10T08:20:35Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425320 wikitext text/x-wiki 98zuek7ob437orzalkptpdd21m7rkvf 2425321 2425320 2026-06-10T08:29:10Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2425321 wikitext text/x-wiki a8e9w6hx0ocwn9z2gazclr11k7g52rp 2425322 2425321 2026-06-10T08:29:48Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2425322 wikitext text/x-wiki mthhgkjpg159v62k0bmqwc8blqvjkz3 2425324 2425322 2026-06-10T08:30:38Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425324 wikitext text/x-wiki 2azlmv0gd5jz19kuesqb0s0k9bz168x 2425327 2425324 2026-06-10T08:54:12Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425327 wikitext text/x-wiki kdj9ao575gribbuvdv5o8cerm28wjo9 2425328 2425327 2026-06-10T08:56:01Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425328 wikitext text/x-wiki doygvy35d7ajqwxpsu08fath1ntjnba 2425329 2425328 2026-06-10T09:00:29Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425329 wikitext text/x-wiki 6rbayqnuz5ehcsjth66z2gypw42uzjt 2425330 2425329 2026-06-10T09:00:41Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425330 wikitext text/x-wiki rkcr53p1o9b58tllzbdqxiakdl7d7w0 2425331 2425330 2026-06-10T09:02:29Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425331 wikitext text/x-wiki db736xlnaopn7hxfjr5o9a1ooflehbc 2425332 2425331 2026-06-10T09:02:37Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425332 wikitext text/x-wiki 6fcn18rjqbkh7zcbrlosrnwmddcbxai 2425333 2425332 2026-06-10T09:03:10Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425333 wikitext text/x-wiki 6dw3khen7lv50ld8ykuih1iwxt74ioy 2425334 2425333 2026-06-10T09:03:19Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425334 wikitext text/x-wiki f4eye2rqd77pf2vok24xh5d2pwgmyoi 2425335 2425334 2026-06-10T09:04:56Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425335 wikitext text/x-wiki dpaq2mwtd1vd9c942kdmicj77cd99ci 2425336 2425335 2026-06-10T09:05:03Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425336 wikitext text/x-wiki 08w4qgtkpeewctlngqaj3nyvbwesngq 2425337 2425336 2026-06-10T09:05:17Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425337 wikitext text/x-wiki dwidp5e2qxbkyfkqgqf8vgwdoswc09y 2425338 2425337 2026-06-10T09:05:30Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425338 wikitext text/x-wiki cce840jvi6rdgf7w5gmci1v1scw8qyj 2425339 2425338 2026-06-10T09:05:38Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2425339 wikitext text/x-wiki hkh3cxcyzsiuz6lwzh2lfedeh2e3bs8 2425340 2425339 2026-06-10T09:15:42Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2425340 wikitext text/x-wiki 2bnxi8101lwdid476o78rhv4wv08yca 2425341 2425340 2026-06-10T09:18:35Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to plain 2425341 wikitext text/x-wiki bojdwimmnb9jqhv13601ghpyjdr7g1t 2425342 2425341 2026-06-10T09:20:41Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus5003.eqsin.wmnet to plain 2425342 wikitext text/x-wiki gdi9c96pc8byqfgmechx952nq5eshp4 2425345 2425342 2026-06-10T09:22:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet 2425345 wikitext text/x-wiki 1tzw70sind8xifvs8yzgdydx21k9rjo 2425346 2425345 2026-06-10T09:23:20Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2425346 wikitext text/x-wiki tc4r30btmndwwvkieabqgwewtrtv9e6 2425347 2425346 2026-06-10T09:23:45Z Stashbot 7414 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2425347 wikitext text/x-wiki ckv9p8fdo1wrs9nbh6150k6c6sbbdks 2425351 2425347 2026-06-10T09:26:12Z Stashbot 7414 moritzm: upgrade routinator in eqiad to 0.15.2 T428456 2425351 wikitext text/x-wiki moq7kt6k1g6m6ijzuq1jl1khmvebhgd 2425356 2425351 2026-06-10T09:32:08Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2425356 wikitext text/x-wiki s240d3pg2c25lfcqqt3st0ciyz3hw36 2425357 2425356 2026-06-10T09:32:39Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary 2425357 wikitext text/x-wiki 60jo60835ze01uqe098d5ipnwzld9kd 2425358 2425357 2026-06-10T09:34:26Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox 2425358 wikitext text/x-wiki ftjttq6k1i2o6bkcolv4pu9sc9i0uht 2425359 2425358 2026-06-10T09:35:23Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox 2425359 wikitext text/x-wiki tbijmm7hkckwwoz77ej2i9h9m23vwy2 2425360 2425359 2026-06-10T09:47:31Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1300058|SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059|SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] 2425360 wikitext text/x-wiki jtuuhb9q792f52zpn3q8hg5u9woohl3 2425361 2425360 2026-06-10T09:49:33Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1300058|SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059|SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425361 wikitext text/x-wiki psiy2pclawtd8ptjact7ie0mipg0d5x 2425362 2425361 2026-06-10T09:52:36Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2425362 wikitext text/x-wiki 0vx9xoo34hf2efmbnz07w126lrrs1aq 2425363 2425362 2026-06-10T09:57:04Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300058|SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059|SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] (duration: 09m 32s) 2425363 wikitext text/x-wiki djdvzohhm673zuegd0adyqhl4pelj0z 2425365 2425363 2026-06-10T10:04:38Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300064|ProductionServices: reboot poolcounter1006.eqiad (T426736)]] 2425365 wikitext text/x-wiki ti0c4qh54tyymyfkuha21rs9j7bmh67 2425366 2425365 2026-06-10T10:06:57Z Stashbot 7414 blake@deploy1003: blake: Backport for [[gerrit:1300064|ProductionServices: reboot poolcounter1006.eqiad (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425366 wikitext text/x-wiki 3gmfiq9gb41uiezi6ffxd14ip2w2c3h 2425368 2425366 2026-06-10T10:07:35Z Stashbot 7414 blake@deploy1003: blake: Continuing with deployment 2425368 wikitext text/x-wiki aqzsjomn27tped5dmisjv7crjgrqppv 2425369 2425368 2026-06-10T10:12:25Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300064|ProductionServices: reboot poolcounter1006.eqiad (T426736)]] (duration: 07m 46s) 2425369 wikitext text/x-wiki csd8c87z413lx6cofcpcps7sg5vsqj3 2425370 2425369 2026-06-10T10:13:16Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet 2425370 wikitext text/x-wiki 4ntukmjkfouwn3d4fe0dw4azxtjmrzl 2425371 2425370 2026-06-10T10:14:32Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Security updates 2425371 wikitext text/x-wiki oaer04apjpm9xh1tehnismzu9npljmd 2425372 2425371 2026-06-10T10:14:34Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2425372 wikitext text/x-wiki i4l15reo5rpwf9qf27tongr1tjaediq 2425373 2425372 2026-06-10T10:14:39Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2425373 wikitext text/x-wiki ipwoby7iecvn72565vgam75ucngu20w 2425374 2425373 2026-06-10T10:14:42Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates 2425374 wikitext text/x-wiki hg442jqunlfq6muyabrsxnrkx7m7kuh 2425375 2425374 2026-06-10T10:17:00Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet 2425375 wikitext text/x-wiki av6oju7mwbssghajs0nhqabs3u1jv8c 2425376 2425375 2026-06-10T10:20:29Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2425376 wikitext text/x-wiki qqlknmdk6bx0ybl3dfjzdo5itpw1sjo 2425377 2425376 2026-06-10T10:21:02Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2425377 wikitext text/x-wiki tk34t9umwxu5aj1f60gmmsdgm1u3u6j 2425378 2425377 2026-06-10T10:21:09Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2425378 wikitext text/x-wiki hcake37qsrwf1gcalur0n0i2phx1f64 2425379 2425378 2026-06-10T10:21:31Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2425379 wikitext text/x-wiki kcf21rv0gbq0p1n9081ys04ox2z02xu 2425380 2425379 2026-06-10T10:21:37Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2425380 wikitext text/x-wiki c7uveef5gwrqkngy8p0wo4gfx888fkh 2425381 2425380 2026-06-10T10:21:40Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300072|ProductionServices: reboot poolcounter1007 (T426736)]] 2425381 wikitext text/x-wiki it5cx6xyxfvy45llzdd0xjg1oj9su0p 2425382 2425381 2026-06-10T10:21:53Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2425382 wikitext text/x-wiki e1a16h19kr2tkembldyi9sh6q3zgeb9 2425383 2425382 2026-06-10T10:21:56Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2425383 wikitext text/x-wiki qszeyc2q9wld2h6tvxuy5o6nd4gxwzu 2425384 2425383 2026-06-10T10:22:30Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2425384 wikitext text/x-wiki r4etuu3t70yae04zrh5c25h66ogkgs3 2425386 2425384 2026-06-10T10:23:46Z Stashbot 7414 blake@deploy1003: blake: Backport for [[gerrit:1300072|ProductionServices: reboot poolcounter1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425386 wikitext text/x-wiki of0r150zv0qz3n44da3d9i59iyv9x8j 2425387 2425386 2026-06-10T10:24:59Z Stashbot 7414 blake@deploy1003: blake: Continuing with deployment 2425387 wikitext text/x-wiki e43ozbetrbvysgdtjqwxh75z6up8nje 2425388 2425387 2026-06-10T10:27:11Z Stashbot 7414 jmm@cumin2002: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for sretest2009.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 2425388 wikitext text/x-wiki silg2ctgchlvlb77ohbuts1n1os6h76 2425389 2425388 2026-06-10T10:27:17Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet 2425389 wikitext text/x-wiki rvuzjhlgx2tvcmccuxogndvh580xr2v 2425390 2425389 2026-06-10T10:29:24Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300072|ProductionServices: reboot poolcounter1007 (T426736)]] (duration: 07m 45s) 2425390 wikitext text/x-wiki nvyvts4i39r8tefkvkyn182co77hx1a 2425391 2425390 2026-06-10T10:29:42Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet 2425391 wikitext text/x-wiki 3ybjzn0powgy9wlw7i5v1y7ku4345nh 2425392 2425391 2026-06-10T10:33:26Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet 2425392 wikitext text/x-wiki qm1lyr4isr4jgacn49iy9v0veivw38o 2425393 2425392 2026-06-10T10:34:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2425393 wikitext text/x-wiki pkt8gqka62psivt3b0hear6g8np3am4 2425394 2425393 2026-06-10T10:34:54Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425394 wikitext text/x-wiki 84iybo4kzzoq32s0yynt42gbf0k23i2 2425395 2425394 2026-06-10T10:35:47Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300082|ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] 2425395 wikitext text/x-wiki 14ukk616m6mnk2n3ctd91xy7pbshmac 2425397 2425395 2026-06-10T10:37:30Z Stashbot 7414 moritzm: failover Ganeti master in eqsin to ganeti5007 T428229 2425397 wikitext text/x-wiki ku242qg0lhrogg54qk2n7o1ow2v0d4d 2425398 2425397 2026-06-10T10:38:10Z Stashbot 7414 blake@deploy1003: blake: Backport for [[gerrit:1300082|ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425398 wikitext text/x-wiki jlc7lrnhxzebcpqhrolz1ua4fithah2 2425399 2425398 2026-06-10T10:38:39Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1152: Security updates 2425399 wikitext text/x-wiki tcsf5u6m6hekzquwioji5ndbr8xya52 2425400 2425399 2026-06-10T10:38:41Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2425400 wikitext text/x-wiki mp864ou6se55kykzkiimdi1rttzc4zx 2425401 2425400 2026-06-10T10:38:51Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2425401 wikitext text/x-wiki r4gc7lo3uuwatmq3pn2yespcd3x5cn9 2425402 2425401 2026-06-10T10:38:54Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: Security updates 2425402 wikitext text/x-wiki b9moyikkzswxpwxu5hrexncene9t3dj 2425403 2425402 2026-06-10T10:38:56Z Stashbot 7414 blake@deploy1003: blake: Continuing with deployment 2425403 wikitext text/x-wiki c6s9m732pu9if40lcfu0yqu7f6ultvv 2425404 2425403 2026-06-10T10:41:05Z Stashbot 7414 moritzm: installing nginx security updates 2425404 wikitext text/x-wiki offt2x8t85bff1uk48456fz9hx311yt 2425405 2425404 2026-06-10T10:43:29Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300082|ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] (duration: 07m 38s) 2425405 wikitext text/x-wiki nj9fg4hlvcdyqgnlryvgvw5sj0c470v 2425406 2425405 2026-06-10T10:43:41Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet 2425406 wikitext text/x-wiki 3rh475asmwki99jb25oafxo6rrcsp6g 2425407 2425406 2026-06-10T10:45:43Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2425407 wikitext text/x-wiki 8scvcmz9szfbt3go0j0ur2hef9w19mf 2425408 2425407 2026-06-10T10:46:16Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425408 wikitext text/x-wiki 8oytkuvsbhwr9br3xr39lizj2vwcvgb 2425409 2425408 2026-06-10T10:46:47Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2425409 wikitext text/x-wiki q3nvd90q2etsgtx374lryeohdoutlnv 2425410 2425409 2026-06-10T10:47:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2425410 wikitext text/x-wiki tm8i4l1q9yqp9ai02ihe5k11wq8n7kg 2425411 2425410 2026-06-10T10:47:38Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet 2425411 wikitext text/x-wiki pdbjzfqlcpe1tf27awgqsyfblv9ydvz 2425412 2425411 2026-06-10T10:49:26Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300087|ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] 2425412 wikitext text/x-wiki k7d9c9hgvxp09gwck99b1jv9k19efvh 2425413 2425412 2026-06-10T10:51:26Z Stashbot 7414 blake@deploy1003: blake: Backport for [[gerrit:1300087|ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425413 wikitext text/x-wiki sm6nfh26bgczgz36idvr98tgs5bmmr1 2425414 2425413 2026-06-10T10:51:34Z Stashbot 7414 moritzm: remove ganeti5004 from eqsin cluster for reimage T428229 2425414 wikitext text/x-wiki td1kzx6lxz9nf7sm4yd4rpkdti92hal 2425415 2425414 2026-06-10T10:51:46Z Stashbot 7414 blake@deploy1003: blake: Continuing with deployment 2425415 wikitext text/x-wiki 6zs2ws04dm4994bjenjjxrc8snv1gs5 2425416 2425415 2026-06-10T10:56:03Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300087|ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] (duration: 06m 42s) 2425416 wikitext text/x-wiki ku0nn8psf779xskzecylty4zw96ipt1 2425417 2425416 2026-06-10T10:56:14Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet 2425417 wikitext text/x-wiki mfg9fh9sdhgbf7pdsqijzto9ihtj3ps 2425418 2425417 2026-06-10T10:56:35Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply 2425418 wikitext text/x-wiki cnmxqf9nxy0k9n3wh7svca1hrinpmru 2425419 2425418 2026-06-10T10:56:44Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply 2425419 wikitext text/x-wiki atja3tc8mvfthl4iur7ym51ou8vkhk0 2425420 2425419 2026-06-10T10:56:52Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply 2425420 wikitext text/x-wiki njxpjhl50eg8v5kufpln9e77tqtjlm0 2425421 2425420 2026-06-10T10:57:09Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply 2425421 wikitext text/x-wiki 735ahrvgmv8yfxq7vj63gux57ibxi9b 2425422 2425421 2026-06-10T10:57:14Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply 2425422 wikitext text/x-wiki 0p2kv74mkjizyb75p2y70zlv6ix6agl 2425423 2425422 2026-06-10T10:57:32Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply 2425423 wikitext text/x-wiki nvge3oapeqcea52kej7vdp3nf5j51xp 2425424 2425423 2026-06-10T11:00:00Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet 2425424 wikitext text/x-wiki g1tlrernpn12fe1b5fbogxqza5yufyd 2425425 2425424 2026-06-10T11:01:53Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300092|ProductionServices: re-add poolcounter2006 (T426736)]] 2425425 wikitext text/x-wiki h77optwaj4qn6fplx6op929kre2nx7k 2425426 2425425 2026-06-10T11:02:24Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2425426 wikitext text/x-wiki 79coocuy81fqak0w138h157j56e84bw 2425427 2425426 2026-06-10T11:03:19Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2425427 wikitext text/x-wiki odhb82ql3qrezpj824c2eiidreemmew 2425428 2425427 2026-06-10T11:04:09Z Stashbot 7414 blake@deploy1003: blake: Backport for [[gerrit:1300092|ProductionServices: re-add poolcounter2006 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2425428 wikitext text/x-wiki nv8inomb1vgi4177bpr2tz0qfmefber 2425429 2425428 2026-06-10T11:04:31Z Stashbot 7414 blake@deploy1003: blake: Continuing with deployment 2425429 wikitext text/x-wiki knnq434hkzeoaj4hdf3u0a1mguhowzc 2425430 2425429 2026-06-10T11:08:47Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300092|ProductionServices: re-add poolcounter2006 (T426736)]] (duration: 06m 55s) 2425430 wikitext text/x-wiki sxxmaoehlmx1h5bp5ok5yhb832rs9bd 2425431 2425430 2026-06-10T11:09:03Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Security updates 2425431 wikitext text/x-wiki letm4i7kf4lg908yzm1sozxejs5d0fh 2425432 2425431 2026-06-10T11:09:05Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2425432 wikitext text/x-wiki ngh1038qdxig1zrf56dszm44560h941 2425433 2425432 2026-06-10T11:09:11Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2425433 wikitext text/x-wiki 3lrx7tnc2dfonuwauk1xvupvhk5ke73 2425434 2425433 2026-06-10T11:09:14Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates 2425434 wikitext text/x-wiki 9i4ydz98y6bg8puuqt12cv0fqx7qxgx 2425436 2425434 2026-06-10T11:15:32Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply 2425436 wikitext text/x-wiki fnexjumv2f0ciys0603v83z2r9frty6 2425437 2425436 2026-06-10T11:15:47Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply 2425437 wikitext text/x-wiki pwmypfwcrw2e5gfhlzmphr4gabov7bt 2425438 2425437 2026-06-10T11:15:51Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply 2425438 wikitext text/x-wiki 8d8ekwk9s6cuzmi0ypksg5g7r2z8am3 2425439 2425438 2026-06-10T11:16:03Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply 2425439 wikitext text/x-wiki rrr3yd5hzk47czgh0tl9zwofxfdy75q 2425440 2425439 2026-06-10T11:23:39Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply 2425440 wikitext text/x-wiki f2efrig5o1glrtfu1d1g49vpxekzid7 2425441 2425440 2026-06-10T11:23:42Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply 2425441 wikitext text/x-wiki 0f88nww8z6vf514cs5mi9r0ejz3s1e2 2425442 2425441 2026-06-10T11:23:47Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply 2425442 wikitext text/x-wiki rnh9bu821ge7u3z7prrbt5sagdq0q9m 2425443 2425442 2026-06-10T11:23:53Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply 2425443 wikitext text/x-wiki 6859wugwvavgks3wkgnm9ijj8cj6l14 2425444 2425443 2026-06-10T11:27:22Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2425444 wikitext text/x-wiki 6holk8vf8kk8ov7codg84ewmji58uto 2425445 2425444 2026-06-10T11:27:44Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2425445 wikitext text/x-wiki at486xm5bklkpsya2zi8h866vy2n1bq 2425447 2425445 2026-06-10T11:30:10Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply 2425447 wikitext text/x-wiki ihm5csfnk9bm71w036vcgzz2rq7tff7 2425448 2425447 2026-06-10T11:30:37Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply 2425448 wikitext text/x-wiki h30u9t1yghvfd8akjx3bgvii2hq2yeu 2425449 2425448 2026-06-10T11:30:51Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply 2425449 wikitext text/x-wiki 2dgsb4qkzcnwg2qfh3zydhxvr79sl48 2425450 2425449 2026-06-10T11:31:21Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply 2425450 wikitext text/x-wiki qfwrkabrwgrdu6s57z17ufvs4oiqb0p 2425451 2425450 2026-06-10T11:33:09Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1151: Security updates 2425451 wikitext text/x-wiki mx4a92w1pqbsi8bgxie9wovww48nm1h 2425452 2425451 2026-06-10T11:33:11Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2425452 wikitext text/x-wiki 3afpjhv1yxxizbkt9mmj1jzr3d4qsrp 2425453 2425452 2026-06-10T11:33:23Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2425453 wikitext text/x-wiki jr8snabljlsorwa77r7gdisj1c7scjt 2425454 2425453 2026-06-10T11:33:25Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1151: Security updates 2425454 wikitext text/x-wiki 7urgn94qtgay67vt6o8on497z25q8t4 2425455 2425454 2026-06-10T11:34:02Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . 2425455 wikitext text/x-wiki 8t0bzlry3gczn6cl0y1oaziddzti8cx 2425456 2425455 2026-06-10T11:34:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5004.eqsin.wmnet with OS bookworm 2425456 wikitext text/x-wiki k3n57l1q5kl96kirk3ob3f5dd812unr 2425457 2425456 2026-06-10T11:34:38Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . 2425457 wikitext text/x-wiki 1mxvlukep6xnhmty8a0u28nqjame45g 2425458 2425457 2026-06-10T11:34:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5004 2425458 wikitext text/x-wiki 4hlce9vuzy34pa2ih4qgpogzy0twl9k 2425459 2425458 2026-06-10T11:35:14Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . 2425459 wikitext text/x-wiki rk2bpsyvaytw06s1hs5asu5vihmcall 2425461 2425459 2026-06-10T11:38:06Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2425461 wikitext text/x-wiki bxxfu2puf2y4lr15z8rinkgufqnukiq 2425462 2425461 2026-06-10T11:45:32Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2425462 wikitext text/x-wiki 2qvsxnw2h9dfsekv883swywnaa2yxcq 2425463 2425462 2026-06-10T11:45:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet 2425463 wikitext text/x-wiki qn7dm4z1kmgpu3lt288uml04f68sa4p 2425464 2425463 2026-06-10T11:46:12Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet 2425464 wikitext text/x-wiki ip1hahhbbpgzogxn01a2sbbqgiu7yfq 2425465 2425464 2026-06-10T11:48:28Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS trixie 2425465 wikitext text/x-wiki r9misg2egw69ye0c6keb2ve7qxs1sw5 2425466 2425465 2026-06-10T11:49:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2425466 wikitext text/x-wiki enli0574hileseavf6fbub19vuqse9c 2425467 2425466 2026-06-10T11:49:21Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" 2425467 wikitext text/x-wiki hd59qqlozsoa06hqu3kuy4pps5aju3g 2425468 2425467 2026-06-10T11:49:26Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" 2425468 wikitext text/x-wiki fq0ayi35om70cuqfi27fa75qubm7ah5 2425469 2425468 2026-06-10T11:49:28Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2425469 wikitext text/x-wiki kpc03tcl0646m911tkllvai3tlvwi0b 2425470 2425469 2026-06-10T11:49:30Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors 2425470 wikitext text/x-wiki 1yw4i4ie02nz2aq4rdizn6mus4rxl3m 2425471 2425470 2026-06-10T11:49:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors 2425471 wikitext text/x-wiki guvaxr8dt9weldo0mot5do0dwvkeh9o 2425472 2425471 2026-06-10T11:49:35Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5004 2425472 wikitext text/x-wiki 3x6acki6i19xljm1jv531ybxzs3wmnt 2425473 2425472 2026-06-10T11:49:37Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet 2425473 wikitext text/x-wiki kzoha3395r3ss56tbooem2roysr74de 2425474 2425473 2026-06-10T11:49:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet 2425474 wikitext text/x-wiki mysrxuec4o9qe2bobz996ugbhgbb8kp 2425475 2425474 2026-06-10T11:51:19Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5004 2425475 wikitext text/x-wiki a7mnzyxmvtx8p4bj2kei24e42y4tdkl 2425476 2425475 2026-06-10T11:51:21Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5004 2425476 wikitext text/x-wiki 48jznf8lmyv28ly6mcbjwqugz1qw2p9 2425477 2425476 2026-06-10T11:53:21Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2170.codfw.wmnet with OS trixie 2425477 wikitext text/x-wiki b1sj65fox8h6dwbm2cu6yb8zs3l477p 2425478 2425477 2026-06-10T11:57:05Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300104|Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102|Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643|wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] 2425478 wikitext text/x-wiki cji8sy3iyuc60fbw7mo8nqgojnvawku 0 24216 2424993 2419525 2026-06-09T16:28:24Z BCornwall-WMF 31521 /* Pooling/depooling a server from all the related services */ Remove cp node mention, we don't use backend varnish any more 2424993 wikitext text/x-wiki e2l4fy6v684nji8ptr0yattfsnw1qpt 2424995 2424993 2026-06-09T16:28:59Z BCornwall-WMF 31521 /* Depool all nodes in a specific datacenter */ Remove unnecessary heading 2424995 wikitext text/x-wiki 85lj7ky2h9ehxb8jeucs0f5khlz9n9a 2424999 2424995 2026-06-09T16:30:49Z BCornwall-WMF 31521 Replace (outdated) enumerated list of services in the opening paragraph with just some examples 2424999 wikitext text/x-wiki d00xuebdf9pxzchyag2fsojnj5nav2d 0 29155 2424992 2380989 2026-06-09T16:26:31Z BCornwall-WMF 31521 Duplicated information, just redirect 2424992 wikitext text/x-wiki ewkm5cy9c8czur43sam5bwp5eciuvo5 498 30942 2425006 2424026 2026-06-09T16:35:56Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.upgrade_mons (T428385) 2425006 wikitext text/x-wiki rmqj8246o7be8xpfmebn074myti06wb 2425023 2425006 2026-06-09T16:59:30Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.upgrade_mons (exit_code=0) 2425023 wikitext text/x-wiki t54dsc8ys6xhe0wytorevozavxw7tuz 2425034 2425023 2026-06-09T17:05:05Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.upgrade_osds (T428385) 2425034 wikitext text/x-wiki 6yvbyrch6dth9n35ctkgz4eg65bxf4l 2425040 2425034 2026-06-09T17:35:04Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.upgrade_osds (exit_code=0) 2425040 wikitext text/x-wiki dvl1cdp8g9i2tw15wf8tgpdn2x9wlqj 0 440819 2424790 2397203 2026-06-09T12:35:34Z BRouberol-WMF 39141 2424790 wikitext text/x-wiki s4x8tmm5cj3eeayxbdqjy42ojqduliv 12 440881 2424919 2401078 2026-06-09T14:41:28Z Sebastian Berlin (WMSE) 7884 /* Container images */ Remove link to section that is no longer around. 2424919 wikitext text/x-wiki tt8x4ot4w8i40ev0at2v828p5z89hys 0 442361 2424947 1790301 2026-06-09T15:35:12Z BCornwall-WMF 31521 Update redirect 2424947 wikitext text/x-wiki 0vvsi6x7353f45y6o9g8pkltxmbmg4u 498 443628 2425050 2424272 2026-06-09T17:59:26Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225356702 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e) 2425050 wikitext text/x-wiki 6fcjfmrj7owzgpkzvo6ae05gju6wvci 498 446211 2425367 2421526 2026-06-10T10:07:00Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27268776225 (https://github.com/cluebotng/component-configs/commits/83d2cb83b77ce82739917e8a6f69d2fa261a35f0) 2425367 wikitext text/x-wiki c48ijo5ipfulp1q4s8s1d7nxsjs875b 0 447862 2425396 2424276 2026-06-10T10:36:05Z FCeratto-WMF 39104 Add table 2425396 wikitext text/x-wiki 8ekmajvtj48cddv3qkq5kgkc3enc6uy 2425435 2425396 2026-06-10T11:12:53Z FCeratto-WMF 39104 /* Using the dbtools scripts to update/reboot multiple hosts */ 2425435 wikitext text/x-wiki g69gi8dabwgu4rk09t9m97m0m5rhygl 2425446 2425435 2026-06-10T11:28:00Z FCeratto-WMF 39104 Update summary 2425446 wikitext text/x-wiki 81vfsesqh2a6izqki9g1x35b1f41f93 0 449160 2425169 2424642 2026-06-10T00:01:42Z Dexbot 30554 Bot: Updating the report 2425169 wikitext text/x-wiki n3djcelqittiqdcsn2xmtbp4wo3e227 0 450789 2424840 2424705 2026-06-09T13:20:35Z Lucas Werkmeister (WMDE) 10037 /* Benchmarking and load testing in production */ rephrase for clarity (you’re not actually targeting a single pod) and add an alternative that (I think?) targets mostly a single “machine” (are there multiple pods in one mw-experimental worker VM?) but has to work around [[phabricator:T428605|T428605]] 2424840 wikitext text/x-wiki ekhkfkc6e3o2xb8nvxc9zt0nfk6drq1 498 452890 2425051 2424274 2026-06-09T17:59:34Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27225356873 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e) 2425051 wikitext text/x-wiki b6hytcmhnyeknyedw2a2703rh40b7ie 116 453906 2424811 2424474 2026-06-09T12:51:30Z Gitlabaccountapprovalbot 37332 @ekrem was approved. 2424811 wikitext text/x-wiki g6i6tzolj9rwkh035dcv0gff7mze9sl 2424894 2424811 2026-06-09T14:03:26Z Gitlabaccountapprovalbot 37332 @sjones-ctr was approved. 2424894 wikitext text/x-wiki thy2x52ql7uygs3h67ed6y48210uzxk 2424911 2424894 2026-06-09T14:21:23Z Gitlabaccountapprovalbot 37332 @linkgenetic was approved. 2424911 wikitext text/x-wiki a7ba112hwe585cdfhzcgwcnj2bbho4q 2425263 2424911 2026-06-10T07:03:22Z Gitlabaccountapprovalbot 37332 @dmiranda was approved. 2425263 wikitext text/x-wiki 4yx9jw5si1u7jo271zihda0456ocjcw 0 454312 2424794 2424652 2026-06-09T12:39:31Z AYounsi (WMF) 10040 /* Provisioning */ 2424794 wikitext text/x-wiki fqw1i4utdx970nvtfp2w4qtn6x062cw 2424795 2424794 2026-06-09T12:39:55Z AYounsi (WMF) 10040 /* Decommissioning */ 2424795 wikitext text/x-wiki mx408jb4o6fkt0dm8werh1setr8jlvv 0 455513 2424929 2395389 2026-06-09T15:07:00Z MFossati (WMF) 29357 add Image Browsing 2424929 wikitext text/x-wiki 6f3evagbvd2vfh5kkmt1z729g0sqa0c 0 456526 2425205 2395053 2026-06-10T00:51:08Z Daimona Eaytoy 11462 /* Manually deleting a failed Job */ +alternative for restricted members, from [[phab:T428301#12002531]] 2425205 wikitext text/x-wiki qvcf3j1dqkvszgp4jlzmau4zfr1zy07 0 458636 2424890 2290544 2026-06-09T14:00:11Z BRouberol-WMF 39141 2424890 wikitext text/x-wiki 2cgxmgcvb6n1nwtlsmvx4urvozpkbf1 2424912 2424890 2026-06-09T14:25:24Z BRouberol-WMF 39141 /* Perform a rolling reboot of the cluster */ 2424912 wikitext text/x-wiki b052d2vfnyxded9lhm1hvntx4ee3kxl 2424917 2424912 2026-06-09T14:40:02Z BRouberol-WMF 39141 /* Ceph Cluster Upgrades */ 2424917 wikitext text/x-wiki s91bjdqrk69w5mb43fovsmigqym8quy 0 458638 2425343 2316441 2026-06-10T09:22:07Z Tgr (WMF) 39134 /* How does it work? */ 2425343 wikitext text/x-wiki 2anfkxct7eh74y4d23c4qnjx6hl4mbz 2425344 2425343 2026-06-10T09:22:21Z Tgr (WMF) 39134 /* How does it work? */ 2425344 wikitext text/x-wiki 4ojjc109idh72m0vh81rzk1urpyxnpp 0 459115 2425144 2422557 2026-06-09T21:07:56Z AndrewBogott (WMF) 51047 2425144 wikitext text/x-wiki dmks2c9mr0iv3xrw2p8hxxkv5q2x5om 2425145 2425144 2026-06-09T21:13:23Z BryanDavis 1604 lang=shell-session instead of lang=bash 2425145 wikitext text/x-wiki r0yfvqfu6rztpcqceqka59lswyvemfv 2425147 2425145 2026-06-09T21:18:10Z AndrewBogott (WMF) 51047 /* Compatibility matrix */ 2425147 wikitext text/x-wiki c6zowj0p5hzwrhbkdqd6hsvgx5je4e3 0 459138 2424744 2422756 2026-06-09T11:59:17Z WBrown (WMF) 37696 /* mediawiki-config */ + 2424744 wikitext text/x-wiki atjzeqa6rc5mnp6ijlyzd79z3axcv1a 498 459244 2425049 2424270 2026-06-09T17:58:56Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225357080 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e) 2425049 wikitext text/x-wiki 6hqv7azr3fo0fapgdalqpydsczzallx 0 459290 2425047 2424398 2026-06-09T17:54:43Z SNwachukwu (WMF) 39521 Add new item to next deployment train 2425047 wikitext text/x-wiki tfetho5sqxywa8r6ipehvmf85x726wa 2425048 2425047 2026-06-09T17:55:12Z SNwachukwu (WMF) 39521 2425048 wikitext text/x-wiki qawu0hlugxecyh6otn98uikgtirhlfy 0 459699 2425152 2423648 2026-06-09T21:39:04Z Stashbot 7414 eileen: civicrm upgraded from 6b61d3a5 to d7113e04 2425152 wikitext text/x-wiki m2jmkzrgvilodqg3ub8665m5jzc8dpy 2425460 2425152 2026-06-10T11:36:54Z Stashbot 7414 jgleeson: SmashPig upgraded from f6b24f7f to e82c2c5f 2425460 wikitext text/x-wiki dudun9pcqrf6pagghy69xtrdwz1k270 0 459920 2424934 2424654 2026-06-09T15:25:28Z Effie Mouzeli (WMF) 12880 /* Deploying changes */ 2424934 wikitext text/x-wiki jom6upwjg2i7bkz4v1tymr6w98gyuld 0 459935 2425348 2390796 2026-06-10T09:24:25Z Tgr (WMF) 39134 2425348 wikitext text/x-wiki drsxuu1n1bx048xhrdenfsd3e4dzlna 2425352 2425348 2026-06-10T09:27:24Z Tgr (WMF) 39134 /* Wikimedia software that uses SUL */ 2425352 wikitext text/x-wiki 1ylzyipvnvkr4mbgph6o6hjhkbxynx0 2425353 2425352 2026-06-10T09:27:45Z Tgr (WMF) 39134 2425353 wikitext text/x-wiki 14ibtejtuhohehhlpxs45ikm695yxcm 2 460039 2425354 2413631 2026-06-10T09:28:45Z Atsuko-WMF 54082 trying translate 2425354 wikitext text/x-wiki ozhy07w6smyrmcl26rpj5j988cg0p14 2425355 2425354 2026-06-10T09:29:33Z Atsuko-WMF 54082 Undid revision [[Special:Diff/2425354|2425354]] by [[Special:Contributions/Atsuko-WMF|Atsuko-WMF]] (translate didn't work) 2425355 wikitext text/x-wiki k88p32jf0ci57lwfbq11zidsdq5g72d 0 460195 2424909 2416912 2026-06-09T14:19:56Z AYounsi (WMF) 10040 2424909 wikitext text/x-wiki 3eq8w4z1skczrglv103hhyk23fkysmk 0 460305 2424972 2424252 2026-06-09T16:05:55Z Filippo Giunchedi 2939 Minimally expand, more details still TODO 2424972 wikitext text/x-wiki ra1b6hjbbx5iwlqpkvj5yn96o2ohryi 0 460310 2424749 2424634 2026-06-09T12:11:23Z BWojtowicz-WMF 42268 Update Enabling gRPC in a model server section 2424749 wikitext text/x-wiki a0lc8ceillhbe3ckbl68b3qimph569p 2424750 2424749 2026-06-09T12:12:00Z BWojtowicz-WMF 42268 2424750 wikitext text/x-wiki 590gslv3cdybcv63laxcn61uij7w374 2425292 2424750 2026-06-10T07:50:17Z BWojtowicz-WMF 42268 Update Cluster and networking configuration 2425292 wikitext text/x-wiki c06hl2tl8qn7tb5czuuin5fd48zrlzc 2425302 2425292 2026-06-10T08:00:41Z BWojtowicz-WMF 42268 2425302 wikitext text/x-wiki awyk7dgwq2y7z15whu36p0s8gd7wt9j 2425304 2425302 2026-06-10T08:01:25Z BWojtowicz-WMF 42268 2425304 wikitext text/x-wiki sty6dqaj9tfqioy0q02pba86nldra50 2425305 2425304 2026-06-10T08:02:09Z BWojtowicz-WMF 42268 2425305 wikitext text/x-wiki 618anomzv09913ipwavcutgjhufwfev 2425306 2425305 2026-06-10T08:04:55Z BWojtowicz-WMF 42268 2425306 wikitext text/x-wiki 0lj7fl7hcgapu4r1iptoxqjjyplt0fh 2425310 2425306 2026-06-10T08:06:33Z BWojtowicz-WMF 42268 2425310 wikitext text/x-wiki qipwlkwgzz34yl3dnn0i10c0rjmcyj4 2425311 2425310 2026-06-10T08:07:41Z BWojtowicz-WMF 42268 2425311 wikitext text/x-wiki suhgxi8les1h0hik539oinvslpxxo3g 2425317 2425311 2026-06-10T08:09:21Z BWojtowicz-WMF 42268 2425317 wikitext text/x-wiki i1v4es30kjvxam8d2fsc454v8f0rvho 0 460311 2425262 2424689 2026-06-10T06:53:36Z JMeybohm (WMF) 16709 /* Deploying to other clusters */ Link to mediawiki.org for SIG meeting protocols 2425262 wikitext text/x-wiki b563d0ypgtiihirwyhg1ef81jxxx9wa 3 460313 2424860 2026-06-09T13:35:45Z StrikerBot 8475 Welcome to Toolforge! 2424860 wikitext text/x-wiki 7asd85j7i2yyzc2eiw347rcqrdtwwpq 498 460314 2424940 2026-06-09T15:32:26Z Stashbot 7414 bd808: Added damian as project member (T428432) 2424940 wikitext text/x-wiki ovi3v1yh3byktcl2dvwfyy3gg77jgwt 498 460315 2424943 2026-06-09T15:33:20Z Stashbot 7414 bd808: Added damian as project member (T428432) 2424943 wikitext text/x-wiki 1043vozqd6pppjm8leyrnait2dlqm9x 1 460316 2425150 2026-06-09T21:28:24Z BCornwall-WMF 31521 /* Merge with Robot policy? */ new section 2425150 wikitext text/x-wiki 9ic9rbaazwu0k0l2wl0fu1lnpktwgcn 2 460317 2425244 2026-06-10T01:53:46Z ASarabadani (WMF) 39030 Created page with "Consistent hashing in MediaWiki powers distribution of parser cache entries and storage of main stash (and soon MediaWiki's lock manager) making this concept quite important and vital in understanding of our system architecture but unfortunately it is not well understood. Specially given the fact that it is similar to the concept of [[:en:Consistent_hashing#Basic_technique|Hash Rings]] but not exactly the same, it has caused some misunderstandings too. This essay tries..." 2425244 wikitext text/x-wiki tt39rjxfjqivnu47vw0e5z8k9j1f3n6 2425245 2425244 2026-06-10T01:54:21Z ASarabadani (WMF) 39030 /* Basic idea */ 2425245 wikitext text/x-wiki hnzzo6x71kqgvz9owei8ej6rrh454c2 2425246 2425245 2026-06-10T01:56:18Z ASarabadani (WMF) 39030 /* Basic idea */ 2425246 wikitext text/x-wiki 6j9xzfb6ozsacrgawdg3rsfglxhur3s 2425250 2425246 2026-06-10T02:03:25Z ASarabadani (WMF) 39030 /* Downsides */ 2425250 wikitext text/x-wiki iu1poau02p7ovlmdl46a9pt5t1twye0 2425253 2425250 2026-06-10T03:33:09Z Krinkle 44 add minor formatting, change example to be more mixed up and not look like a neatly ordered ring so that the example is not pc3->pc4 but pc3->pc1, more clear that it doesn't go to a global neighbour. 2425253 wikitext text/x-wiki 4rkwy9yebeqppgld4eeu0mmikkufiym 2425254 2425253 2026-06-10T03:35:15Z Krinkle 44 /* Implications */ 2425254 wikitext text/x-wiki nixkzyma3hdzz3wcbq4oevgmiiehde5 0 460318 2425323 2026-06-10T08:30:22Z Atsuko-WMF 54082 New page 2425323 wikitext text/x-wiki amqswb180vybz28x52uo20wn2fmfc73 2425325 2425323 2026-06-10T08:30:48Z Atsuko-WMF 54082 crlf 2425325 wikitext text/x-wiki 4rqiutq4wtmo9oldajrwc6su1g11u7b 2425326 2425325 2026-06-10T08:35:25Z DCausse (WMF) 5604 link to cirrussearch opensearch cluster 2425326 wikitext text/x-wiki cwxb90j4fzd7igqi6171di4wb817h0p 2425364 2425326 2026-06-10T10:01:11Z Atsuko-WMF 54082 examples 2425364 wikitext text/x-wiki bn1zmseoweg3j0s01lfu58x4qg7lsb7 2425385 2425364 2026-06-10T10:23:23Z Atsuko-WMF 54082 dash 2425385 wikitext text/x-wiki 34xllw2rvtw7lhhn9uqq3b7o88gcpno 0 460319 2425349 2026-06-10T09:25:11Z GGoncalves-WMF 42848 Publish report for the Nov/2025 incident. 2425349 wikitext text/x-wiki r0hotetyhjl774hneeng8rsfyma80f5 2425350 2425349 2026-06-10T09:25:25Z GGoncalves-WMF 42848 2425350 wikitext text/x-wiki op3uwr1xycpi7v4uazhv98wfd60ashp