Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.4 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 1516 2422031 2419725 2026-06-02T01:03:22Z Stashbot 7414 Krinkle: /cs flags #cvn-sw Divinations voiced 2422031 wikitext text/x-wiki n9bg1z6zezjcsf60a8pnrhdbo9n1rkx 498 3086 2421970 2420565 2026-06-01T20:00:39Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2421970 wikitext text/x-wiki fugzof5j3g3y54yl5ksrw74nphsrcva 2421977 2421970 2026-06-01T20:13:17Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2421977 wikitext text/x-wiki ps2r2zb2a9kmzz2shvdux4f8sm50sef 2421978 2421977 2026-06-01T20:13:48Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2421978 wikitext text/x-wiki dkjhsyqr4s525bvufrwrseure0ja2t2 2421984 2421978 2026-06-01T20:22:21Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2421984 wikitext text/x-wiki 8h1bnf0p7eqzeox764qcgtztarc8lib 2422033 2421984 2026-06-02T01:33:50Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2422033 wikitext text/x-wiki 2qjjzhiyyurle5rg9xjjkmz794e0z90 2422035 2422033 2026-06-02T01:46:15Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2422035 wikitext text/x-wiki e3mxys98pradj96bzhgvc4pkccstftt 2422219 2422035 2026-06-02T10:33:58Z Stashbot 7414 taavi: migrate acme-chief, puppetdb to nftables T427799 2422219 wikitext text/x-wiki 8rpngn1qusi92qkc78lb6d6idoy1tr3 0 4108 2421689 2421579 2026-06-01T12:21:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1295875]] to Monday, June 01 UTC afternoon backport window 2421689 wikitext text/x-wiki 6l5edboqac3w6rgw6jdv4bzuqlgzil4 2421773 2421689 2026-06-01T13:39:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1295502]] to Tuesday, June 02 UTC afternoon backport window 2421773 wikitext text/x-wiki cjf2mvgs4nygoh900rds3d2ys67iu1h 2421774 2421773 2026-06-01T13:40:10Z ScheduleDeploymentBot 37566 Add [[gerrit:1283106]] to Tuesday, June 02 UTC afternoon backport window 2421774 wikitext text/x-wiki 5oc4e2c8kywm9uiw1esp1e0iiu5qls5 2421775 2421774 2026-06-01T13:40:54Z ScheduleDeploymentBot 37566 Add [[gerrit:1295918]] to Monday, June 01 UTC afternoon backport window 2421775 wikitext text/x-wiki gleowiv54lgj2928wz3l7zjumw14dvq 2421895 2421775 2026-06-01T16:30:16Z ScheduleDeploymentBot 37566 Add [[gerrit:1295962]] to Monday, June 01 UTC late backport window 2421895 wikitext text/x-wiki r9pfk2wksj023i1ry2qavb3u9lmpgh1 2421916 2421895 2026-06-01T16:57:45Z ScheduleDeploymentBot 37566 Add [[gerrit:1295968]] to Monday, June 01 UTC late backport window 2421916 wikitext text/x-wiki pqmkbv441a0zje61dmy352kxw3ahzur 2421953 2421916 2026-06-01T19:30:42Z ScheduleDeploymentBot 37566 Add [[gerrit:1296003]] to Monday, June 01 UTC late backport window 2421953 wikitext text/x-wiki dsahwqzi1ldkxzkhwftwvzl1qx82cnx 2421954 2421953 2026-06-01T19:31:25Z ABreault (WMF) 37749 2421954 wikitext text/x-wiki 7srcpa0ehnc9fc0pd6dcanmnt6ifxn7 2421975 2421954 2026-06-01T20:09:09Z ScheduleDeploymentBot 37566 Add [[gerrit:1296009]] to Monday, June 01 UTC late backport window 2421975 wikitext text/x-wiki eudfjab12xftg0fzv0spcu280n2nyup 2422131 2421975 2026-06-02T07:45:30Z ScheduleDeploymentBot 37566 Add [[gerrit:1296262]] to Tuesday, June 02 UTC morning backport window 2422131 wikitext text/x-wiki l1j3j6fosqmpmc770pvonwe2hwvjq79 2422132 2422131 2026-06-02T07:45:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1296262]] to Tuesday, June 02 UTC morning backport window 2422132 wikitext text/x-wiki abzk4lvjuue3uwu76efoekah0w3q6y4 2422133 2422132 2026-06-02T07:47:23Z DCausse (WMF) 5604 /* {{Deployment_day|date=2026-06-02}} */ 2422133 wikitext text/x-wiki styl0gafb19y51vhck5zhsl9i8lj9yw 2422200 2422133 2026-06-02T09:46:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1296516]] to Tuesday, June 02 UTC afternoon backport window 2422200 wikitext text/x-wiki b2mq0zsqkd0o5xlq4bv947r3sihswex 2422202 2422200 2026-06-02T09:46:24Z ScheduleDeploymentBot 37566 Add [[gerrit:1296517]] to Tuesday, June 02 UTC afternoon backport window 2422202 wikitext text/x-wiki gv33qv3yqrfs9sy2en2e4rr9wb0v9w9 2422206 2422202 2026-06-02T10:07:17Z MSzwarc-WMF 44780 /* {{Deployment_day|date=2026-06-02}} */ I'll deploy for Tran 2422206 wikitext text/x-wiki o1pv5nki4oehkcyts2iu66vj1ez30gg 498 6726 2421955 2420066 2026-06-01T19:36:08Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2421955 wikitext text/x-wiki k23grca5hv79qojz4hje4rc5bj7le4i 2421961 2421955 2026-06-01T19:48:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2421961 wikitext text/x-wiki 36vmi3cnis2k7ptragku94wj23g4qkj 2421963 2421961 2026-06-01T19:50:28Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2421963 wikitext text/x-wiki 9ryd2fyno4pxwj8yvn1vzx5y363m08i 2421967 2421963 2026-06-01T19:58:16Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2421967 wikitext text/x-wiki 9eud10qpxpbnodhx1i26xdkt9gz1cvd 2422030 2421967 2026-06-02T00:53:48Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2422030 wikitext text/x-wiki 7ouqh14nwcu1yfioi31dt1ds1bfeea7 2422032 2422030 2026-06-02T01:05:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2422032 wikitext text/x-wiki o3gafhma6dcb4mbyd2c96gwgwxp65jd 0 7919 2421675 2421674 2026-06-01T11:59:31Z Stashbot 7414 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance 2421675 wikitext text/x-wiki t3dxv2feqrx2zmhof6psd2s3a8zink9 2421676 2421675 2026-06-01T11:59:33Z Stashbot 7414 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance 2421676 wikitext text/x-wiki l8q9h1nao1ak168nx0wa0jzhqloc110 2421679 2421676 2026-06-01T12:04:38Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet 2421679 wikitext text/x-wiki 5fdk5i3unpta4pacj90aam0sr7lr088 2421680 2421679 2026-06-01T12:04:40Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2027.codfw.wmnet 2421680 wikitext text/x-wiki oovepysnp08b3wauvgosm6puvo54g2t 2421681 2421680 2026-06-01T12:04:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet 2421681 wikitext text/x-wiki h4i2vyurp6fn8sidg8ismph3dggg22e 2421682 2421681 2026-06-01T12:05:53Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet 2421682 wikitext text/x-wiki dzpeq56g6j6oztrqy1dt7lwpdpsy8wb 2421683 2421682 2026-06-01T12:07:55Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to drbd 2421683 wikitext text/x-wiki ms5n5sk3wp19nkanty7lyag1pbbno8e 2421684 2421683 2026-06-01T12:11:29Z Stashbot 7414 dpogorzelski@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad 2421684 wikitext text/x-wiki j5ppakitvf1u6tbqoinss8w9vkek8pc 2421685 2421684 2026-06-01T12:15:16Z Stashbot 7414 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in eqiad/ml-serve-eqiad: maintenance 2421685 wikitext text/x-wiki 1xxl5h82w8fgapvuodwtk7032iujtl3 2421686 2421685 2026-06-01T12:15:51Z Stashbot 7414 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in eqiad/ml-serve-eqiad: maintenance 2421686 wikitext text/x-wiki 7mdrt9cgqo9tr9mxjr1cz8196uy4rme 2421687 2421686 2026-06-01T12:17:55Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421687 wikitext text/x-wiki ram7nhm2sn7yqm1h3otlt65hsqkn40g 2421688 2421687 2026-06-01T12:20:12Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421688 wikitext text/x-wiki gy9h2ed0tas4alstks9bb31o2qq2zuy 2421690 2421688 2026-06-01T12:23:45Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to drbd 2421690 wikitext text/x-wiki 44pteiknyav5ut0v8zpykdjwb6l2hvb 2421691 2421690 2026-06-01T12:26:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet 2421691 wikitext text/x-wiki 2lmj037teiazwias4hp65w20qfm032d 2421692 2421691 2026-06-01T12:26:30Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet 2421692 wikitext text/x-wiki hn4kaprjy0ppuadz6265dtumnud0ufa 2421693 2421692 2026-06-01T12:26:57Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to plain 2421693 wikitext text/x-wiki 8538h3fk0i0jnej57gfevhdftxti529 2421694 2421693 2026-06-01T12:27:40Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to plain 2421694 wikitext text/x-wiki 76flrbv72h5b6iprp2uffjlobw24mgt 2421695 2421694 2026-06-01T12:27:52Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2421695 wikitext text/x-wiki szo5sgkec9pht5zpxwke7o0iz76tnmj 2421696 2421695 2026-06-01T12:28:37Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2421696 wikitext text/x-wiki 9uedzlbtr7lrx8qj9nt92ihzflae591 2421697 2421696 2026-06-01T12:28:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet 2421697 wikitext text/x-wiki mdotzaz87gxe7yg7ghusrazf184mky6 2421698 2421697 2026-06-01T12:29:21Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2421698 wikitext text/x-wiki aqbuec855jhhoqhcfi4sjd5uzul8s43 2421699 2421698 2026-06-01T12:35:08Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421699 wikitext text/x-wiki s003mhgcq6fas9bht2fck7f4fok1if8 2421700 2421699 2026-06-01T12:35:45Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421700 wikitext text/x-wiki squ4i2iwg2iyi1ql8kdxxyo3tkcy6eq 2421701 2421700 2026-06-01T12:39:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93435 and previous config saved to /var/cache/conftool/dbconfig/20260601-123926-fceratto.json 2421701 wikitext text/x-wiki d9d2ot52zjyh2fx67tswe4phi5jjmhn 2421702 2421701 2026-06-01T12:41:50Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421702 wikitext text/x-wiki hm4mu2hw5ml2zo9wqfyvu153zfo9kdv 2421703 2421702 2026-06-01T12:42:45Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421703 wikitext text/x-wiki il1vxd33wwog4gitnll9q72d8acehxb 2421704 2421703 2026-06-01T12:43:25Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421704 wikitext text/x-wiki 7z9r7ese8fkpebg8wjk1uun2z1wn2uf 2421705 2421704 2026-06-01T12:44:03Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421705 wikitext text/x-wiki ryry3rxha008b9kja2mw0xx5ywuzopx 2421708 2421705 2026-06-01T12:46:55Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421708 wikitext text/x-wiki dyn5uc1hpk0ieme9xcwvw14o893w8em 2421710 2421708 2026-06-01T12:47:32Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421710 wikitext text/x-wiki nmp5d73wg4d08on71av8t92rvwxz1ba 2421711 2421710 2026-06-01T12:48:40Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421711 wikitext text/x-wiki 6bjaytr1nhejzw655b7oizkli7nui9j 2421712 2421711 2026-06-01T12:49:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93436 and previous config saved to /var/cache/conftool/dbconfig/20260601-124934-fceratto.json 2421712 wikitext text/x-wiki tomtixzj1567sgt8uv1mgfqkkvu6lt2 2421713 2421712 2026-06-01T12:49:37Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421713 wikitext text/x-wiki mwxs8xwofubqmbwnmeadgns6ko2sn8a 2421714 2421713 2026-06-01T12:50:55Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421714 wikitext text/x-wiki t23fm8n37gh5k0ab75rm0n5qdmt3ki8 2421715 2421714 2026-06-01T12:52:25Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421715 wikitext text/x-wiki rga0h1mkrbi16g9ty5kgiqpx8o7p104 2421716 2421715 2026-06-01T12:55:27Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . 2421716 wikitext text/x-wiki 75lgituho6b5m86u1jaitdjm72z9wta 2421717 2421716 2026-06-01T12:55:31Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2421717 wikitext text/x-wiki 1o1t53dwn4vfwmf6phgm0uni23xsotk 2421718 2421717 2026-06-01T12:55:35Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2421718 wikitext text/x-wiki agv0awpxb5nodqms6i8orl69finrrpo 2421719 2421718 2026-06-01T12:55:38Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2421719 wikitext text/x-wiki omelirzh3o4yjri5w3zx0ydas22872e 2421720 2421719 2026-06-01T12:55:41Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2421720 wikitext text/x-wiki q2w4fofxxpv7j2r0v9zcj8g8aaz1v7f 2421721 2421720 2026-06-01T12:55:45Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2421721 wikitext text/x-wiki 2duc7qupj37lch1v3f5ru0ksrknsrny 2421722 2421721 2026-06-01T12:55:48Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2421722 wikitext text/x-wiki 4vre37c4u0j4t6g90qi0pmpibcjp19x 2421723 2421722 2026-06-01T12:55:53Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2421723 wikitext text/x-wiki nh1hvf021duurhl6ty1h1pyalur070w 2421724 2421723 2026-06-01T12:55:56Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . 2421724 wikitext text/x-wiki tj48xig400g0ghz7p6fqtuzc47bl6ag 2421725 2421724 2026-06-01T12:55:59Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2421725 wikitext text/x-wiki spxqgas43sw85ntify9qec7sa0wuwo0 2421726 2421725 2026-06-01T12:56:02Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . 2421726 wikitext text/x-wiki 50e4o1irf9ktyaxrjh6a9top3j8duct 2421727 2421726 2026-06-01T12:56:04Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . 2421727 wikitext text/x-wiki dhgv0pejthoabhar364n643x1ug2hd8 2421728 2421727 2026-06-01T12:56:08Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2421728 wikitext text/x-wiki 2oftlrcsxyan1ebo9ux0urbadwul68r 2421729 2421728 2026-06-01T12:56:11Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2421729 wikitext text/x-wiki mks1hfs8o6da3dwcrlg7j79yw9d1g5k 2421730 2421729 2026-06-01T12:56:14Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2421730 wikitext text/x-wiki 0qk9yyzsjbtvp5y9jadn24q2zs6iz3j 2421731 2421730 2026-06-01T12:56:17Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2421731 wikitext text/x-wiki l4cyde1c0jekhmswy2q6hzbohb8o4rz 2421732 2421731 2026-06-01T12:56:22Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2421732 wikitext text/x-wiki 3nj19rt4r1s5xlh23n1zm52zx0suju3 2421733 2421732 2026-06-01T12:56:23Z Stashbot 7414 dpogorzelski@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad 2421733 wikitext text/x-wiki cu3kzxxpbcyktbkciuviv1t8u966xnl 2421734 2421733 2026-06-01T12:59:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93437 and previous config saved to /var/cache/conftool/dbconfig/20260601-125941-fceratto.json 2421734 wikitext text/x-wiki 04dcpw4fmsh2cnqsykrv8uty7u04oln 2421735 2421734 2026-06-01T13:02:42Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295536|swwiki: Enable the Visual Editor on the project namespace (T427117)]] 2421735 wikitext text/x-wiki 9kqfijgnralgsxwr2piyq3m9zc2c6gi 2421736 2421735 2026-06-01T13:03:16Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421736 wikitext text/x-wiki h0mg4ivqama3m8itd98waohopjclvf5 2421737 2421736 2026-06-01T13:04:12Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421737 wikitext text/x-wiki tgndpov5xtzucsnvmibjyjnsy3q30qb 2421738 2421737 2026-06-01T13:04:22Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2421738 wikitext text/x-wiki 1z4jcqr9axr1vl5zd7xxwxgzaitnq7r 2421739 2421738 2026-06-01T13:04:33Z Stashbot 7414 mszwarc@deploy1003: codenamenoreste, mszwarc: Backport for [[gerrit:1295536|swwiki: Enable the Visual Editor on the project namespace (T427117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421739 wikitext text/x-wiki 7i940ret5zyyn9h5othepd5792qe921 2421740 2421739 2026-06-01T13:05:25Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2421740 wikitext text/x-wiki 10nvcnmtptuai72a8szbjt3umo60hcp 2421741 2421740 2026-06-01T13:06:42Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . 2421741 wikitext text/x-wiki ier3rpru10q8wn75sqfup92dodv1kpl 2421742 2421741 2026-06-01T13:07:07Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2421742 wikitext text/x-wiki e879v783ritkmwvob4wuyfllop8f4so 2421743 2421742 2026-06-01T13:08:34Z Stashbot 7414 mszwarc@deploy1003: codenamenoreste, mszwarc: Continuing with deployment 2421743 wikitext text/x-wiki 84xlm09ju9e02j5zmqpp271c6d2x5bn 2421744 2421743 2026-06-01T13:09:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93438 and previous config saved to /var/cache/conftool/dbconfig/20260601-130949-fceratto.json 2421744 wikitext text/x-wiki t609ge68l22az1v80gg8gizeu7hcpz6 2421745 2421744 2026-06-01T13:12:49Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295536|swwiki: Enable the Visual Editor on the project namespace (T427117)]] (duration: 10m 06s) 2421745 wikitext text/x-wiki 08bdvofh6v9dt98l9sarjgrv7rn2487 2421746 2421745 2026-06-01T13:14:14Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295875|Add SetGlobalPreference maintenance script (T427476)]] 2421746 wikitext text/x-wiki 5bmt33gg1ngc3xhxmm3b17cr8lpt8jv 2421747 2421746 2026-06-01T13:14:17Z Stashbot 7414 atsukoito: sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' 2421747 wikitext text/x-wiki iw2ou94dafrrdesztzrqgqdbcu2xnm6 2421748 2421747 2026-06-01T13:15:58Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1295875|Add SetGlobalPreference maintenance script (T427476)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421748 wikitext text/x-wiki 6jtvhcihxe1ut7ltogdbg6f9v9tkdwo 2421749 2421748 2026-06-01T13:16:26Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2421749 wikitext text/x-wiki 4svdc7lfnzmqt3we5t9yq9i2w9g9qjg 2421750 2421749 2026-06-01T13:18:25Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test1001.eqiad.wmnet 2421750 wikitext text/x-wiki 7a86xknz4gh4kpei31qd6qhjubhffgc 2421751 2421750 2026-06-01T13:18:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie 2421751 wikitext text/x-wiki k60x6m4gxibrzp5ixyr3ylhk9na364f 2421752 2421751 2026-06-01T13:19:08Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test2001.codfw.wmnet 2421752 wikitext text/x-wiki m5m1g1wp07g55a9ra7exetv4x1aou3x 2421753 2421752 2026-06-01T13:20:36Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295875|Add SetGlobalPreference maintenance script (T427476)]] (duration: 06m 22s) 2421753 wikitext text/x-wiki 8upb6zijxi4m0o3ojue6l845tp8fl82 2421755 2421753 2026-06-01T13:20:55Z Stashbot 7414 Msz2001: UTC afternoon backpot+config window done 2421755 wikitext text/x-wiki eu5fvcefmjucohwjwd8s4mg9yhiwgx9 2421756 2421755 2026-06-01T13:20:58Z Stashbot 7414 atsukoito: restarted pybal.service on lvs1020 2421756 wikitext text/x-wiki ovi0nuo11nlo62yhhd1ijpcio6e457j 2421757 2421756 2026-06-01T13:21:59Z Stashbot 7414 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance 2421757 wikitext text/x-wiki 1tc3l1gs68ltev8fgsap7hugmprkpgc 2421758 2421757 2026-06-01T13:22:22Z Stashbot 7414 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance 2421758 wikitext text/x-wiki sw3tefp151o3g1nmhlw7lvk7tl327sn 2421759 2421758 2026-06-01T13:22:26Z Stashbot 7414 atsukoito: restarted pybal.service on lvs1019 2421759 wikitext text/x-wiki 1w70lbgfhcy4gcz1pnydxmwys80u2rw 2421760 2421759 2026-06-01T13:24:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet 2421760 wikitext text/x-wiki 8ykwqoe1ewwsi3qq5wcifexidvk4nf3 2421761 2421760 2026-06-01T13:24:52Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test2001.codfw.wmnet 2421761 wikitext text/x-wiki 75bmwki8oqwtcrrhzbzd9ost3c2qqgd 2421763 2421761 2026-06-01T13:31:17Z Stashbot 7414 atsukoito: restarted pybal.service on lvs2014 2421763 wikitext text/x-wiki j7hpo0l5z91pvrwfjm6p178moe3r72f 2421766 2421763 2026-06-01T13:31:47Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage 2421766 wikitext text/x-wiki hk4fqv6pzef5mcwh2tnrwc1qxnku79i 2421769 2421766 2026-06-01T13:35:42Z Stashbot 7414 atsukoito: restarted pybal.service on lvs2013 2421769 wikitext text/x-wiki 29llcf63eonrhtsvr8axaxx99a378s0 2421770 2421769 2026-06-01T13:37:49Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage 2421770 wikitext text/x-wiki pmpmp27wuony7vh1krlnz4f93bb24p9 2421771 2421770 2026-06-01T13:39:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance 2421771 wikitext text/x-wiki kunp203qu9nalr1o4bam1j3299ee3e1 2421772 2421771 2026-06-01T13:39:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93439 and previous config saved to /var/cache/conftool/dbconfig/20260601-133947-fceratto.json 2421772 wikitext text/x-wiki 0812f603wzwlsgvvff4ygozqx04qthm 2421777 2421772 2026-06-01T13:52:33Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie 2421777 wikitext text/x-wiki dih63csnm5ar0qylvnjk4hl5epcu2r2 2421778 2421777 2026-06-01T13:52:50Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421778 wikitext text/x-wiki p2hbgmj54n1j5h74g5fdkma6k029fw2 2421779 2421778 2026-06-01T13:58:55Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295918|Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] 2421779 wikitext text/x-wiki ee1snzmzly5s9uflipupprwhlq25i8z 2421780 2421779 2026-06-01T14:01:57Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:sessionstore 2421780 wikitext text/x-wiki hzs7xdjkehzam1pz3rsqfnurexfna90 2421781 2421780 2026-06-01T14:02:08Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421781 wikitext text/x-wiki 41d511sk8eanim2r5gl68sq6ns2odqa 2421782 2421781 2026-06-01T14:03:41Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Backport for [[gerrit:1295918|Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421782 wikitext text/x-wiki ibpu0sdihobpt0uz968c4dvlv18u1hv 2421783 2421782 2026-06-01T14:05:48Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Continuing with deployment 2421783 wikitext text/x-wiki 1zc2ebwp99e2tggc43jirvpab71nyut 2421784 2421783 2026-06-01T14:06:11Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421784 wikitext text/x-wiki e41sh7a6nou6sn8b9uun34jbmfl13z0 2421785 2421784 2026-06-01T14:10:01Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295918|Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] (duration: 11m 06s) 2421785 wikitext text/x-wiki 2lqva92lf7a8go8vwnuitrnv949j3z9 2421786 2421785 2026-06-01T14:11:56Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2421786 wikitext text/x-wiki 5b5sobqvmb5y423dsl2z5w2g8fes1cr 2421787 2421786 2026-06-01T14:12:47Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421787 wikitext text/x-wiki dukqcjpo8bwdpa0cevfwy0rzydjbgzs 2421788 2421787 2026-06-01T14:12:52Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421788 wikitext text/x-wiki fgm5jpufmnyrq1wxzhherx3388rkm01 2421789 2421788 2026-06-01T14:16:07Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421789 wikitext text/x-wiki 59vh9rhh3eypebg3iau2k7ou9a64k0e 2421790 2421789 2026-06-01T14:17:59Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421790 wikitext text/x-wiki 7q9gnvznxjv8rvc7zga2pzghadpa9uo 2421791 2421790 2026-06-01T14:21:53Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2421791 wikitext text/x-wiki md7ey5uxdngoda6huwo80za5v0t4j84 2421792 2421791 2026-06-01T14:22:05Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2421792 wikitext text/x-wiki 07pn60qqhnws5oqygvmhtm4xvda0sih 2421793 2421792 2026-06-01T14:23:35Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2421793 wikitext text/x-wiki d2x17wvp9n0vz6gp6zapq6b8rhog1h3 2421794 2421793 2026-06-01T14:25:16Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2421794 wikitext text/x-wiki 4d49a2n8gufk2m5y173ax7skhkfqp82 2421795 2421794 2026-06-01T14:27:18Z Stashbot 7414 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Deploy portals (T421797) (duration: 06m 10s) 2421795 wikitext text/x-wiki 6ugb5p12mdw2wz0kaitsiwf6q2nbutu 2421796 2421795 2026-06-01T14:28:02Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421796 wikitext text/x-wiki 1mh9q45eytb7sshsjjgxsvud8sv96r5 2421797 2421796 2026-06-01T14:30:02Z Stashbot 7414 ladsgroup@deploy1003: Synchronized portals: Deploy portals (T421797) (duration: 02m 43s) 2421797 wikitext text/x-wiki 5pwkhun91083okxmfqiebk1er909zka 2421798 2421797 2026-06-01T14:30:30Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421798 wikitext text/x-wiki hh2k2s2wjtk3okfr5ojv47o60k3clfd 2421799 2421798 2026-06-01T14:36:11Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421799 wikitext text/x-wiki o45x2d3s0b2kw6aqmle18ly6aeu33u4 2421800 2421799 2026-06-01T14:37:58Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421800 wikitext text/x-wiki cmq5y574jbb5ji4mly6hdb0lbeevs69 2421801 2421800 2026-06-01T14:40:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93440 and previous config saved to /var/cache/conftool/dbconfig/20260601-144002-fceratto.json 2421801 wikitext text/x-wiki exh7k7x2fdpwssro0sm7qmn26y7uzd4 2421802 2421801 2026-06-01T14:41:59Z Stashbot 7414 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2421802 wikitext text/x-wiki oguq25u6z764rn9psk0jsyexsn2nbpg 2421803 2421802 2026-06-01T14:42:19Z Stashbot 7414 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2421803 wikitext text/x-wiki srwn7n0ajazdddestby8hzf24z8f8ds 2421804 2421803 2026-06-01T14:48:02Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421804 wikitext text/x-wiki e4yc3k022vtf3wtsm2ogb9xwt44rr83 2421805 2421804 2026-06-01T14:49:22Z Stashbot 7414 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2421805 wikitext text/x-wiki 4zioange69m3obih5yo58omfmcsdvm8 2421806 2421805 2026-06-01T14:49:53Z Stashbot 7414 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2421806 wikitext text/x-wiki cgfi34iuk4w5iu3d4pg5d2we973jrra 2421807 2421806 2026-06-01T14:50:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93441 and previous config saved to /var/cache/conftool/dbconfig/20260601-145010-fceratto.json 2421807 wikitext text/x-wiki cvpsz295uf3pchuroq79pleja69a4e0 2421808 2421807 2026-06-01T14:50:39Z Stashbot 7414 atsuko@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2421808 wikitext text/x-wiki e4ahuee72x3wachtpzi3zgi0cib0v5v 2421809 2421808 2026-06-01T14:51:06Z Stashbot 7414 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2421809 wikitext text/x-wiki k8xjdauhygez8b907pvam2nav63y4sm 2421810 2421809 2026-06-01T14:52:00Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2421810 wikitext text/x-wiki 6g9ocsgkwo9t3fg4xu98zhumpyiiwwt 2421811 2421810 2026-06-01T14:52:06Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie 2421811 wikitext text/x-wiki gta5zkhdssl81p3q9fzxbpevdk5fjah 2421812 2421811 2026-06-01T14:52:21Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1209: Upgrading db1209.eqiad.wmnet 2421812 wikitext text/x-wiki 6lfhveszwawxmcp5hjl5fdfshhrnp7z 2421813 2421812 2026-06-01T14:52:36Z Stashbot 7414 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2421813 wikitext text/x-wiki jl77uao9yhb8zgpaj80ggxnx5ope89m 2421814 2421813 2026-06-01T14:52:51Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1209: Upgrading db1209.eqiad.wmnet 2421814 wikitext text/x-wiki f7kj9f7zc21ts2ilae8n8lpyouq8rct 2421815 2421814 2026-06-01T14:52:53Z Stashbot 7414 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2421815 wikitext text/x-wiki o8q1ntbdu62d9l2vqao9wpn03jh2sr8 2421818 2421815 2026-06-01T14:55:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1209.eqiad.wmnet with OS trixie 2421818 wikitext text/x-wiki 9trqg1l8m72fdrwbmyd20zxfntozi7c 2421820 2421818 2026-06-01T15:00:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93443 and previous config saved to /var/cache/conftool/dbconfig/20260601-150017-fceratto.json 2421820 wikitext text/x-wiki tryxkfy4j11wga1w2hl344mp44z0oto 2421824 2421820 2026-06-01T15:08:28Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:sessionstore 2421824 wikitext text/x-wiki dpd9i12ib5fmdxbiyr74mr3yabspmhr 2421825 2421824 2026-06-01T15:10:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93445 and previous config saved to /var/cache/conftool/dbconfig/20260601-151024-fceratto.json 2421825 wikitext text/x-wiki p8wzrt3bjh83m9n52vjjq0sbtuzvcdn 2421826 2421825 2026-06-01T15:10:47Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage 2421826 wikitext text/x-wiki qphi4b6uovuiai80on9cbgbdyt1mlyt 2421828 2421826 2026-06-01T15:12:26Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295946|hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] 2421828 wikitext text/x-wiki 7lecsnofarj4pxn26gh46dv898mqbo2 2421829 2421828 2026-06-01T15:12:53Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2421829 wikitext text/x-wiki 1pxt2bk2ch4zh0vqymmtabynsds0f9d 2421830 2421829 2026-06-01T15:13:09Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2421830 wikitext text/x-wiki fmued1mktdvhraok1rgneswaa6i3fs0 2421831 2421830 2026-06-01T15:14:11Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1295946|hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421831 wikitext text/x-wiki 46yggh7bh654rc16mnnbnhqb18ozu07 2421832 2421831 2026-06-01T15:14:31Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage 2421832 wikitext text/x-wiki a66njbmzk92e9u28gy40s75zeksk05s 2421833 2421832 2026-06-01T15:16:38Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2421833 wikitext text/x-wiki sojm4vzi7f3xb8f5lj9frwkykzq02ta 2421834 2421833 2026-06-01T15:19:29Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2421834 wikitext text/x-wiki ebo29dh5ujiygz2ww5244jirl9brk5d 2421835 2421834 2026-06-01T15:19:34Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2421835 wikitext text/x-wiki t4swibg2shh03u5i170hukxd9bll5ye 2421836 2421835 2026-06-01T15:20:50Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295946|hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] (duration: 08m 24s) 2421836 wikitext text/x-wiki lwdwy02mwl9ptstp0oere4to6poi1g0 2421837 2421836 2026-06-01T15:22:05Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2421837 wikitext text/x-wiki 9b6sq34hw8qt27z5ny6xeopchlnkit6 2421838 2421837 2026-06-01T15:22:10Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2421838 wikitext text/x-wiki co6cu8pdbcgvc6vlua8cezltno4ejxp 2421839 2421838 2026-06-01T15:22:19Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2421839 wikitext text/x-wiki rp7rrtoha6d2hdfip7vwnjj01axirog 2421840 2421839 2026-06-01T15:22:24Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2421840 wikitext text/x-wiki 4hn5c5qm2o9mctta7ngyzxlfssasfy0 2421841 2421840 2026-06-01T15:22:30Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295802|hCaptcha: Raise SiteVerify error threshold to 100]] 2421841 wikitext text/x-wiki j13pr2b1k3wpn0ma4ybi1lyjwnlyt4e 2421842 2421841 2026-06-01T15:22:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host testvm2005.codfw.wmnet with OS bullseye 2421842 wikitext text/x-wiki 28c8kcytpvfq59esobi525ue2efsb65 2421843 2421842 2026-06-01T15:24:16Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295802|hCaptcha: Raise SiteVerify error threshold to 100]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421843 wikitext text/x-wiki 2vsrle2yxa92wmefe4icepzeyh7msar 2421844 2421843 2026-06-01T15:24:36Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2421844 wikitext text/x-wiki r02ozokxfwicncn8o2j0uge2mafgk6d 2421845 2421844 2026-06-01T15:25:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling 2421845 wikitext text/x-wiki d7zotvb59u4q6wwvlhhrbi2znm92wjn 2421846 2421845 2026-06-01T15:25:44Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling 2421846 wikitext text/x-wiki iz6bhvkz2nzxq66adw8wtb1jxo57i1s 2421847 2421846 2026-06-01T15:25:59Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet 2421847 wikitext text/x-wiki k1vwuygx01cf3440z4hrnpa1fhn8qbt 2421848 2421847 2026-06-01T15:26:00Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet 2421848 wikitext text/x-wiki kkseuyk27dizdpxp6orayteillancei 2421849 2421848 2026-06-01T15:26:07Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling 2421849 wikitext text/x-wiki 6ljgjul4h9i7o9ltontxifvl2c7hpla 2421850 2421849 2026-06-01T15:26:32Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance 2421850 wikitext text/x-wiki r0dc65ptostn6vyb0icos5nz6lsok7e 2421851 2421850 2026-06-01T15:26:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93446 and previous config saved to /var/cache/conftool/dbconfig/20260601-152638-fceratto.json 2421851 wikitext text/x-wiki peki2rtcj3wnyvjlpb61hnym5l2ue25 2421852 2421851 2026-06-01T15:28:46Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295802|hCaptcha: Raise SiteVerify error threshold to 100]] (duration: 06m 15s) 2421852 wikitext text/x-wiki g1hzrbtw26w0dxyjnfvszbms7ed4x5f 2421853 2421852 2026-06-01T15:31:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1209.eqiad.wmnet with OS trixie 2421853 wikitext text/x-wiki 80fg06av8bmwa69an0ev0zr166owb8o 2421854 2421853 2026-06-01T15:37:47Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421854 wikitext text/x-wiki r3kypw66j0vl8e7f7zmonsuh2c1gix2 2421855 2421854 2026-06-01T15:38:00Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage 2421855 wikitext text/x-wiki bf4fv9wm5183sx8i8xw3zwc89mxz3ry 2421856 2421855 2026-06-01T15:38:48Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling 2421856 wikitext text/x-wiki 39j7fsfdfb77dk4bwg7vzx5xfy7kblw 2421857 2421856 2026-06-01T15:38:52Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling 2421857 wikitext text/x-wiki 79xd5yjrk9ftikdt0bnzf7pprx8bxay 2421858 2421857 2026-06-01T15:39:10Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421858 wikitext text/x-wiki 7fcmugtevorti6xzv2ohwjp41495a0l 2421859 2421858 2026-06-01T15:39:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Migration of db1209.eqiad.wmnet completed 2421859 wikitext text/x-wiki 327kom9vo7u6teli5dt4v12v2jwjeof 2421860 2421859 2026-06-01T15:39:22Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2421860 wikitext text/x-wiki rjtqqfbydn1kjzgf90s693xse4isde0 2421861 2421860 2026-06-01T15:39:41Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2421861 wikitext text/x-wiki cisukfzz1xaonnjzuw3xj2nxkactbwj 2421862 2421861 2026-06-01T15:40:01Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet 2421862 wikitext text/x-wiki 0cus2fz7usqkkmtqaklq9hfvqnbnqw0 2421863 2421862 2026-06-01T15:40:03Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet 2421863 wikitext text/x-wiki 9i7dnxancdz9b49zn7dsfr8igij8zpw 2421864 2421863 2026-06-01T15:40:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet 2421864 wikitext text/x-wiki 6l45vcjm4xuh5f44291d5mdnxkber9o 2421865 2421864 2026-06-01T15:40:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet 2421865 wikitext text/x-wiki 8xtvshprx3sz1e8fh5dunfx3zx26lun 2421866 2421865 2026-06-01T15:40:14Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling 2421866 wikitext text/x-wiki 6hwwcf7u0il9ws4uxrr41ic2mxmafr3 2421867 2421866 2026-06-01T15:40:17Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling 2421867 wikitext text/x-wiki 2g217i3nvin79wob0ghtemwja3f9y3w 2421869 2421867 2026-06-01T15:44:30Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage 2421869 wikitext text/x-wiki khqgte8wizuu3comkdycj84bwqbdt1p 2421870 2421869 2026-06-01T15:45:23Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling 2421870 wikitext text/x-wiki qo6xr0vaqu9pl3x5zh9ltwcebw3c4ew 2421871 2421870 2026-06-01T15:48:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet 2421871 wikitext text/x-wiki kjnqlg2r0borp8thdcjg9g29iin6oos 2421872 2421871 2026-06-01T15:49:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling 2421872 wikitext text/x-wiki ksupgu08bqd71yt6zfiqrn7jv49vlf4 2421873 2421872 2026-06-01T15:49:15Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling 2421873 wikitext text/x-wiki rr78bqm8wb9lf7cmwtbyixrxsut4jsu 2421874 2421873 2026-06-01T15:51:09Z Stashbot 7414 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie 2421874 wikitext text/x-wiki myoqkj3r2bqj7iyj6da7qz5ldhu07iq 2421875 2421874 2026-06-01T15:53:27Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Pooling 2421875 wikitext text/x-wiki dlfjy0mwg5447wywknifu2swgbrt0qp 2421876 2421875 2026-06-01T15:56:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2005.codfw.wmnet with OS bullseye 2421876 wikitext text/x-wiki 47bohh9uxlb1uxl1j7aazwhslh7ypwe 2421877 2421876 2026-06-01T15:56:48Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db1224: Pooling 2421877 wikitext text/x-wiki npm6jpiqmh2ueu8hrqit0by6gnofsji 2421878 2421877 2026-06-01T15:56:56Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling 2421878 wikitext text/x-wiki 9o5hyok25y0z3oqdwr5uktcmjcpczo8 2421879 2421878 2026-06-01T16:02:32Z Stashbot 7414 moritzm: temporarily remove ganeti2027 from the codfw cluster T427357 2421879 wikitext text/x-wiki 3fzo3mwz0jykl8g1si7gb5y7ltpyqyf 2421880 2421879 2026-06-01T16:03:10Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet 2421880 wikitext text/x-wiki pqlkfky96fshmug1nhp86vgyzqu1dhf 2421881 2421880 2026-06-01T16:04:01Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet 2421881 wikitext text/x-wiki jbmmw2nu8jpbgyc738hix43yrw5knc3 2421882 2421881 2026-06-01T16:05:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd 2421882 wikitext text/x-wiki fzpg7u0p31wpgvlanklq84nu0nzx3xf 2421883 2421882 2026-06-01T16:06:02Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2421883 wikitext text/x-wiki 8ffzoml39n70lhehg3hlzm7bmddkr79 2421884 2421883 2026-06-01T16:07:58Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421884 wikitext text/x-wiki qhe7ltwmpa7a1sb2bh7k5k9go2w9xca 2421885 2421884 2026-06-01T16:08:57Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421885 wikitext text/x-wiki rskgaz8ongax6xrwhoxs52u8i0bdiid 2421886 2421885 2026-06-01T16:09:19Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1236: Update 2421886 wikitext text/x-wiki n1j2x5a72ouhld68urp07wr0wuqknyp 2421887 2421886 2026-06-01T16:09:49Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1236: Update 2421887 wikitext text/x-wiki l8zu3k2mfxtyb9l16bl8hs4t8s3co0r 2421888 2421887 2026-06-01T16:10:28Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update T426633 2421888 wikitext text/x-wiki pptq8upe01a9qz3f6gzjn14k7vsqofo 2421889 2421888 2026-06-01T16:24:43Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Migration of db1209.eqiad.wmnet completed 2421889 wikitext text/x-wiki coufh4334jpgv82raoylzd0n8nrfvb0 2421890 2421889 2026-06-01T16:24:45Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2421890 wikitext text/x-wiki 2mnddydt303spkqxskh8njyrp3w95s4 2421891 2421890 2026-06-01T16:26:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93455 and previous config saved to /var/cache/conftool/dbconfig/20260601-162653-fceratto.json 2421891 wikitext text/x-wiki b3ibnw9oyq7ub8nkajk87qpk39uxz0h 2421892 2421891 2026-06-01T16:29:14Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd 2421892 wikitext text/x-wiki 3p0w3l3yoxpy1xazf7ygc56fx2vcvk6 2421893 2421892 2026-06-01T16:29:27Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update 2421893 wikitext text/x-wiki iwr1oh7bs189tbffzd8gyk406z9xa7b 2421894 2421893 2026-06-01T16:29:54Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update 2421894 wikitext text/x-wiki ofonigwuk5yfvzyzbws40a3m9h8rn83 2421896 2421894 2026-06-01T16:30:45Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update 2421896 wikitext text/x-wiki 8nmyc8aksqipcejprlkb5y7fikpmnwy 2421897 2421896 2026-06-01T16:30:55Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet 2421897 wikitext text/x-wiki tjmb8tetixv6o8l4kocv9z2zhbxr5wl 2421898 2421897 2026-06-01T16:30:57Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet 2421898 wikitext text/x-wiki ldovvmvbx1n4z620ddc9lvy6hau5u38 2421899 2421898 2026-06-01T16:31:14Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421899 wikitext text/x-wiki fvq0958fztil17i4mjtonr58x09xjt9 2421900 2421899 2026-06-01T16:31:36Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update T426633 2421900 wikitext text/x-wiki k5e6bn0grc9t6jir7cmj9r6zbb0ylgw 2421901 2421900 2026-06-01T16:34:20Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421901 wikitext text/x-wiki 2ju3gphzgak6z92anfxznfiu16mu23t 2421902 2421901 2026-06-01T16:34:22Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2421902 wikitext text/x-wiki bpqgtide1hl2fhsqy7us8iuqjxe7fe5 2421903 2421902 2026-06-01T16:34:28Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update 2421903 wikitext text/x-wiki ia23mif08d3jwe2bf9kt7gxykcjyxvg 2421904 2421903 2026-06-01T16:34:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update 2421904 wikitext text/x-wiki of50693myazee0f3lpu57adm8bc21e6 2421905 2421904 2026-06-01T16:35:08Z Stashbot 7414 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs,service=wdqs-main,name=wdqs1015.eqiad.wmnet 2421905 wikitext text/x-wiki mp5a4a3tierfkegoyhguglcocohgpdp 2421906 2421905 2026-06-01T16:35:32Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet 2421906 wikitext text/x-wiki r4co3qgdytr30gdivbcob9obd0f6yze 2421907 2421906 2026-06-01T16:35:42Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet 2421907 wikitext text/x-wiki aupzmxsajywjxx1kych9hx4q5hx63e3 2421908 2421907 2026-06-01T16:36:41Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2421908 wikitext text/x-wiki a6n01fnfd82oatq3wonpdviyykc6tbj 2421909 2421908 2026-06-01T16:37:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93458 and previous config saved to /var/cache/conftool/dbconfig/20260601-163701-fceratto.json 2421909 wikitext text/x-wiki tmpvoa8efbp9u2sjn492usgtkcu0q7k 2421910 2421909 2026-06-01T16:37:13Z Stashbot 7414 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs-main,service=wdqs-main,name=wdqs1015.eqiad.wmnet 2421910 wikitext text/x-wiki 94jaaeaw01lbwad34tkytooassru9nt 2421911 2421910 2026-06-01T16:42:25Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling 2421911 wikitext text/x-wiki 8myf9qbrlqbtx3klk0cs7sp93xc7nqe 2421912 2421911 2026-06-01T16:47:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93460 and previous config saved to /var/cache/conftool/dbconfig/20260601-164709-fceratto.json 2421912 wikitext text/x-wiki ahcw4ahycvfa2gzb8q5tluwokox22t0 2421914 2421912 2026-06-01T16:57:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93462 and previous config saved to /var/cache/conftool/dbconfig/20260601-165717-fceratto.json 2421914 wikitext text/x-wiki n8gwrneicxpmawxfam7n6qk86v4azm5 2421915 2421914 2026-06-01T16:57:31Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie 2421915 wikitext text/x-wiki q8nvuo1wi19xcp5hstuwxa3ck1lyogv 2421917 2421915 2026-06-01T16:58:15Z Stashbot 7414 Amir1: drop flaggedrevs tables on wikinews wikis (T423577) 2421917 wikitext text/x-wiki ta07pw1gdg9hkzemw7ionpe7178erg9 2421918 2421917 2026-06-01T16:59:01Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd 2421918 wikitext text/x-wiki ligptghzcwmxly1egemrla5k0dq67gw 2421919 2421918 2026-06-01T17:03:38Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling 2421919 wikitext text/x-wiki h101dnayfx1zb2snbr5rmswlm6tng0b 2421920 2421919 2026-06-01T17:03:42Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling 2421920 wikitext text/x-wiki hmffzczqwgusvjzvikf12p9c12do65q 2421921 2421920 2026-06-01T17:03:59Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling 2421921 wikitext text/x-wiki 351rx25abvuyg9ahq8vmlmpc95yt76v 2421922 2421921 2026-06-01T17:04:02Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1180: Pooling 2421922 wikitext text/x-wiki l78fbu29tvj8myn61qryf02d7kvoq4e 2421923 2421922 2026-06-01T17:04:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling 2421923 wikitext text/x-wiki fhedmnd6aktl4firchmjwltesk0j9oh 2421924 2421923 2026-06-01T17:04:11Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling 2421924 wikitext text/x-wiki doc4qxad792oljvqe7c4bx3m8mgmiyp 2421925 2421924 2026-06-01T17:10:12Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd 2421925 wikitext text/x-wiki 1pi0561wvqfk6adc71nongwvvso5wga 2421928 2421925 2026-06-01T17:20:22Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1236: Update 2421928 wikitext text/x-wiki 5bqa6jsxpa8jq2wnmnhnb27ekhskcor 2421929 2421928 2026-06-01T17:34:33Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1295976|nlwiki: change to Wikipedia 25 logo (T424519)]] 2421929 wikitext text/x-wiki 4suoaliiggxn0i2765a81kg06e0xgzb 2421930 2421929 2026-06-01T17:36:28Z Stashbot 7414 samtar@deploy1003: chlod, samtar: Backport for [[gerrit:1295976|nlwiki: change to Wikipedia 25 logo (T424519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421930 wikitext text/x-wiki 6tlf6h2brcqbb7cqd4x1r1mc4dth7gv 2421931 2421930 2026-06-01T17:37:46Z Stashbot 7414 samtar@deploy1003: chlod, samtar: Continuing with deployment 2421931 wikitext text/x-wiki lje3ncwsbapjfbsr6dibnw100xfiqei 2421932 2421931 2026-06-01T17:42:03Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295976|nlwiki: change to Wikipedia 25 logo (T424519)]] (duration: 07m 29s) 2421932 wikitext text/x-wiki 0q0f07xacvclpj7zc47kbtwu9m60zqh 2421933 2421932 2026-06-01T17:53:05Z Stashbot 7414 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie 2421933 wikitext text/x-wiki jmqg83opbornwuku28wi49g2kg8c72h 2421934 2421933 2026-06-01T17:58:59Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet 2421934 wikitext text/x-wiki 0zkx57rysvqk9d26zcjw3t3x4q7v8bo 2421935 2421934 2026-06-01T17:59:57Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet 2421935 wikitext text/x-wiki bw36jmbu3vf091v6wqjis4qzgrgphck 2421936 2421935 2026-06-01T18:01:05Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain 2421936 wikitext text/x-wiki 4j1idtp8me9fl6q6ph9tepra8wgca61 2421937 2421936 2026-06-01T18:01:33Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2421937 wikitext text/x-wiki 6ste68g5wi00w225unyegxal48ov6no 2421938 2421937 2026-06-01T18:01:48Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain 2421938 wikitext text/x-wiki 46fe81tfx2amagoz6frz7pbj9sln8y8 2421939 2421938 2026-06-01T18:02:11Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain 2421939 wikitext text/x-wiki 9kvcs9le38s4olrrem1zrs3q7n9hp30 2421940 2421939 2026-06-01T18:02:46Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2421940 wikitext text/x-wiki hs7iayb303rgcpuj3ofoitbb5m5xho2 2421941 2421940 2026-06-01T18:03:15Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain 2421941 wikitext text/x-wiki aip3pppmnd5uhq3oibi1ocsx0waugov 2421942 2421941 2026-06-01T18:05:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet 2421942 wikitext text/x-wiki fpvn6wqxazvi2f81zntqwo4471o4wgx 2421943 2421942 2026-06-01T18:06:10Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet 2421943 wikitext text/x-wiki cyuu5rdmagpkdi1fk47bpriov3g2k68 2421944 2421943 2026-06-01T18:17:40Z Stashbot 7414 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1295950|mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] 2421944 wikitext text/x-wiki elnzhube4giaocnbocdya1pkkv94vcs 2421945 2421944 2026-06-01T18:19:30Z Stashbot 7414 otto@deploy1003: otto: Backport for [[gerrit:1295950|mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421945 wikitext text/x-wiki h7ys0p2itfznob8k32xn48x8yzfn9e5 2421946 2421945 2026-06-01T18:20:06Z Stashbot 7414 otto@deploy1003: otto: Continuing with deployment 2421946 wikitext text/x-wiki 3fmi9lzrzgy2o44p09p6alh3zfqa16b 2421947 2421946 2026-06-01T18:24:22Z Stashbot 7414 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295950|mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] (duration: 06m 42s) 2421947 wikitext text/x-wiki o9kptq07xc9e2e26c3uxjev4aj17cdv 2421949 2421947 2026-06-01T19:00:37Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync 2421949 wikitext text/x-wiki fmg03nj1oozt2mbc7li1gpefptcnbja 2421950 2421949 2026-06-01T19:01:06Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync 2421950 wikitext text/x-wiki muun5rovzjobrhc6r1xukwnunp6j5nk 2421956 2421950 2026-06-01T19:46:00Z Stashbot 7414 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2421956 wikitext text/x-wiki dc2np6x0l7ogtkr0srbzdz7w9omtzok 2421957 2421956 2026-06-01T19:46:09Z Stashbot 7414 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2421957 wikitext text/x-wiki 4g31bj0j9wimwenn1fji0pgtnyjgw4t 2421958 2421957 2026-06-01T19:46:57Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2421958 wikitext text/x-wiki czt06s6ekl6gp0iea10owu46t5jv7pg 2421959 2421958 2026-06-01T19:47:10Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2421959 wikitext text/x-wiki 0c4frvr4pv4evwtueaf3qfmibr8z96q 2421960 2421959 2026-06-01T19:47:58Z Stashbot 7414 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2421960 wikitext text/x-wiki hvz48mggb1x4b76t58eb5bem3nuxc0t 2421962 2421960 2026-06-01T19:48:30Z Stashbot 7414 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2421962 wikitext text/x-wiki arcsliw8qsd1vik8udcnens0tgonfiu 2421971 2421962 2026-06-01T20:05:18Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1295504|passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] 2421971 wikitext text/x-wiki rvu2q4yr2p76uhnzt0v3ezvr5k3kj0i 2421972 2421971 2026-06-01T20:07:02Z Stashbot 7414 catrope@deploy1003: catrope: Backport for [[gerrit:1295504|passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421972 wikitext text/x-wiki 6iyfqbg0lpw1jdp7wgb46bpj5e8gbb9 2421974 2421972 2026-06-01T20:08:43Z Stashbot 7414 catrope@deploy1003: catrope: Continuing with deployment 2421974 wikitext text/x-wiki kgvkojvipfmjozrfhhrrmbo8dxvd8fg 2421976 2421974 2026-06-01T20:12:56Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295504|passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] (duration: 07m 37s) 2421976 wikitext text/x-wiki jlvt3jwtkn9q2vwtq1fjp2b6hh9x3tp 2421980 2421976 2026-06-01T20:18:33Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1285412|Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531|Enable AbuseFilter block action on nlwiki (T427384)]] 2421980 wikitext text/x-wiki aq3xxu45w4fv8ryy0xywliihphxgqix 2421982 2421980 2026-06-01T20:20:19Z Stashbot 7414 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Backport for [[gerrit:1285412|Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531|Enable AbuseFilter block action on nlwiki (T427384)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421982 wikitext text/x-wiki hx1bkyb3bsb9udxakhat3v12odnwclb 2421983 2421982 2026-06-01T20:22:10Z Stashbot 7414 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Continuing with deployment 2421983 wikitext text/x-wiki 5fdg71589whz217g9ud8t6yh4jxpjfo 2421987 2421983 2026-06-01T20:26:21Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1285412|Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531|Enable AbuseFilter block action on nlwiki (T427384)]] (duration: 07m 48s) 2421987 wikitext text/x-wiki 414qgrc95jz48q9f9gmxs78yvaviics 2421988 2421987 2026-06-01T20:37:43Z Stashbot 7414 ryankemper@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: T427852 hw failure 2421988 wikitext text/x-wiki bhv6lc96083b2oxmkcqm7thayu2c3dg 2421989 2421988 2026-06-01T20:53:41Z Stashbot 7414 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2421989 wikitext text/x-wiki gj5wpqua4mpnfa9hbsl39vl6lynp3v5 2421990 2421989 2026-06-01T21:04:33Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1296002|Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003|Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009|Redirect Special:AccountRecovery to the shared domain (T427692)]] 2421990 wikitext text/x-wiki elwrza9rn5qmk0yii77t410pmjzbfdb 2421991 2421990 2026-06-01T21:06:15Z Stashbot 7414 catrope@deploy1003: catrope, arlolra: Backport for [[gerrit:1296002|Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003|Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009|Redirect Special:AccountRecovery to the shared domain (T427692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2421991 wikitext text/x-wiki sez5v4tc3j8hjt1j5h2dv49lyl5wtrl 2421992 2421991 2026-06-01T21:07:47Z Stashbot 7414 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2421992 wikitext text/x-wiki h7rz9fxb7i7sky2e2guc6v9kcpbanlj 2421993 2421992 2026-06-01T21:07:59Z Stashbot 7414 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2421993 wikitext text/x-wiki 08q1cn9mlaq8n22t7gcqb9lmhca4ty4 2421994 2421993 2026-06-01T21:08:23Z Stashbot 7414 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2421994 wikitext text/x-wiki j8ubc15brsx6wqaclxki5vhr0lq9elf 2421995 2421994 2026-06-01T21:09:18Z Stashbot 7414 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2421995 wikitext text/x-wiki 2i3m5ur6n97gg8bq0j9zqb0rayiuh4r 2421996 2421995 2026-06-01T21:09:27Z Stashbot 7414 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2421996 wikitext text/x-wiki 5s1nejdagmzkkp5gud2a3dcy7z7g7pz 2421997 2421996 2026-06-01T21:09:33Z Stashbot 7414 catrope@deploy1003: catrope, arlolra: Continuing with deployment 2421997 wikitext text/x-wiki 1nrnsbluaj3f7k40anvtkj67pu0pz2h 2421998 2421997 2026-06-01T21:13:47Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296002|Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003|Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009|Redirect Special:AccountRecovery to the shared domain (T427692)]] (duration: 09m 20s) 2421998 wikitext text/x-wiki j5r0e3rkyj5pyhee0wfoiunuzljseru 2421999 2421998 2026-06-01T21:27:33Z Stashbot 7414 maryum: Deployed security fix for T427235 2421999 wikitext text/x-wiki 0bjpq9alc7pravbzcae8lyrina0rkdf 2422000 2421999 2026-06-01T21:33:00Z Stashbot 7414 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2422000 wikitext text/x-wiki lpfqy6ujfzhdhj3v528zpx7w8n8g916 2422001 2422000 2026-06-01T21:33:09Z Stashbot 7414 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2422001 wikitext text/x-wiki tpg0suye8le9uzpa9df5nqlt3b8chao 2422003 2422001 2026-06-01T21:35:17Z Stashbot 7414 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2422003 wikitext text/x-wiki fklnzk3hst86m7i5h35136z8akul206 2422004 2422003 2026-06-01T21:35:37Z Stashbot 7414 maryum: Deployed security fix for T427611 2422004 wikitext text/x-wiki dogeznmbyeze3hqenb2tdoznjv6zuz9 2422005 2422004 2026-06-01T21:35:55Z Stashbot 7414 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2422005 wikitext text/x-wiki 59kztzn0i5r8m4t1trf9icdc4e6m8b7 2422006 2422005 2026-06-01T21:50:44Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422006 wikitext text/x-wiki o5q2b1mklswsdzzhhuj3sn75kaqjkim 2422007 2422006 2026-06-01T21:51:47Z Stashbot 7414 sbassett: Deployed updated mitigation for T326691 2422007 wikitext text/x-wiki eykj6qhpukay41hct62ws5mabl7rhph 2422008 2422007 2026-06-01T21:56:40Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422008 wikitext text/x-wiki cll6qm853pgqga3lmbaqn9g7c73t6ve 2422009 2422008 2026-06-01T21:58:24Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422009 wikitext text/x-wiki qp3slxn517x0dllbga4p27r9cnma4pm 2422010 2422009 2026-06-01T22:00:57Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422010 wikitext text/x-wiki f3gv0iany0v61w0e0ot2lm8n9lssrgz 2422011 2422010 2026-06-01T22:06:04Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422011 wikitext text/x-wiki 6gyeixi4vv83td976x4pok4ws6ygxxt 2422012 2422011 2026-06-01T22:07:47Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422012 wikitext text/x-wiki 5lo4n766zq4ifg12heyyzdbf3f0uyqt 2422013 2422012 2026-06-01T22:30:09Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1296024|Add maintenance script to scrape SVG render files]] 2422013 wikitext text/x-wiki 7ylp7zbbcnn8wrkm1i1f7yh5syf02in 2422014 2422013 2026-06-01T22:31:54Z Stashbot 7414 reedy@deploy1003: reedy: Backport for [[gerrit:1296024|Add maintenance script to scrape SVG render files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422014 wikitext text/x-wiki i49z3ci3veo2s8r6476q3qesrz6mi9f 2422015 2422014 2026-06-01T22:32:16Z Stashbot 7414 reedy@deploy1003: reedy: Continuing with deployment 2422015 wikitext text/x-wiki jsg03zk2vu1m76jj7zq57rhx3w7yr66 2422016 2422015 2026-06-01T22:36:32Z Stashbot 7414 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296024|Add maintenance script to scrape SVG render files]] (duration: 06m 22s) 2422016 wikitext text/x-wiki 2f181n3s9jfwl22b0e5ae6irsltserg 2422017 2422016 2026-06-01T23:04:39Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6015.* 2422017 wikitext text/x-wiki jjdvteai07rrekbeui6fewnvq0smcsr 2422018 2422017 2026-06-01T23:06:03Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1296022|Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028|styles: Limit selector to badge client pref (T427407)]] 2422018 wikitext text/x-wiki jwq62svhy96h13qvu2gm3pqv6drcgc4 2422019 2422018 2026-06-01T23:07:46Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1296022|Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028|styles: Limit selector to badge client pref (T427407)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422019 wikitext text/x-wiki 1lzqno2j6tn9o2193kigs5626hdgyxr 2422020 2422019 2026-06-01T23:11:23Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2422020 wikitext text/x-wiki 0ovnqcmz6u4ehtdq7bdqqpsf07cng36 2422021 2422020 2026-06-01T23:15:36Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296022|Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028|styles: Limit selector to badge client pref (T427407)]] (duration: 09m 33s) 2422021 wikitext text/x-wiki m5wo4on4ime3gcghg1tkjfs48pcwse4 2422022 2422021 2026-06-01T23:20:23Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295963|Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962|Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] 2422022 wikitext text/x-wiki az3u1fl8mf39i5v5tfbr4qmo8q9779l 2422023 2422022 2026-06-01T23:22:06Z Stashbot 7414 jdlrobson@deploy1003: mfossati, jdlrobson: Backport for [[gerrit:1295963|Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962|Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422023 wikitext text/x-wiki kynt1ufgo3o9nudrrrtswq5g7bh7rdc 2422024 2422023 2026-06-01T23:23:30Z Stashbot 7414 jdlrobson@deploy1003: mfossati, jdlrobson: Continuing with deployment 2422024 wikitext text/x-wiki qx6c6oayd9esb0pnff9yvep4u3bm4gx 2422025 2422024 2026-06-01T23:27:40Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295963|Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962|Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] (duration: 07m 17s) 2422025 wikitext text/x-wiki oyvgbfetkubq7d3vn2te62eeuep28c6 2422036 2422025 2026-06-02T04:05:41Z Stashbot 7414 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.2 (duration: 05m 33s) 2422036 wikitext text/x-wiki rbtkhvcfi33p685awf85kc424ng30ki 2422037 2422036 2026-06-02T04:36:27Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2422037 wikitext text/x-wiki ortref8td1fxtzmy6xbuuyv48o1r52h 2422038 2422037 2026-06-02T04:40:44Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2422038 wikitext text/x-wiki dfll1hk9bnfndfoextytwqfvdgbewba 2422039 2422038 2026-06-02T04:46:39Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422039 wikitext text/x-wiki hawqxmpl9mr1brac809wbq23x9sin9m 2422040 2422039 2026-06-02T04:49:53Z Stashbot 7414 ryankemper: T425007 (k8s) created 4 wdqs namespaces on `dse-k8s-codfw`'s `admin_ng` ns: `wdqs-[internal,external]` & `wdqs-[internal,external]-next`; certs issued 2422040 wikitext text/x-wiki 2rx17rqniugh081464gthfa9txbgz1b 2422041 2422040 2026-06-02T04:56:14Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422041 wikitext text/x-wiki ewmm6g5x90cie97emn7uvxd2m3wixz3 2422042 2422041 2026-06-02T05:00:33Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422042 wikitext text/x-wiki 6dcb5m3pr5onejs81p60zlg2ehqwofc 2422043 2422042 2026-06-02T05:02:09Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422043 wikitext text/x-wiki 4x90x9uf79exv52krcozjlyxp2av427 2422044 2422043 2026-06-02T05:05:52Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422044 wikitext text/x-wiki 6pr70x6g4t3ichfkmo8fanj1hr4driw 2422045 2422044 2026-06-02T05:06:14Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1052: Upgrading es1052.eqiad.wmnet 2422045 wikitext text/x-wiki t1mw5n1k9cykifyemardqa07gg6i232 2422046 2422045 2026-06-02T05:06:44Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1052: Upgrading es1052.eqiad.wmnet 2422046 wikitext text/x-wiki 2ml19p8zigjw2lsy3p13txcu3llwgeq 2422047 2422046 2026-06-02T05:07:27Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1052.eqiad.wmnet with OS trixie 2422047 wikitext text/x-wiki ity2u33rrinqty0mg11pe39ovlnryp7 2422048 2422047 2026-06-02T05:21:02Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422048 wikitext text/x-wiki kuu4l0d3jjhul830v9u6pt64d5uwtl8 2422049 2422048 2026-06-02T05:22:44Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage 2422049 wikitext text/x-wiki qxsehqte1e7o4ue8cozraj4346tzm6r 2422050 2422049 2026-06-02T05:25:13Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422050 wikitext text/x-wiki nqgdbpjs664lnn3o8jf8o3a6em4rbij 2422051 2422050 2026-06-02T05:26:14Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422051 wikitext text/x-wiki 9ui3n9tvfatg3utt8e0mnnswlllp4uh 2422052 2422051 2026-06-02T05:28:49Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422052 wikitext text/x-wiki tdgi52etoa1w1a961r3dcp8lt35k06p 2422053 2422052 2026-06-02T05:29:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage 2422053 wikitext text/x-wiki bk9hif2552zroa6c5infto0qwmwia5q 2422054 2422053 2026-06-02T05:29:45Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422054 wikitext text/x-wiki 6t5cphvfc9j24ttk8o186crrpxzprxg 2422055 2422054 2026-06-02T05:30:34Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422055 wikitext text/x-wiki 1hmvuodnzn7ivre1jaasyaeicla0006 2422056 2422055 2026-06-02T05:33:21Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422056 wikitext text/x-wiki tbtatb9atbe8ng14q7a7ompmf3itdat 2422057 2422056 2026-06-02T05:36:41Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422057 wikitext text/x-wiki 9kp99ybs04wamf92gbgm3gqhkg7eqp8 2422058 2422057 2026-06-02T05:45:22Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1052.eqiad.wmnet with OS trixie 2422058 wikitext text/x-wiki 5126iau9zbem0k102llwiiskpy7piwu 2422059 2422058 2026-06-02T05:46:23Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422059 wikitext text/x-wiki tvdkbfrp5a9dtfizr9wie0k4o6x13o2 2422060 2422059 2026-06-02T05:47:23Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422060 wikitext text/x-wiki s6ppdqfm4aaggemje8992dwy3sputf3 2422062 2422060 2026-06-02T05:50:48Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422062 wikitext text/x-wiki tai7rejwpe1m3c6vyq8nfmli68kpufw 2422063 2422062 2026-06-02T05:50:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1052: repool after upgrade 2422063 wikitext text/x-wiki muojl9n9l3p0zgm127npa20nw99b06x 2422064 2422063 2026-06-02T05:51:41Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T426088 2422064 wikitext text/x-wiki k0ey8a0tsacms5i0cqj87ugtt6jqmfm 2422065 2422064 2026-06-02T05:51:56Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1236 with weight 0 T426088', diff saved to https://phabricator.wikimedia.org/P93470 and previous config saved to /var/cache/conftool/dbconfig/20260602-055153-marostegui.json 2422065 wikitext text/x-wiki qnaebfyu3qain05f808cpevfzsp9pui 2422067 2422065 2026-06-02T06:00:04Z Stashbot 7414 marostegui: Starting s7 eqiad failover from db1181 to db1236 - T426088 2422067 wikitext text/x-wiki 5sc9o34lxcgx8d59x5tdd4purliamza 2422068 2422067 2026-06-02T06:00:21Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - T426088', diff saved to https://phabricator.wikimedia.org/P93471 and previous config saved to /var/cache/conftool/dbconfig/20260602-060018-marostegui.json 2422068 wikitext text/x-wiki b1kulop60a07o02yuc29315crkmizs8 2422069 2422068 2026-06-02T06:00:44Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write T426088', diff saved to https://phabricator.wikimedia.org/P93472 and previous config saved to /var/cache/conftool/dbconfig/20260602-060041-marostegui.json 2422069 wikitext text/x-wiki 7wubhvb2g07tz5ub0kfmzeubd0cgmwx 2422070 2422069 2026-06-02T06:01:24Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2422070 wikitext text/x-wiki as0epyqj2hh5ebrkgomgs9w6sitne53 2422071 2422070 2026-06-02T06:02:00Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1181 T426088', diff saved to https://phabricator.wikimedia.org/P93473 and previous config saved to /var/cache/conftool/dbconfig/20260602-060157-marostegui.json 2422071 wikitext text/x-wiki 9pf8kelg8phhi714aw3zbt2sjiqgu0r 2422072 2422071 2026-06-02T06:02:50Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2422072 wikitext text/x-wiki 5b4zlypc9401lpa6bjygqm6x4sb7hat 2422073 2422072 2026-06-02T06:05:00Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422073 wikitext text/x-wiki djc03whb4sdt4ioz0cg1vzocuajbmkd 2422074 2422073 2026-06-02T06:05:10Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Upgrading db1181.eqiad.wmnet 2422074 wikitext text/x-wiki 9qw208jmp9gdhsca9cueflb6wdwv66c 2422075 2422074 2026-06-02T06:05:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Upgrading db1181.eqiad.wmnet 2422075 wikitext text/x-wiki bp3vymm308r56zgwl1b9hw2rxbszlyc 2422076 2422075 2026-06-02T06:08:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1181.eqiad.wmnet with OS trixie 2422076 wikitext text/x-wiki dgho72ip91dql7kga1rlysnxcwajztz 2422077 2422076 2026-06-02T06:15:46Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2422077 wikitext text/x-wiki 31cxykj3u21viv9gdkb62a5o7tgceoi 2422078 2422077 2026-06-02T06:16:22Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2422078 wikitext text/x-wiki se4zp98gp3p1pztksrjxwzhh230znb2 2422079 2422078 2026-06-02T06:21:21Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2422079 wikitext text/x-wiki thzbpnsxlnny4xivc3l54y9tace9sp3 2422080 2422079 2026-06-02T06:22:12Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2422080 wikitext text/x-wiki inf5durn4y5zypozl8x18risylx6veg 2422081 2422080 2026-06-02T06:24:32Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage 2422081 wikitext text/x-wiki 48fibrfpg03niz1rithcnpwmy3elfob 2422082 2422081 2026-06-02T06:29:51Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage 2422082 wikitext text/x-wiki rofdb30ibpg173jrnfa742dm3lnqy7g 2422083 2422082 2026-06-02T06:36:20Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1052: repool after upgrade 2422083 wikitext text/x-wiki fa13inxxea0ecgx872zawyrsun0vzaq 2422084 2422083 2026-06-02T06:36:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet 2422084 wikitext text/x-wiki isk8qecjahm4ocgde5dq6ooldkw5qaa 2422085 2422084 2026-06-02T06:37:23Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet 2422085 wikitext text/x-wiki kuzidjuran2tsj2vxhg03g6ceychgqm 2422086 2422085 2026-06-02T06:37:34Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet 2422086 wikitext text/x-wiki b2w1zaeot2hrohqcs6u44rlqj94vlyw 2422087 2422086 2026-06-02T06:41:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422087 wikitext text/x-wiki ch4i69ohnxtky0g443jwb5pmvs13puk 2422088 2422087 2026-06-02T06:42:00Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2053: Upgrading es2053.codfw.wmnet 2422088 wikitext text/x-wiki 0o80tutgwsieft65sqne590z2cilu05 2422089 2422088 2026-06-02T06:42:21Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2053: Upgrading es2053.codfw.wmnet 2422089 wikitext text/x-wiki q6inwi9qmn3uda6dllavyke1swzwuyn 2422090 2422089 2026-06-02T06:43:22Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2053.codfw.wmnet with OS trixie 2422090 wikitext text/x-wiki t8d7z17ii437isdbh40s8gg93k3878m 2422091 2422090 2026-06-02T06:46:45Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1181.eqiad.wmnet with OS trixie 2422091 wikitext text/x-wiki 97pue0qlea7mbllcgj6sjq77q5ymaw1 2422092 2422091 2026-06-02T06:55:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance 2422092 wikitext text/x-wiki 3g56u5sq5lvsx7rrn65ivpzqgb8wulw 2422093 2422092 2026-06-02T06:55:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Migration of db1181.eqiad.wmnet completed 2422093 wikitext text/x-wiki m35s0vqj4zbkfpa9168oo218stdl5bv 2422094 2422093 2026-06-02T06:55:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93478 and previous config saved to /var/cache/conftool/dbconfig/20260602-065533-fceratto.json 2422094 wikitext text/x-wiki 32i8rthfv90bzskurj6a56g48q7ejn0 2422096 2422094 2026-06-02T06:59:27Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2053.codfw.wmnet with reason: host reimage 2422096 wikitext text/x-wiki qyszymjbi4lszrzs46zh45gwg3y7080 2422097 2422096 2026-06-02T07:04:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2053.codfw.wmnet with reason: host reimage 2422097 wikitext text/x-wiki 2dq8kkevwsrhy96fz7dc56w0bvxovjb 2422098 2422097 2026-06-02T07:12:21Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Depool for rack maintenance 2422098 wikitext text/x-wiki 50p0swk6woq2sgplwavzqqs13kjyfm4 2422099 2422098 2026-06-02T07:12:23Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2186.codfw.wmnet with reason: upgrade 2422099 wikitext text/x-wiki fotrb7pgzvklrdzbdpw4swqibx5xc0c 2422100 2422099 2026-06-02T07:12:53Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Depool for rack maintenance 2422100 wikitext text/x-wiki odnl13rjd0pe5ggq6mcnx8xotyh0w98 2422101 2422100 2026-06-02T07:14:49Z Stashbot 7414 marostegui: Install mariadb 10.11.17 on db2186 T427345 2422101 wikitext text/x-wiki 8qn2kqw9v97sxbizozgza71jo5icfoe 2422102 2422101 2026-06-02T07:15:03Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2241.codfw.wmnet with reason: Depool for rack maintenance 2422102 wikitext text/x-wiki ha7uuyfe5lbczgbem4npj4gfa0q2bdn 2422103 2422102 2026-06-02T07:19:23Z Stashbot 7414 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1294949|translate: adding separate read/write endpoints (T425377)]] 2422103 wikitext text/x-wiki 86zopt8yuzogb8btcb4c3zxan19ipnw 2422104 2422103 2026-06-02T07:20:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2053.codfw.wmnet with OS trixie 2422104 wikitext text/x-wiki r2vqw0x1ihhcm9l2qghzqy8ddbmy704 2422105 2422104 2026-06-02T07:21:19Z Stashbot 7414 atsuko@deploy1003: atsuko: Backport for [[gerrit:1294949|translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422105 wikitext text/x-wiki a05g40pgld02rwx7yjbcxytqw9soaxq 2422106 2422105 2026-06-02T07:23:07Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet 2422106 wikitext text/x-wiki fvqpozflllkzdj0bn07picwuzjd7274 2422107 2422106 2026-06-02T07:23:09Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet 2422107 wikitext text/x-wiki 4rroe8c9qqb8lwhprtbdjb214qvz1ni 2422108 2422107 2026-06-02T07:23:39Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Depool for rack maintenance 2422108 wikitext text/x-wiki 6rc9m2n42nmpc4zbgpjx2foxai0pvf7 2422109 2422108 2026-06-02T07:25:50Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc2021: rack A3 maintenance 2422109 wikitext text/x-wiki n3j0tum4n4wqlf2yp6xhdqblg84gprb 2422110 2422109 2026-06-02T07:25:52Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2422110 wikitext text/x-wiki 4qyl15qcxcc6f211fvptq52nr67kr8v 2422111 2422110 2026-06-02T07:26:06Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2422111 wikitext text/x-wiki 06u4rcmtrmwlkq4qozd1d8iqy94u8kp 2422112 2422111 2026-06-02T07:26:07Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2021: rack A3 maintenance 2422112 wikitext text/x-wiki j1lyrfszi1xregamarmnmuu4az0gk92 2422113 2422112 2026-06-02T07:26:49Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on pc2021.codfw.wmnet with reason: rack A3 maintenance 2422113 wikitext text/x-wiki 4a4r2gpn55gnswyozuqsajy2j1dpodp 2422114 2422113 2026-06-02T07:27:40Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422114 wikitext text/x-wiki 188i24vso7l9iiv77d6wcde6z95tpc9 2422115 2422114 2026-06-02T07:28:34Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2158: rack A3 maintenance 2422115 wikitext text/x-wiki ht6zepegimq3iju0d821dsysdugubm5 2422116 2422115 2026-06-02T07:28:55Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: rack A3 maintenance 2422116 wikitext text/x-wiki nq7kpw315y5af7d9hpojcd38wdekt1n 2422117 2422116 2026-06-02T07:28:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T426633)', diff saved to https://phabricator.wikimedia.org/P93487 and previous config saved to /var/cache/conftool/dbconfig/20260602-072856-fceratto.json 2422117 wikitext text/x-wiki 71bylcr37jrhaovhjhzlbspluxdiwh5 2422118 2422117 2026-06-02T07:29:36Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2158.codfw.wmnet with reason: rack A3 maintenance 2422118 wikitext text/x-wiki 9marqz52fo75cjqfxrldja2gbd8dh0n 2422119 2422118 2026-06-02T07:30:40Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2053: repool after upgrade 2422119 wikitext text/x-wiki 99ggicoxqjk2lzum4jii9sh7gaq5e9o 2422120 2422119 2026-06-02T07:32:39Z Stashbot 7414 XioNoX: pfw1-eqiad# delete protocols bgp group Production family inet6 - T423384 2422120 wikitext text/x-wiki 1s5qdettmjl5rqe2m9m2z7zwts8r6k4 2422121 2422120 2026-06-02T07:39:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93490 and previous config saved to /var/cache/conftool/dbconfig/20260602-073904-fceratto.json 2422121 wikitext text/x-wiki f5fx4p99z6v49lrgggp24z17u8gpjdz 2422122 2422121 2026-06-02T07:39:35Z Stashbot 7414 atsuko@deploy1003: atsuko: Rolling back deployment 2422122 wikitext text/x-wiki 3b3npf5cem0i5xo28ypzlw77fiu3eiz 2422123 2422122 2026-06-02T07:40:25Z Stashbot 7414 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294949|translate: adding separate read/write endpoints (T425377)]] (duration: 21m 01s) 2422123 wikitext text/x-wiki py1xtmjbn671jgi8gk5je6xdt4puh4g 2422124 2422123 2026-06-02T07:41:02Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Migration of db1181.eqiad.wmnet completed 2422124 wikitext text/x-wiki 64txt93llai2os8el1xb7a7incspn6v 2422125 2422124 2026-06-02T07:41:04Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2422125 wikitext text/x-wiki t3rpnbi9ys4mwv2h5vwtfyso2jqn9vf 2422126 2422125 2026-06-02T07:41:55Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling 2422126 wikitext text/x-wiki ndribdwxnaoalm6gs34my4lvieaqjbw 2422127 2422126 2026-06-02T07:42:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1181.eqiad.wmnet with reason: Reboot 2422127 wikitext text/x-wiki e7pczcop7lyfvssfwp9mf3edij15gja 2422128 2422127 2026-06-02T07:43:10Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Reboot 2422128 wikitext text/x-wiki k6pp23nacwj53ljaz3h5ghqmygbdomq 2422129 2422128 2026-06-02T07:44:02Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Reboot 2422129 wikitext text/x-wiki as3p2n3g8aoyjt62oufavzizqapbcsg 2422134 2422129 2026-06-02T07:47:31Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Pooling 2422134 wikitext text/x-wiki 4fidwhq7jqirm06qggkcm478hk18sj6 2422135 2422134 2026-06-02T07:48:28Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1181: Pooling 2422135 wikitext text/x-wiki 2xwz519w6u358igod2pwbf59jwmwvnd 2422136 2422135 2026-06-02T07:49:04Z Stashbot 7414 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296262|translate: fixing missed variable in credentials formatting closure (T425377)]] 2422136 wikitext text/x-wiki luf06d8czd6s49ot6edixo19hn76wub 2422137 2422136 2026-06-02T07:50:48Z Stashbot 7414 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296262|translate: fixing missed variable in credentials formatting closure (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422137 wikitext text/x-wiki ttjnc1lpudoqw26bl53ihaq70dc5fim 2422138 2422137 2026-06-02T07:57:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling 2422138 wikitext text/x-wiki giratwjxn1fd5jl9w9s45hhr1uyhx2k 2422139 2422138 2026-06-02T07:57:53Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance 2422139 wikitext text/x-wiki 1flsqwkx1ke005xub97nlh1prb30ha6 2422140 2422139 2026-06-02T07:58:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1181 (T419635)', diff saved to https://phabricator.wikimedia.org/P93498 and previous config saved to /var/cache/conftool/dbconfig/20260602-075759-fceratto.json 2422140 wikitext text/x-wiki rn9xar8oa371hl2837e9ka7qstzw277 2422141 2422140 2026-06-02T07:58:50Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2422141 wikitext text/x-wiki 0sx3y4951tv66e6uszvmevhw00i3yn8 2422143 2422141 2026-06-02T07:59:39Z Stashbot 7414 atsuko@deploy1003: atsuko: Rolling back deployment 2422143 wikitext text/x-wiki 5pgh1uawc16085asvxj74xf86fn72sp 2422144 2422143 2026-06-02T07:59:48Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2422144 wikitext text/x-wiki 5hs4erwo6tr5f4lnuw4dd10atm1qf7x 2422145 2422144 2026-06-02T08:00:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T419635)', diff saved to https://phabricator.wikimedia.org/P93499 and previous config saved to /var/cache/conftool/dbconfig/20260602-080011-fceratto.json 2422145 wikitext text/x-wiki bbyf7j5q2zp6cuvgkcmryikma5lm9q8 2422146 2422145 2026-06-02T08:03:51Z Stashbot 7414 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296262|translate: fixing missed variable in credentials formatting closure (T425377)]] (duration: 14m 47s) 2422146 wikitext text/x-wiki ol73926gof5igy0hp6usy2kvfigf8c0 2422147 2422146 2026-06-02T08:09:09Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Depool for rack maintenance 2422147 wikitext text/x-wiki b09cr5nh044y50ws41xwailni0j7pat 2422148 2422147 2026-06-02T08:09:35Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2422148 wikitext text/x-wiki gqt7h3hn6ztsi2tesbnqknno3ux5ffn 2422149 2422148 2026-06-02T08:10:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93502 and previous config saved to /var/cache/conftool/dbconfig/20260602-081018-fceratto.json 2422149 wikitext text/x-wiki cfldqtk9x5xs1wvlep1m6quswe72cyl 2422150 2422149 2026-06-02T08:10:31Z Stashbot 7414 marostegui: Install mariadb 10.11.17 on es2053 T427345 2422150 wikitext text/x-wiki l1yeo5yzh613jhhc9g3bko86nwdxvob 2422151 2422150 2026-06-02T08:11:18Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2422151 wikitext text/x-wiki gxbappid29fl8f7dqlgxybydhc63x9d 2422152 2422151 2026-06-02T08:13:32Z Stashbot 7414 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296488|Revert "translate: adding separate read/write endpoints" (T425377)]] 2422152 wikitext text/x-wiki n7ryafw2v1n9u4j1ukufqy6r3q7x1hz 2422153 2422152 2026-06-02T08:15:17Z Stashbot 7414 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296488|Revert "translate: adding separate read/write endpoints" (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422153 wikitext text/x-wiki 61bothldlr9lrctc2t7wnpdw1r7gqdt 2422154 2422153 2026-06-02T08:16:04Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2053: repool after upgrade 2422154 wikitext text/x-wiki s5yzvupzi79ylfp8wfucy5w5c0ohg1m 2422155 2422154 2026-06-02T08:16:34Z Stashbot 7414 atsuko@deploy1003: atsuko: Rolling back deployment 2422155 wikitext text/x-wiki ico66u40uanp7veiukqie6x5e734ncr 2422156 2422155 2026-06-02T08:17:06Z Stashbot 7414 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296488|Revert "translate: adding separate read/write endpoints" (T425377)]] (duration: 03m 33s) 2422156 wikitext text/x-wiki koncrukeaxgfjqbx5azjng4tlf278kb 2422157 2422156 2026-06-02T08:18:27Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2422157 wikitext text/x-wiki 6fm9rdlo3sz2u7c5qdpdvy0su6rs212 2422158 2422157 2026-06-02T08:18:44Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2422158 wikitext text/x-wiki j92t0l6an5jhhnpz2axh8xjw3xtbggu 2422159 2422158 2026-06-02T08:19:41Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2422159 wikitext text/x-wiki e8dh3jkst3wcodiwwtwp9g2wysoxg7h 2422160 2422159 2026-06-02T08:20:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93504 and previous config saved to /var/cache/conftool/dbconfig/20260602-082026-fceratto.json 2422160 wikitext text/x-wiki l1j0cpi4e5e9awn1xigzv31z97qtc60 2422161 2422160 2026-06-02T08:20:44Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2422161 wikitext text/x-wiki iftkdprwlw55mbdqegsmi2hcoiq2tym 2422162 2422161 2026-06-02T08:29:36Z Stashbot 7414 slyngs: IDP, new configuration in preparation for webauthn 2422162 wikitext text/x-wiki kwy8u6pt4x75b8jit0i9vonqcjoq8rp 2422163 2422162 2026-06-02T08:30:08Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422163 wikitext text/x-wiki 2nydmiq6fqxl1vo8xb7vhpzp3oky1n9 2422164 2422163 2026-06-02T08:30:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T419635)', diff saved to https://phabricator.wikimedia.org/P93505 and previous config saved to /var/cache/conftool/dbconfig/20260602-083033-fceratto.json 2422164 wikitext text/x-wiki 3axsbc6q8kj72gjmnxi6qyxnoezof5q 2422165 2422164 2026-06-02T08:30:50Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance 2422165 wikitext text/x-wiki dkgh81jmg75hf2snyoqyubh9wuvb8n8 2422166 2422165 2026-06-02T08:37:16Z Stashbot 7414 urbanecm: Reset user email of Barras@votewiki to the one of Barras@SUL 2422166 wikitext text/x-wiki qwto173o8s2ntsbvd9pfge02r1j3yk1 2422167 2422166 2026-06-02T08:39:24Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422167 wikitext text/x-wiki 22q0izs2ivizxl0h1mjotl1fqnjhgt0 2422168 2422167 2026-06-02T08:41:24Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422168 wikitext text/x-wiki dynei7cy8oj1eg12ws1usvye42sg3yl 2422169 2422168 2026-06-02T08:46:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet 2422169 wikitext text/x-wiki 0quzeo7v5xpxpv2fkuo9lgji9iot2kg 2422170 2422169 2026-06-02T08:47:47Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422170 wikitext text/x-wiki nqymp5d9oo2l4j36j554woekaqnrrl6 2422171 2422170 2026-06-02T08:50:47Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422171 wikitext text/x-wiki ksnhr15d6dgzfzta6aag7g8aakyxxpl 2422172 2422171 2026-06-02T08:50:58Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2422172 wikitext text/x-wiki 8zxl5p8f4u6jh0ii19uugvjh5weo8wb 2422173 2422172 2026-06-02T08:51:53Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2422173 wikitext text/x-wiki dzwa6h4x7qrl9ydt5nqen4uc7guxmdp 2422174 2422173 2026-06-02T08:52:20Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2422174 wikitext text/x-wiki spt8cvmzddol3tw1nn7yhhmoeqtj2bv 2422175 2422174 2026-06-02T08:53:17Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2422175 wikitext text/x-wiki 7dc2sb4fcmjprj4n55slz0dmdnywj2y 2422176 2422175 2026-06-02T08:54:25Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422176 wikitext text/x-wiki 45x38rjz7d6pw63wg8z5qqg6mks3b8x 2422177 2422176 2026-06-02T08:54:36Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422177 wikitext text/x-wiki fsh87v6txri3tj0r6an6cxji5qbwfq5 2422178 2422177 2026-06-02T08:55:08Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2422178 wikitext text/x-wiki 946ipujm6dw8anf1mi3ya0cfvr41a93 2422179 2422178 2026-06-02T08:56:32Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1055.eqiad.wmnet with OS trixie 2422179 wikitext text/x-wiki 5smj3ww25o3q07lub7fbz77hsl6emid 2422180 2422179 2026-06-02T08:56:49Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422180 wikitext text/x-wiki 5wgifswzgmnf3qcf73dp287m0vkqbqk 2422181 2422180 2026-06-02T08:59:16Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2250.codfw.wmnet with reason: rack A3 maintenance 2422181 wikitext text/x-wiki 4l65acpz85eo3fw945be1kfph29wedk 2422182 2422181 2026-06-02T09:04:26Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance 2422182 wikitext text/x-wiki gfbhco21xje4chvd5qia6zgm8gvnmzg 2422183 2422182 2026-06-02T09:04:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1187 (T426633)', diff saved to https://phabricator.wikimedia.org/P93506 and previous config saved to /var/cache/conftool/dbconfig/20260602-090432-fceratto.json 2422183 wikitext text/x-wiki tsletx7dei7u4nxitwj9e9u5dtm844q 2422184 2422183 2026-06-02T09:09:29Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage 2422184 wikitext text/x-wiki ohvhf8wj13yj8hxbwc4u8p4o04qs8vr 2422185 2422184 2026-06-02T09:11:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T426633)', diff saved to https://phabricator.wikimedia.org/P93508 and previous config saved to /var/cache/conftool/dbconfig/20260602-091126-fceratto.json 2422185 wikitext text/x-wiki nfqdi9rw9udhyrf0en5nt5b6shu9e0c 2422186 2422185 2026-06-02T09:14:08Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage 2422186 wikitext text/x-wiki 03augr9qdujfhrn74muko5eb67u0imc 2422187 2422186 2026-06-02T09:15:23Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1187: Pooling 2422187 wikitext text/x-wiki qjho0wd8cwt2vebkh39uy26w5vyz8ws 2422188 2422187 2026-06-02T09:30:22Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1055.eqiad.wmnet with OS trixie 2422188 wikitext text/x-wiki 4js4rvev9is4xr16256k5bq35qfr16l 2422189 2422188 2026-06-02T09:32:58Z Stashbot 7414 moritzm: temporarily remove ganeti2045 from the codfw cluster T427357 2422189 wikitext text/x-wiki e9mrxwpmi6tn7bvph2fhpa524f6x7um 2422190 2422189 2026-06-02T09:34:21Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to plain 2422190 wikitext text/x-wiki 3rzvm411x21sjv0480l4bnca77sar7b 2422191 2422190 2026-06-02T09:34:25Z Stashbot 7414 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - T422937 2422191 wikitext text/x-wiki 4suycxeugvz98t4hre45sb7gnv00kde 2422192 2422191 2026-06-02T09:34:59Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to plain 2422192 wikitext text/x-wiki d7owkjtit1g9xkhacdmkwwrj47cl1vy 2422193 2422192 2026-06-02T09:35:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to plain 2422193 wikitext text/x-wiki iui7x9tbfdcdd2t5g5afulmi11nga76 2422194 2422193 2026-06-02T09:37:10Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2422194 wikitext text/x-wiki lurxxxugja5bimlvff3nf947tso1jbu 2422195 2422194 2026-06-02T09:37:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T426633)', diff saved to https://phabricator.wikimedia.org/P93511 and previous config saved to /var/cache/conftool/dbconfig/20260602-093716-fceratto.json 2422195 wikitext text/x-wiki rw2c0is8tkkh9p8lqzirtbotf6txykd 2422196 2422195 2026-06-02T09:37:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to plain 2422196 wikitext text/x-wiki 67oh39jioufifhxd9ju1qqzztleslia 2422197 2422196 2026-06-02T09:37:42Z Stashbot 7414 claime: Running puppet on cp6010 and cp6011 - T422937 2422197 wikitext text/x-wiki 7h8tv1o08z3q7hq5qpcs5yanwkpjx7e 2422199 2422197 2026-06-02T09:45:43Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Pooling 2422199 wikitext text/x-wiki du3ps3q8jamu1x1lvtepg5bpysxm3bd 2422201 2422199 2026-06-02T09:46:14Z Stashbot 7414 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cumin2003.codfw.wmnet with reason: in setup 2422201 wikitext text/x-wiki f6mlq1dctvdtouhlek5kovhepmkkh76 2422203 2422201 2026-06-02T09:56:14Z Stashbot 7414 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - T422937 2422203 wikitext text/x-wiki m9ie4jj6zobyxeef0255sfvoxfikq7k 2422204 2422203 2026-06-02T10:06:27Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2422204 wikitext text/x-wiki 0ei4ukeho9tks6c0kkmnzmlf14bjhtv 2422205 2422204 2026-06-02T10:06:53Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2422205 wikitext text/x-wiki k9belkild2t6ldguj7o15349s7dzlsa 2422207 2422205 2026-06-02T10:08:04Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422207 wikitext text/x-wiki 0ae8r624ksyb96rbgtob9no1c1b07af 2422208 2422207 2026-06-02T10:08:25Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2056: Upgrading es2056.codfw.wmnet 2422208 wikitext text/x-wiki l3nu621vj0jkgcme3bb35nrgvpg60tn 2422209 2422208 2026-06-02T10:08:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2056: Upgrading es2056.codfw.wmnet 2422209 wikitext text/x-wiki 98c0n7qger9j6v7e0zg6xso7niobqqx 2422210 2422209 2026-06-02T10:09:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2056.codfw.wmnet with OS trixie 2422210 wikitext text/x-wiki ps34kko36wjxw45xl2529iqal86dd69 2422214 2422210 2026-06-02T10:21:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T426633)', diff saved to https://phabricator.wikimedia.org/P93515 and previous config saved to /var/cache/conftool/dbconfig/20260602-102139-fceratto.json 2422214 wikitext text/x-wiki 4kaxdn10o53wef22bh7e04ytwla35gt 2422215 2422214 2026-06-02T10:25:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2056.codfw.wmnet with reason: host reimage 2422215 wikitext text/x-wiki izq1me83tuvsu4zpch4kwif4ypy1i54 2422216 2422215 2026-06-02T10:27:16Z Stashbot 7414 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - T422937 2422216 wikitext text/x-wiki 28hsb3k2ic9ux6762jza4969epo6nmw 2422217 2422216 2026-06-02T10:28:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2056.codfw.wmnet with reason: host reimage 2422217 wikitext text/x-wiki 9gp8lju50e4lsdq8w4519ljpah8qby8 2422218 2422217 2026-06-02T10:31:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93516 and previous config saved to /var/cache/conftool/dbconfig/20260602-103146-fceratto.json 2422218 wikitext text/x-wiki 3xoryvf0y5o506wdei8yw7idnceh76h 2422220 2422218 2026-06-02T10:41:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93517 and previous config saved to /var/cache/conftool/dbconfig/20260602-104154-fceratto.json 2422220 wikitext text/x-wiki 8jq509thme3hciuw3bwtgn4nq94m9pq 2422221 2422220 2026-06-02T10:42:14Z Stashbot 7414 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - T422937 2422221 wikitext text/x-wiki qaqqqztsw9gbaudp7n7mgu487rg4429 2422222 2422221 2026-06-02T10:42:29Z Stashbot 7414 moritzm: installing busybox security updates 2422222 wikitext text/x-wiki komsiwp1g5k21iskloh486x9fchk8hj 2422223 2422222 2026-06-02T10:45:10Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2056.codfw.wmnet with OS trixie 2422223 wikitext text/x-wiki tg1sk52dadte2x74xdi10ps2f64mdmf 2422224 2422223 2026-06-02T10:52:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T426633)', diff saved to https://phabricator.wikimedia.org/P93518 and previous config saved to /var/cache/conftool/dbconfig/20260602-105202-fceratto.json 2422224 wikitext text/x-wiki q0g4e1nl3zzxw96czb1u0frbig1lvfj 2422225 2422224 2026-06-02T10:52:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance 2422225 wikitext text/x-wiki ajpl1nnh5d9aqd2gopvgjquk41ma6h0 2422226 2422225 2026-06-02T10:52:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2422226 wikitext text/x-wiki 6husfy17ed095b3yzgrsgfylrx89hrt 2422227 2422226 2026-06-02T10:52:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 (T426633)', diff saved to https://phabricator.wikimedia.org/P93519 and previous config saved to /var/cache/conftool/dbconfig/20260602-105239-fceratto.json 2422227 wikitext text/x-wiki nvff78il24k49fxllngun1009bqvttu 2422228 2422227 2026-06-02T10:59:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T426633)', diff saved to https://phabricator.wikimedia.org/P93520 and previous config saved to /var/cache/conftool/dbconfig/20260602-105939-fceratto.json 2422228 wikitext text/x-wiki mjcwbp3n9sfq3onvwek6h9vm8p5y7rj 2422229 2422228 2026-06-02T11:01:46Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422229 wikitext text/x-wiki bap04hn5oh738aggl41eyo01vxo9uau 2422230 2422229 2026-06-02T11:02:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2056: repool after upgrade 2422230 wikitext text/x-wiki 2ibvqhluwcs2kjbz4a3k147uk0jy36n 2422232 2422230 2026-06-02T11:03:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 T427892 2422232 wikitext text/x-wiki 6zmcy48fft6m893j1y2c4507jasudr7 2422234 2422232 2026-06-02T11:04:25Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2165 with weight 0 T427892', diff saved to https://phabricator.wikimedia.org/P93522 and previous config saved to /var/cache/conftool/dbconfig/20260602-110420-cwilliams.json 2422234 wikitext text/x-wiki bi181jykib5jopexzerfbpxdwfris3m 2422236 2422234 2026-06-02T11:09:06Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS trixie 2422236 wikitext text/x-wiki qczuz7w8jz1hu67lo4ifweur7i67l59 2422237 2422236 2026-06-02T11:09:24Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS trixie 2422237 wikitext text/x-wiki rxt4z7mbd57763ttuej61fnhle1um03 2422238 2422237 2026-06-02T11:09:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93523 and previous config saved to /var/cache/conftool/dbconfig/20260602-110947-fceratto.json 2422238 wikitext text/x-wiki s97l1pi76s63kp7uldlns86wddgdzmo 2422239 2422238 2026-06-02T11:10:51Z Stashbot 7414 cezmunsta: Starting s8 codfw failover from db2161 to db2165 - T427892 2422239 wikitext text/x-wiki o0c3mladz0yj94sorex85j93qdfwogm 2422240 2422239 2026-06-02T11:12:03Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2165 to s8 primary T427892', diff saved to https://phabricator.wikimedia.org/P93524 and previous config saved to /var/cache/conftool/dbconfig/20260602-111200-cwilliams.json 2422240 wikitext text/x-wiki tc9gy3t7kke08s0iiinkw9i0au9uoxu 2422241 2422240 2026-06-02T11:15:15Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2161 T427892', diff saved to https://phabricator.wikimedia.org/P93525 and previous config saved to /var/cache/conftool/dbconfig/20260602-111511-cwilliams.json 2422241 wikitext text/x-wiki k1zs8n3m81g88skxayo5h5nnsqjvu0h 2422242 2422241 2026-06-02T11:19:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93527 and previous config saved to /var/cache/conftool/dbconfig/20260602-111954-fceratto.json 2422242 wikitext text/x-wiki oikv8sga30i4dbq15pwaodhuzejib5n 2422244 2422242 2026-06-02T11:21:24Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422244 wikitext text/x-wiki gtgyll16nd3w01rs4771rn4b689kikp 2422245 2422244 2026-06-02T11:21:27Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage 2422245 wikitext text/x-wiki 75o6xcgl4s71lbxsmhwl5dv9i3uba5t 2422246 2422245 2026-06-02T11:21:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2161: Upgrading db2161.codfw.wmnet 2422246 wikitext text/x-wiki jgj32gsoab851w3gmliwym76tlwskrv 2422247 2422246 2026-06-02T11:21:44Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: Upgrading db2161.codfw.wmnet 2422247 wikitext text/x-wiki gfu6fsnwo4an3gome1mtd5wy2hpa38i 2422248 2422247 2026-06-02T11:22:13Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage 2422248 wikitext text/x-wiki r3g2ke3smg4mxaenvae63j0yw7uoqdi 2422249 2422248 2026-06-02T11:23:21Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS trixie 2422249 wikitext text/x-wiki dtrm8vjyuzsosxlru9hvdv5bo9lz47k 2422250 2422249 2026-06-02T11:26:06Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Repooling 2422250 wikitext text/x-wiki 3ci88mhyjcwzt5s2k6rpncokg2mgjit 2422251 2422250 2026-06-02T11:26:16Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Repooling 2422251 wikitext text/x-wiki dc1wiwws1dht1gj08qluyc5re9iclkv 2422252 2422251 2026-06-02T11:29:18Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage 2422252 wikitext text/x-wiki pwdtxks1dzxmn65cz2ij8aktp24xo9u 2422253 2422252 2026-06-02T11:30:13Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance 2422253 wikitext text/x-wiki 8drtu2dbrpt8h527442bra10xkpshh6 2422254 2422253 2026-06-02T11:30:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 (T426633)', diff saved to https://phabricator.wikimedia.org/P93529 and previous config saved to /var/cache/conftool/dbconfig/20260602-113019-fceratto.json 2422254 wikitext text/x-wiki sasm4d2pff3beaetn7w35ofrawu9n8i 2422255 2422254 2026-06-02T11:33:07Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage 2422255 wikitext text/x-wiki gq0zvaqiuct3j586hj8ozx23vnm1efx 2422256 2422255 2026-06-02T11:37:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T426633)', diff saved to https://phabricator.wikimedia.org/P93531 and previous config saved to /var/cache/conftool/dbconfig/20260602-113705-fceratto.json 2422256 wikitext text/x-wiki 6wb3acpwznk5v9wnnvwoa8i6g2ii137 2422257 2422256 2026-06-02T11:40:39Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage 2422257 wikitext text/x-wiki 4j26b4i6w7719d10kgy183tx0ckl2r6 2422258 2422257 2026-06-02T11:44:48Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage 2422258 wikitext text/x-wiki knoew37mrcjkkbqq9if0u2rsypt1087 2422259 2422258 2026-06-02T11:45:53Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS trixie 2422259 wikitext text/x-wiki ogcztb47a9qnzhg9lf64p12xx20ugpo 2422260 2422259 2026-06-02T11:47:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93532 and previous config saved to /var/cache/conftool/dbconfig/20260602-114713-fceratto.json 2422260 wikitext text/x-wiki hehf2o2el3rukoser2ql3kx6y743ln0 2422261 2422260 2026-06-02T11:47:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2056: repool after upgrade 2422261 wikitext text/x-wiki nluld4cn90jlz2ycg23poctwoce1lbl 2422262 2422261 2026-06-02T11:47:39Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS trixie 2422262 wikitext text/x-wiki 9gzyypaxjty1xm06a2t5flthrk00yas 2422263 2422262 2026-06-02T11:47:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422263 wikitext text/x-wiki j9mllt7qzl9e9h4l0v4x5jnelgzl0a0 2422264 2422263 2026-06-02T11:48:06Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2049: Upgrading es2049.codfw.wmnet 2422264 wikitext text/x-wiki s8rrxyk4p2rr223tz3gu1qvuwqnofsy 2422265 2422264 2026-06-02T11:48:39Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2049: Upgrading es2049.codfw.wmnet 2422265 wikitext text/x-wiki iokq5xptxln1ahbb7e7xggdjx7hnvou 2422266 2422265 2026-06-02T11:49:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2049.codfw.wmnet with OS trixie 2422266 wikitext text/x-wiki jchcf9f0srl7zpf800l3enyydhzbj6w 2422267 2422266 2026-06-02T11:49:32Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS trixie 2422267 wikitext text/x-wiki eyua8xpoq6l087eg6jl8l4f5oj8cymk 2422268 2422267 2026-06-02T11:50:12Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS trixie 2422268 wikitext text/x-wiki 4x4xez4z221f6pwurz8ms13a3hxwp5v 2422269 2422268 2026-06-02T11:53:06Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422269 wikitext text/x-wiki nclbpcvym012s8mqiq4e1uv1timoyye 2422270 2422269 2026-06-02T11:53:30Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2422270 wikitext text/x-wiki 2yj0kul7imzdo5z8rfs6exle43jxyr1 2422271 2422270 2026-06-02T11:53:49Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2422271 wikitext text/x-wiki 9jyq4norasgg9cuihpg36dgk95i1zqc 2422272 2422271 2026-06-02T11:55:01Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2422272 wikitext text/x-wiki mhlst9f291ik6nvpg8ns1z3jfavus2z 2422273 2422272 2026-06-02T11:55:30Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422273 wikitext text/x-wiki 82he3wl3u379zw1kf6q7akw9ttjvdez 2422274 2422273 2026-06-02T11:55:35Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422274 wikitext text/x-wiki tc0a4x24k05cth1vkhi4f8guod2mdmc 2422275 2422274 2026-06-02T11:55:42Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2422275 wikitext text/x-wiki sqxsci31qbhxsd5aruttgtucel4bina 2422276 2422275 2026-06-02T11:57:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93535 and previous config saved to /var/cache/conftool/dbconfig/20260602-115721-fceratto.json 2422276 wikitext text/x-wiki lreqj2vhbi8ae8rnl2sdqf8z3wl4tah 2422277 2422276 2026-06-02T11:58:25Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422277 wikitext text/x-wiki 8a5ke5bio80yosywd8zzdrm4fc00olv 0 17290 2422002 2421641 2026-06-01T21:34:58Z Stashbot 7414 jeena: Updating development images on contint primary for T424663 2422002 wikitext text/x-wiki s93xwlkwzsrn6kl6qdeketefltv0ol3 0 22503 2422034 2418601 2026-06-02T01:43:27Z Krinkle 44 /* See also */ 2422034 wikitext text/x-wiki 7se92hriui601rmn57uxr2vvqgoyz1w 0 327375 2421767 2387809 2026-06-01T13:34:45Z Ottomata 88 Added link to stream-internal 2421767 wikitext text/x-wiki ksr09l9nx0aebp91q2i9ymayaonyb09 0 331526 2421926 2325721 2026-06-01T17:13:45Z Atsuko-WMF 54082 Updating evenstream-internal 2421926 wikitext text/x-wiki jt7uv60p34um5ns15reo647csleisai 0 441095 2422130 2421583 2026-06-02T07:44:03Z AYounsi (WMF) 10040 /* RejectingBGPPrefixes */ 2422130 wikitext text/x-wiki 3iq5z6f553i6sklo669p8vil8k5p73m 498 443261 2421868 2405168 2026-06-01T15:40:59Z Stashbot 7414 taavi: restart memcached on cloudinfra-idp-1 T427826 2421868 wikitext text/x-wiki 1x5qmst5yy67z58stilhqtim4d0o8gj 498 443628 2421765 2421546 2026-06-01T13:31:23Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757972034 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1) 2421765 wikitext text/x-wiki kvuvyv1mjymakxq0p3iwrwouq493s5b 2421822 2421765 2026-06-01T15:02:55Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978145 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254) 2421822 wikitext text/x-wiki tfyay0pf8wr3b69l1so5ze3qka4g6q7 0 444677 2422028 2389700 2026-06-02T00:41:29Z RLazarus (WMF) 15215 /* Service ports */ add 4970 (function-evaluator) and 4973 (function-orchestrator callbacks) 2422028 wikitext text/x-wiki 1hxbxz14il7h3kzhvxj2zr4h1p0vxtj 2422029 2422028 2026-06-02T00:42:49Z RLazarus (WMF) 15215 4974, that is 2422029 wikitext text/x-wiki mjxldzbc69khhgj735gch1upxlv0rae 0 446462 2422211 2414101 2026-06-02T10:13:16Z HNowlan (WMF) 15760 /* Make sure you hear the notifications even if your phone is in do not disturb mode */ Add iOS 2422211 wikitext text/x-wiki 7y4gpayhtlgytql6ie3npn79b9rx4yx 2422212 2422211 2026-06-02T10:13:54Z HNowlan (WMF) 15760 /* Page all SREs (Batphone) */ Klaxon now pages the on-call rota, not batphone 2422212 wikitext text/x-wiki euz1ennankk0wnwv112bwtdmo8yhvce 2422213 2422212 2026-06-02T10:18:12Z HNowlan (WMF) 15760 /* Escalating a page */ Don't escalate to batphone, escalate to a manager. 2422213 wikitext text/x-wiki lqilb31zii2ukhpgt4luyq9hee9bnnv 498 446686 2422243 2386253 2026-06-02T11:20:19Z Stashbot 7414 taavi: renew puppet CA certificate T427905 2422243 wikitext text/x-wiki 7s3xajm10op3v2b6be9h1j4izt5llpc 0 446977 2421913 2417243 2026-06-01T16:50:01Z APaskulin (WMF) 14880 Add list of all pages 2421913 wikitext text/x-wiki d5fa5k44bx0ou5goznvktvnvtjre54q 498 447034 2421948 2418799 2026-06-01T18:32:06Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed f37884c87e (l10n updates: tr) 2421948 wikitext text/x-wiki mmdux7i04bqpuwfj79dtzmezl2d0jpz 10 448476 2421768 2421620 2026-06-01T13:35:21Z Ottomata 88 Added link to stream-internal 2421768 wikitext text/x-wiki 9610xuowmwrc90bg9i7i2ean8w2gafr 0 448996 2421927 2402053 2026-06-01T17:19:09Z DAdedoyin (WMF) 39150 /* Accounts & Privileges to get */ Add processor consoles 2421927 wikitext text/x-wiki fgulpy5w9gwtqop45iusfoeg7sk40yq 0 448997 2421776 2394339 2026-06-01T13:44:44Z DAdedoyin (WMF) 39150 /* Donation Interface */ 2421776 wikitext text/x-wiki maob46i4zpmgwknh0tqwzjspsfm7ac0 0 449160 2422026 2421638 2026-06-02T00:01:29Z Dexbot 30554 Bot: Updating the report 2422026 wikitext text/x-wiki kxoxlphfeb53c7i4fnrmbukyf3dcpxa 2422066 2422026 2026-06-02T05:53:10Z Dexbot 30554 Bot: Updating the report 2422066 wikitext text/x-wiki bl61gxyxvwceravc21xxxgwk2qeed74 2422095 2422066 2026-06-02T06:56:19Z Dexbot 30554 Bot: Updating the report 2422095 wikitext text/x-wiki gi7kvxcwwzow1epk57umvz0u0vy2qjj 2422142 2422095 2026-06-02T07:59:27Z Dexbot 30554 Bot: Updating the report 2422142 wikitext text/x-wiki 6uer99qds25efu485rmyq1ncrbkde1i 2422198 2422142 2026-06-02T09:38:39Z Dexbot 30554 Bot: Updating the report 2422198 wikitext text/x-wiki 4lslwp4q5d1un5txmczn5ju0clwrtmv 2422235 2422198 2026-06-02T11:05:51Z Dexbot 30554 Bot: Updating the report 2422235 wikitext text/x-wiki 8zw7olz2joq4kaeobotjsy8gmvlqbzm 0 451285 2422231 2380653 2026-06-02T11:02:02Z HNowlan (WMF) 15760 Add details about using OCO 2422231 wikitext text/x-wiki ncr01oh86zzdkqsh21n34lhyhlssqle 0 451490 2421979 2420976 2026-06-01T20:17:01Z LSandergreen-WMF 46345 /* Or best practices */ Remove drupal mention 2421979 wikitext text/x-wiki 4amwevs03lzdj0axu85hlecwx27f208 2421981 2421979 2026-06-01T20:20:08Z LSandergreen-WMF 46345 /* Deduping */ 2421981 wikitext text/x-wiki ibeksexutvu7htkaqu9rpurerxyn8cv 2421985 2421981 2026-06-01T20:23:11Z LSandergreen-WMF 46345 /* Custom fields */ 2421985 wikitext text/x-wiki m20i30iowg61z6pi7hxvtwujhfw8sh1 498 452890 2421764 2421547 2026-06-01T13:31:23Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757971428 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1) 2421764 wikitext text/x-wiki 623psuzeks03mpaxujk317fclw25btq 2421823 2421764 2026-06-01T15:03:38Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978339 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254) 2421823 wikitext text/x-wiki iowt3o786jpgg6bkt18jx7ddc4fbc9i 116 453906 2421677 2421283 2026-06-01T12:03:17Z Gitlabaccountapprovalbot 37332 watshell was rejected. 2421677 wikitext text/x-wiki nxbaeff3mnvb3pv082e6x3zr7meeq0s 2421819 2421677 2026-06-01T14:57:12Z Gitlabaccountapprovalbot 37332 vikram2101 was rejected. 2421819 wikitext text/x-wiki ecck3rzag3cb9xk2a4buwuotmt1av7k 2422061 2421819 2026-06-02T05:48:23Z Gitlabaccountapprovalbot 37332 @tmoney was approved. 2422061 wikitext text/x-wiki 92ztmu7wl0626igq21l6yzhimvn6wtc 0 456422 2421706 2368685 2026-06-01T12:44:55Z DAdedoyin (WMF) 39150 /* Gravy */ add documentation for Apple Pay domain association file 2421706 wikitext text/x-wiki qzauo04usr1x250ke3c7et5564m69gv 2421707 2421706 2026-06-01T12:45:30Z DAdedoyin (WMF) 39150 2421707 wikitext text/x-wiki 3l3ip6wdvjodvxnkfqsshm1j516i5og 2421709 2421707 2026-06-01T12:47:07Z DAdedoyin (WMF) 39150 /* Verifying merchant domains */ 2421709 wikitext text/x-wiki tdt7073fpafocgckkdxlu7hdinzw639 498 458787 2421762 2421545 2026-06-01T13:31:04Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757969997 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1) 2421762 wikitext text/x-wiki c1iotdzokwrglsyc016vs2hmgw95dse 2421821 2421762 2026-06-01T15:02:27Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978281 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254) 2421821 wikitext text/x-wiki i8hnsjax1nib36kqmbl3qgg0m8mya44 0 459699 2421754 2421074 2026-06-01T13:20:47Z Stashbot 7414 damilare: donorwiki upgraded from 9f3734e0 to 3bc70a73 2421754 wikitext text/x-wiki peiowmvn6gi77o8q31s019n1w1excih 2421964 2421754 2026-06-01T19:53:49Z Stashbot 7414 eileen: civicrm upgraded from a4a885f4 to 663c0c30 2421964 wikitext text/x-wiki m5jt46rjzry1wx382h67p5ijvb3ll79 0 460007 2421678 2398630 2026-06-01T12:03:27Z JGleeson (WMF) 11337 /* IDE-Embedded AI Chat Assistants */ point to the JetBrains AI T&Cs where they disclose background remote file context indexing. 2421678 wikitext text/x-wiki q67fln5mvf6k8n2jrp0mwlqi3nsw46l 0 460052 2422027 2421559 2026-06-02T00:34:58Z EggRoll97 37347 /* Access */ c/e 2422027 wikitext text/x-wiki hqejpx5rgi2wxsja4zynkqccixy1apc 0 460232 2421816 2421558 2026-06-01T14:53:33Z MPopov (WMF) 5891 Removed https 2421816 wikitext text/x-wiki ktiudk1se6otrfasy6lyda9qbvzsyaf 2421951 2421816 2026-06-01T19:07:49Z MPopov (WMF) 5891 2421951 wikitext text/x-wiki 3pmgmeikuitrfg58wcg15r2oeynozlh 2421966 2421951 2026-06-01T19:57:40Z MPopov (WMF) 5891 2421966 wikitext text/x-wiki cikg9r4g5hxidvce0juqtsawe6dpizg 0 460233 2421827 2421554 2026-06-01T15:11:36Z MPopov (WMF) 5891 2421827 wikitext text/x-wiki egk1itcs6m3f4of7m5cpkkxkuoikmvg 2421952 2421827 2026-06-01T19:21:08Z MPopov (WMF) 5891 /* Defining a new metric */ 2421952 wikitext text/x-wiki 1qf4p7o2gb9virz30v20ne8bjrht89i 2421965 2421952 2026-06-01T19:54:31Z MPopov (WMF) 5891 2421965 wikitext text/x-wiki dsg5la6iywb942tbw875bmtlc6y5xvw 2421969 2421965 2026-06-01T19:59:49Z MPopov (WMF) 5891 2421969 wikitext text/x-wiki ojb0b2xvb8rvdxd8mhffiis8jaa7sl6 2421973 2421969 2026-06-01T20:07:23Z MPopov (WMF) 5891 2421973 wikitext text/x-wiki pv0kjrbbr3og5y3ggpn01ghy4i28vj8 2421986 2421973 2026-06-01T20:25:18Z MPopov (WMF) 5891 /* Overview */ 2421986 wikitext text/x-wiki gzafgagagnwvoo1mxj35h9vqk26r5ye 0 460234 2421968 2421065 2026-06-01T19:59:40Z MPopov (WMF) 5891 2421968 wikitext text/x-wiki 4o8zt52n0uurc6fiepzmfyfpntrl2pu 0 460268 2421817 2421557 2026-06-01T14:53:38Z MPopov (WMF) 5891 Removed https 2421817 wikitext text/x-wiki 0hec3o48x99rwc5gntdcbxsv13l9ye6 0 460272 2422233 2026-06-02T11:03:14Z HNowlan (WMF) 15760 Redirected page to [[SRE/Oncall/Schedule]] 2422233 wikitext text/x-wiki 8nfyl5s2liqby2or21h1ml9g6v0peyd