Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.21 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2460282 2459786 2026-09-24T08:36:11Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2460282 wikitext text/x-wiki ojs71kz6ubx7587i5l5r5r95lc1p82r 2460286 2460282 2026-09-24T08:41:28Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2460286 wikitext text/x-wiki q2god1qpzawjftpoivcvgpkkjchomai 2460313 2460286 2026-09-24T09:06:11Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2460313 wikitext text/x-wiki 050g7h5cmk0pruhko3dnwl4mp8tespj 2460315 2460313 2026-09-24T09:11:41Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2460315 wikitext text/x-wiki qjttaq4lm14tlblzticco1z5l43gq27 0 3170 2460142 2442786 2026-09-23T19:17:25Z Taavi-WMF 41365 /* Automatic */ fix dns name 2460142 wikitext text/x-wiki ez43ixrj9aq0cemfazab3glhcrdg7m1 0 4108 2459800 2459733 2026-09-23T12:08:24Z ScheduleDeploymentBot 37566 Add [[gerrit:1344246]] to Thursday, September 24 UTC morning backport window 2459800 wikitext text/x-wiki 5u1pwl9muc773ehvsfsc625m4r5h8um 2459803 2459800 2026-09-23T12:17:43Z Adam Wight (WMDE) 3064 /* {{Deployment_day|date=2026-09-24}} */ switch deployment minder 2459803 wikitext text/x-wiki bujvhw6pu3k2kmnhylmlmprvnllr1ay 2460271 2459803 2026-09-24T08:03:54Z Adam Wight (WMDE) 3064 /* {{Deployment_day|date=2026-09-24}} */ mark not done—there was an ongoing incident which blocked deployment 2460271 wikitext text/x-wiki q3g6geaomhfk94hkzczfhztn7nutaud 2460338 2460271 2026-09-24T10:18:17Z ScheduleDeploymentBot 37566 Add [[gerrit:1344617]] to Thursday, September 24 UTC afternoon backport window 2460338 wikitext text/x-wiki gpftj4jg4oruibesds7v60gpjn2v6zi 2460343 2460338 2026-09-24T10:24:31Z ScheduleDeploymentBot 37566 Add [[gerrit:1344619]] to Thursday, September 24 UTC afternoon backport window 2460343 wikitext text/x-wiki 5jn2ohwh36hivrw1yx32r8ouy15lppp 2460357 2460343 2026-09-24T10:40:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1343957]] to Tuesday, September 29 UTC afternoon backport window 2460357 wikitext text/x-wiki ji3liie6whye9d7vlwoo0rgr39r07m3 2460358 2460357 2026-09-24T10:40:53Z ScheduleDeploymentBot 37566 Add [[gerrit:1343958]] to Tuesday, September 29 UTC afternoon backport window 2460358 wikitext text/x-wiki bp8xyhg3yzfsz753v7595bqx61nqdiz 2460363 2460358 2026-09-24T10:50:54Z ScheduleDeploymentBot 37566 Add [[gerrit:1344627]] to Thursday, September 24 UTC afternoon backport window 2460363 wikitext text/x-wiki bgtm07pjo5xkpiab2g5pxqld5la9htv 2460364 2460363 2026-09-24T10:58:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1344630]] to Thursday, September 24 UTC afternoon backport window 2460364 wikitext text/x-wiki ca47bnaui3d6zeorj0nyoke04b2dybu 2460369 2460364 2026-09-24T11:58:03Z Mmullie (WMF) 291 /* {{Deployment_day|date=2026-09-24}} */ 2460369 wikitext text/x-wiki c1u807atoxaukzfan0e55p3cen446f5 498 6726 2460280 2459758 2026-09-24T08:31:32Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2460280 wikitext text/x-wiki q2vp3l4maq2oyj8mbqxlgvt8r6l66eb 2460281 2460280 2026-09-24T08:36:05Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2460281 wikitext text/x-wiki 42wmgdpjema968z8j06478rygqdptta 2460297 2460281 2026-09-24T08:55:27Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api 2460297 wikitext text/x-wiki oh4okbf4rpw91cn9w37dzhfin8fc204 2460298 2460297 2026-09-24T08:55:31Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component alerts-api 2460298 wikitext text/x-wiki 0wahpnzy38mwbfgia8aakukugb4749r 2460301 2460298 2026-09-24T08:56:10Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2460301 wikitext text/x-wiki 90vtnxqohmfm732ib6csdizvsrpphuq 2460302 2460301 2026-09-24T08:57:59Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api 2460302 wikitext text/x-wiki q48zctayzs0juoqfyxssil9ruvzsy0h 2460304 2460302 2026-09-24T08:59:42Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2460304 wikitext text/x-wiki cvpsdch8osqrk94qqmg4bcvv4hp2rzr 2460309 2460304 2026-09-24T09:02:47Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2460309 wikitext text/x-wiki qn7lqqzwso2g7rmdr27x578afejmkd3 0 7919 2459796 2459794 2026-09-23T12:01:03Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343982|AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] 2459796 wikitext text/x-wiki 8a0erigvchmvc41yiw9e0sq7bgmxk8z 2459801 2459796 2026-09-23T12:15:11Z Stashbot 7414 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] 2459801 wikitext text/x-wiki qghzk457u62xz2ovopdb9jyo79mstah 2459802 2459801 2026-09-23T12:15:17Z Stashbot 7414 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] 2459802 wikitext text/x-wiki gb53t8l9w3vv8pr3tnet6cvco6nuhjh 2459804 2459802 2026-09-23T12:21:49Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343982|AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2459804 wikitext text/x-wiki g0daxnuco97hsji4g2siffkrde0nkpx 2459805 2459804 2026-09-23T12:22:48Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2459805 wikitext text/x-wiki s7m03y1eyzs8p9dyv3dpk7xi6mciu6v 2459806 2459805 2026-09-23T12:29:33Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 2459806 wikitext text/x-wiki 9rl8gci92cr041n4esqt3fzaqhyz6nn 2459807 2459806 2026-09-23T12:30:46Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd 2459807 wikitext text/x-wiki qp53ah4bt3if178uk9f1qf9c19fjvvc 2459808 2459807 2026-09-23T12:30:54Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) 2459808 wikitext text/x-wiki rghg7a8rjko36icte1kww74wtsax5yr 2459809 2459808 2026-09-23T12:32:01Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd 2459809 wikitext text/x-wiki 6mdk7g8qvpj9891mthz9zbq3j2rt7oj 2459810 2459809 2026-09-23T12:32:02Z Stashbot 7414 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) 2459810 wikitext text/x-wiki j67q73irk7qkc3x83au0yv4i57i4pr1 2459811 2459810 2026-09-23T12:32:38Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd 2459811 wikitext text/x-wiki ef4llcdizo52496okccb7xaupv0kdji 2459812 2459811 2026-09-23T12:32:42Z Stashbot 7414 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) 2459812 wikitext text/x-wiki l1r489wjfvlls498beuy4dnt4kfsudj 2459813 2459812 2026-09-23T12:33:36Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd 2459813 wikitext text/x-wiki 5n3n1jt49exofkry8b9sh2cd7t5m20s 2459814 2459813 2026-09-23T12:33:50Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) 2459814 wikitext text/x-wiki dgjej0tdt2xf2cqeq9xwyfbwir931go 2459815 2459814 2026-09-23T12:34:36Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343982|AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] (duration: 33m 33s) 2459815 wikitext text/x-wiki kw1zjgkwuq2eyi5nosbzpai5e1bfpqz 2459816 2459815 2026-09-23T12:38:59Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd 2459816 wikitext text/x-wiki nrx84n5271yp22bas23duv3gjfbwxbg 2459817 2459816 2026-09-23T12:39:06Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) 2459817 wikitext text/x-wiki p2iq461c3yjjw871o6dmmke0by4gyto 2459818 2459817 2026-09-23T12:39:47Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 2459818 wikitext text/x-wiki rl6b3swe7wlgoektuihf8ovwbr7rbd3 2459820 2459818 2026-09-23T12:44:57Z Stashbot 7414 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad 2459820 wikitext text/x-wiki j8gygmisn5ufrvvtfcwfj8jd2ucw9ff 2459821 2459820 2026-09-23T12:45:23Z Stashbot 7414 awight: add seanleong-wmde to deployment-prep 2459821 wikitext text/x-wiki 5158w43xoeulpdzr89835ekcny8qsyd 2459822 2459821 2026-09-23T12:55:51Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd 2459822 wikitext text/x-wiki lhs4e6a6qszdw2ktns9cgo7f9hy2104 2459823 2459822 2026-09-23T12:55:59Z Stashbot 7414 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) 2459823 wikitext text/x-wiki l2i7qgdt9z2mwbdrvdcdnw7cko05vis 2459824 2459823 2026-09-23T12:56:10Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd 2459824 wikitext text/x-wiki c0ng1moyk6t42jl8t3l56xnj4it2a76 2459825 2459824 2026-09-23T12:56:11Z Stashbot 7414 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) 2459825 wikitext text/x-wiki 4pcmetzsmoewzeeih2or3b9x65ud6pz 2459826 2459825 2026-09-23T12:56:16Z Stashbot 7414 awight: manually run puppet agent 2459826 wikitext text/x-wiki 72gjy9avaydv1qf5fkaa3flvn4o1tc9 2459828 2459826 2026-09-23T12:57:24Z Stashbot 7414 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd 2459828 wikitext text/x-wiki qqb3w4dzdkdnilb7xvv807ubhos3jzh 2459829 2459828 2026-09-23T12:57:26Z Stashbot 7414 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) 2459829 wikitext text/x-wiki nlbef57lrt1hqg5wu9no39nwm18rz6q 2459831 2459829 2026-09-23T13:04:05Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 2459831 wikitext text/x-wiki dy2soz5zdbjyrf7ait151em2tbl5kws 2459832 2459831 2026-09-23T13:09:19Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 2459832 wikitext text/x-wiki qjh811tn6ooooez21ccjq4g37sxdjjb 2459833 2459832 2026-09-23T13:10:32Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 2459833 wikitext text/x-wiki qc2fnvk01k6ki33um7uke49pvef1pko 2459835 2459833 2026-09-23T13:13:22Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 2459835 wikitext text/x-wiki qfxrgus4ajgfjwc10zm2i5ic03zv8hb 2459837 2459835 2026-09-23T13:18:45Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie 2459837 wikitext text/x-wiki dcvf23ud2d9kewgmmgxvec96ab6w7o7 2459840 2459837 2026-09-23T13:24:27Z Stashbot 7414 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues 2459840 wikitext text/x-wiki 25sfvwl63r84tjj0vm2po1ol7olgvd6 2459841 2459840 2026-09-23T13:24:29Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 2459841 wikitext text/x-wiki 30c5995p2nrcro2gwk7djhaclcmpbkg 2459842 2459841 2026-09-23T13:25:30Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 2459842 wikitext text/x-wiki c8ea4hpchjqbd95sbm2rqueslz54zmu 2459845 2459842 2026-09-23T13:29:26Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) 2459845 wikitext text/x-wiki g5dg2hknan00g68l12x5b4gcleptq64 2459846 2459845 2026-09-23T13:29:30Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet 2459846 wikitext text/x-wiki p5hay56opezmn2cj5xfpiw8x4i1znmt 2459848 2459846 2026-09-23T13:30:04Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet 2459848 wikitext text/x-wiki b44wvlichqqd64bexpgvbku44kd195b 2459851 2459848 2026-09-23T13:35:55Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie 2459851 wikitext text/x-wiki pe3roii9j46h9odpxzden7im2i125ng 2459852 2459851 2026-09-23T13:37:50Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet 2459852 wikitext text/x-wiki 8fl4ji131ttfscxoikh3qkmpfeuu7h6 2459853 2459852 2026-09-23T13:37:52Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet 2459853 wikitext text/x-wiki pueha9zjdzdkj858ctynsyo2zlrchkm 2459854 2459853 2026-09-23T13:37:53Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) 2459854 wikitext text/x-wiki 6eariku5ecwa5q371hkzv4vqxoseicx 2459855 2459854 2026-09-23T13:38:54Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828) 2459855 wikitext text/x-wiki 3ior5d0rslymsbfaxeb2r7fr8lfl1kd 2459856 2459855 2026-09-23T13:53:18Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage 2459856 wikitext text/x-wiki 8tkjztyoqtwpsd7exeehp0z6x9x5pvo 2459858 2459856 2026-09-23T13:56:29Z Stashbot 7414 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie 2459858 wikitext text/x-wiki n5mf7dmv5pj1z7wrf5h215e4k2ftg9b 2459861 2459858 2026-09-23T13:58:26Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage 2459861 wikitext text/x-wiki 6gkl5qndl16pwk396ytqb1w9fquwl1p 2459864 2459861 2026-09-23T14:07:48Z Stashbot 7414 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2459864 wikitext text/x-wiki il52pi7qcyer350wyh079f16ohfylc2 2459865 2459864 2026-09-23T14:07:55Z Stashbot 7414 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2459865 wikitext text/x-wiki ju8an3f5srph1xjvjxv1da8gh9iwjpo 2459866 2459865 2026-09-23T14:09:24Z Stashbot 7414 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2459866 wikitext text/x-wiki emhq8jvlpnc45eci7a8p3nf0mbn0qdw 2459867 2459866 2026-09-23T14:10:15Z Stashbot 7414 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2459867 wikitext text/x-wiki hunsylvmpcz93d4cpyuq12yt4gthmgo 2459870 2459867 2026-09-23T14:17:39Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie 2459870 wikitext text/x-wiki lie5xy1t09vd3idfuw6eqtukmud2wcv 2459880 2459870 2026-09-23T14:23:06Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344292|feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293|feat(AddLink): Do not resuggest an already reviewed page (T429417)]] 2459880 wikitext text/x-wiki 9m6nxls9ld1uiuqnmtgmsvg4isgngiw 2459881 2459880 2026-09-23T14:23:26Z Stashbot 7414 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2459881 wikitext text/x-wiki nfggx19pqntubt2zvtlja0xjr490biz 2459883 2459881 2026-09-23T14:24:12Z Stashbot 7414 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2459883 wikitext text/x-wiki 3xwbijkjo6vl53v98h7j90o7780dq7c 2459891 2459883 2026-09-23T14:26:07Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2459891 wikitext text/x-wiki qhaak9ha14ev7qyaq7s07ud0b2tz7ec 2459892 2459891 2026-09-23T14:26:14Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2459892 wikitext text/x-wiki jdbqx18n0x9udh11dzb0pfagxenpsqf 2459898 2459892 2026-09-23T14:28:19Z Stashbot 7414 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.* 2459898 wikitext text/x-wiki 6fy7lqye65q4m05y4uyzonkh86bolrf 2459900 2459898 2026-09-23T14:28:37Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie 2459900 wikitext text/x-wiki j3a6v9dlo8ptl69v1xf6akasrbs1mis 2459901 2459900 2026-09-23T14:36:16Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828) 2459901 wikitext text/x-wiki 0kxsvvngax38phqquilmls0dgz9znoc 2459902 2459901 2026-09-23T14:36:18Z Stashbot 7414 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2459902 wikitext text/x-wiki 9f70qd2z6hvb4hx8wt3m4ksuoxj0rbb 2459903 2459902 2026-09-23T14:36:48Z Stashbot 7414 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2459903 wikitext text/x-wiki g3kcz8gqi3ijti4fnk01ythj0mm8gcl 2459904 2459903 2026-09-23T14:37:18Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2459904 wikitext text/x-wiki 8z348fwa3g16guav6wjc68x6di59xap 2459905 2459904 2026-09-23T14:37:30Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828) 2459905 wikitext text/x-wiki 8mke70ouuq6pv7dfkyhizfru2cs58t3 2459906 2459905 2026-09-23T14:37:41Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344292|feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293|feat(AddLink): Do not resuggest an already reviewed page (T429417)]] (duration: 14m 34s) 2459906 wikitext text/x-wiki 2444hc9nvjol13alsr679yf4hd51jod 2459907 2459906 2026-09-23T14:37:57Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2459907 wikitext text/x-wiki d0kxqcova5ze2trd6n3tnrpszuip2hp 2459908 2459907 2026-09-23T14:38:08Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2459908 wikitext text/x-wiki r0hnbkc0cywps4q6k4l9pd8wp0yxljz 2459909 2459908 2026-09-23T14:38:43Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2459909 wikitext text/x-wiki mdq2eddh604ksd5dl773u7v43yufz48 2459910 2459909 2026-09-23T14:39:31Z Stashbot 7414 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866 2459910 wikitext text/x-wiki am41vtoohklgxy9liq1jkiaeeszdox3 2459911 2459910 2026-09-23T14:40:08Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 2459911 wikitext text/x-wiki 3hhwefeqstgqqoo660t9rdxyqukblrc 2459912 2459911 2026-09-23T14:40:57Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 2459912 wikitext text/x-wiki l8gpwl1n61wvs9pnwxnuzq8oqphqrhu 2459913 2459912 2026-09-23T14:42:08Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie 2459913 wikitext text/x-wiki aij6jctmawrene55qxxsljtc2vfbp8n 2459914 2459913 2026-09-23T14:42:18Z Stashbot 7414 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] 2459914 wikitext text/x-wiki rxvbyza14l4fj2gruxfjhgs26du07pe 2459915 2459914 2026-09-23T14:42:20Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 2459915 wikitext text/x-wiki 4c32whalth8rklw0245c5q71z6myacl 2459916 2459915 2026-09-23T14:42:30Z Stashbot 7414 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s) 2459916 wikitext text/x-wiki cp3jd3yd4odm20xuxgydo8r84712yg3 2459917 2459916 2026-09-23T14:42:43Z Stashbot 7414 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2459917 wikitext text/x-wiki g1bdhs30g4pytnle1f0twpntk2tknkx 2459919 2459917 2026-09-23T14:43:41Z Stashbot 7414 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2459919 wikitext text/x-wiki dvvwgjo0w2yjiub5op3uxtr8mhor0jn 2459920 2459919 2026-09-23T14:43:59Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 2459920 wikitext text/x-wiki bdpcifzdkeoqtufui30l1o1m9hf7woe 2459921 2459920 2026-09-23T14:44:28Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 2459921 wikitext text/x-wiki omcudh2507qz0z84s8aerd8obudsgwk 2459922 2459921 2026-09-23T14:44:43Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2459922 wikitext text/x-wiki 4sq7310ezy15sh9tdcx90nsyizpt8uw 2459923 2459922 2026-09-23T14:45:18Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2459923 wikitext text/x-wiki f62hh6guiczf728rbye7wtxxfvgvz0a 2459924 2459923 2026-09-23T14:45:26Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2459924 wikitext text/x-wiki hbw8usu2rw7lcpdc72x4cxhli18rilu 2459925 2459924 2026-09-23T14:45:28Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 2459925 wikitext text/x-wiki je4j9oyvgmwprfjfxaf57us86mki7a0 2459926 2459925 2026-09-23T14:46:26Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 2459926 wikitext text/x-wiki lln317axp0itp6m1jimeoia05z7tpmy 2459927 2459926 2026-09-23T14:47:00Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 2459927 wikitext text/x-wiki 9fw6kt2jtmd6n8qekj3x861t9cqsvur 2459928 2459927 2026-09-23T14:47:20Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2459928 wikitext text/x-wiki d8fojchgsdm5qaf5w303rth2zhlcigp 2459929 2459928 2026-09-23T14:47:33Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2459929 wikitext text/x-wiki e5jztxkys0k5exc86rnfgjxt30ao21n 2459934 2459929 2026-09-23T14:53:04Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 2459934 wikitext text/x-wiki 0o7ddqttvgjbrkcoc02dkg4j8xwuaxx 2459936 2459934 2026-09-23T14:53:25Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2459936 wikitext text/x-wiki s0zc6u6u6gzslb3wky4ouwisdfmrkr6 2459939 2459936 2026-09-23T14:53:54Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 2459939 wikitext text/x-wiki 3qo1wd0n77ovb7xwbzmf019dbwpspzp 2459942 2459939 2026-09-23T14:54:13Z Stashbot 7414 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance 2459942 wikitext text/x-wiki 6f08i8qn0zxdm8bpq6m8r1m6gg7xhwk 2459943 2459942 2026-09-23T14:54:13Z Stashbot 7414 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance 2459943 wikitext text/x-wiki n5kg9hkus5dlhlh6ommkxqy9lmh4fix 2459945 2459943 2026-09-23T14:54:33Z Stashbot 7414 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance 2459945 wikitext text/x-wiki 82kof2ifernkbamu67seo9b4ksk7sku 2459946 2459945 2026-09-23T14:54:40Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 2459946 wikitext text/x-wiki sdlo05h4l0l72mfenvc9towoxcbphb2 2459949 2459946 2026-09-23T14:55:10Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 2459949 wikitext text/x-wiki l8l5be96x75w965eqvo5ftkn9tql8qi 2459951 2459949 2026-09-23T14:55:27Z Stashbot 7414 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance 2459951 wikitext text/x-wiki rpvzntqhs2nbvit5hj65ao8w5im0voq 2459954 2459951 2026-09-23T14:55:51Z Stashbot 7414 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance 2459954 wikitext text/x-wiki nu64t5zavfjodjr7irst1enpwoyxaoo 2459955 2459954 2026-09-23T14:55:52Z Stashbot 7414 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance 2459955 wikitext text/x-wiki f95lry3ydozl0lpphp7mfksnvzjtnlq 2459959 2459955 2026-09-23T14:56:24Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage 2459959 wikitext text/x-wiki aobypecrdmd20co08xhoys6iwyoarm5 2459965 2459959 2026-09-23T14:57:41Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply 2459965 wikitext text/x-wiki l25r0q7mt3y3t1q6abxjfwpytsbpcdo 2459966 2459965 2026-09-23T14:57:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply 2459966 wikitext text/x-wiki cx9ytivxvofdm5j6z9g20k8m4nvcw1k 2459967 2459966 2026-09-23T14:57:49Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply 2459967 wikitext text/x-wiki dqq2fk2tx4w7ycwyfwc7ztz891q6q4a 2459968 2459967 2026-09-23T14:57:55Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply 2459968 wikitext text/x-wiki cxd65s7gae7hpha5jcqpotzzrwe0jua 2459970 2459968 2026-09-23T14:58:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply 2459970 wikitext text/x-wiki 4o29ji8gp9v1eyxigvjmixe2nw811st 2459971 2459970 2026-09-23T14:58:03Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply 2459971 wikitext text/x-wiki 67fdf7lnaetyk60vbzqsuyizqnhpbto 2459983 2459971 2026-09-23T15:00:48Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply 2459983 wikitext text/x-wiki oh602jlqtm8q7fjd3tf5nyl6ge2z3f8 2459985 2459983 2026-09-23T15:00:56Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s) 2459985 wikitext text/x-wiki nn1upznzcztt2r5xel6ziy5zdq1npvi 2459986 2459985 2026-09-23T15:01:11Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply 2459986 wikitext text/x-wiki ad0zwwlh082di6jjuqywa7tngfxqits 2459987 2459986 2026-09-23T15:01:16Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply 2459987 wikitext text/x-wiki nvc7cai4xyhhk5xg401lxrhn8tbie3n 2459988 2459987 2026-09-23T15:01:42Z Stashbot 7414 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad 2459988 wikitext text/x-wiki a26ffes9jjyhatlorvyypop5dxeno9j 2459989 2459988 2026-09-23T15:01:44Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply 2459989 wikitext text/x-wiki im9q0d63hg0jfqju7gapnlxdj866u57 2459990 2459989 2026-09-23T15:01:54Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply 2459990 wikitext text/x-wiki 8vi0f0zfev1jgyu9duzigpupicwefc4 2459991 2459990 2026-09-23T15:02:21Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply 2459991 wikitext text/x-wiki jbsbs4ohip1loxsj0qfwrns2v6wnv28 2459992 2459991 2026-09-23T15:02:43Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage 2459992 wikitext text/x-wiki ld9hphbik6p99m0ddskq0f5xen5nojh 2459993 2459992 2026-09-23T15:02:58Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts 2459993 wikitext text/x-wiki 2h4qnmn0jtyz2nzz6huhzuv43knkffv 2459994 2459993 2026-09-23T15:06:37Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828) 2459994 wikitext text/x-wiki dieb8c2bzual6mk0757maep15ubnoaz 2459995 2459994 2026-09-23T15:12:08Z Stashbot 7414 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618 2459995 wikitext text/x-wiki 2gb38tim17uwc54pzw5b1csoan5h80n 2459996 2459995 2026-09-23T15:12:49Z Stashbot 7414 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad 2459996 wikitext text/x-wiki 6u6hpk4c4n4o1hd284m4yj8q65bs0d9 2459997 2459996 2026-09-23T15:12:53Z Stashbot 7414 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad 2459997 wikitext text/x-wiki j02nq0341guhphb4v3yoff7q2ob0ka7 2459998 2459997 2026-09-23T15:14:37Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage 2459998 wikitext text/x-wiki gzodg4hmecq9yq19xkt6qt88eblt3ws 2459999 2459998 2026-09-23T15:18:18Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage 2459999 wikitext text/x-wiki d7nc3eh45nqmbvnqqx40e1bp2zttse5 2460001 2459999 2026-09-23T15:19:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2460001 wikitext text/x-wiki 1eiwvhy8hhsogrnmlkkb1wiz802pvsf 2460002 2460001 2026-09-23T15:20:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2460002 wikitext text/x-wiki t7pllqxsx40n6tkhodfozwea33l545b 2460003 2460002 2026-09-23T15:20:30Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply 2460003 wikitext text/x-wiki 09ixjtil0qosb3byb4bbs6xaqu1qnv0 2460004 2460003 2026-09-23T15:21:13Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply 2460004 wikitext text/x-wiki 5tgfr7rrxgx9pu47tk3k887i6li3m2f 2460005 2460004 2026-09-23T15:21:20Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2460005 wikitext text/x-wiki al9dhbei0my3kg2mkfj3n31d65z4rbx 2460006 2460005 2026-09-23T15:21:52Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie 2460006 wikitext text/x-wiki 73btj1lfdmko665aakbgvkq0g6wc02j 2460008 2460006 2026-09-23T15:23:14Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade (T438828) 2460008 wikitext text/x-wiki piuhs5qgnosldlln7ymv80u8ni9hju1 2460009 2460008 2026-09-23T15:23:33Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2460009 wikitext text/x-wiki bff0us1cdgnx5og19xj1712jua5nnif 2460012 2460009 2026-09-23T15:32:07Z Stashbot 7414 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 T438961 2460012 wikitext text/x-wiki 50zvsbthgl2s8v3pvf6p0jdzlk6mztt 2460013 2460012 2026-09-23T15:33:44Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 2460013 wikitext text/x-wiki m40wt1u4yddwdxxru9vu058sup6edlh 2460015 2460013 2026-09-23T15:34:06Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.netbox 2460015 wikitext text/x-wiki 69650zgyb9mwe6vh882u45s9aolbh0l 2460016 2460015 2026-09-23T15:34:47Z Stashbot 7414 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie 2460016 wikitext text/x-wiki nwawex3ut9avpecv7irtdwtvk5540fy 2460017 2460016 2026-09-23T15:35:00Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2460017 wikitext text/x-wiki aqtwalmo584tftp2odb2tkfodvz8qno 2460018 2460017 2026-09-23T15:35:17Z Stashbot 7414 mutante: zuul1005 - reimage - should not have had nftables on it before T438786 2460018 wikitext text/x-wiki bggj4zjeolc291sjkxf9d72fvgnflfw 2460019 2460018 2026-09-23T15:35:20Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2460019 wikitext text/x-wiki oc7sjojy0vmkipkm7xjimybmsn1bra5 2460020 2460019 2026-09-23T15:35:25Z Stashbot 7414 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d T438961 2460020 wikitext text/x-wiki owkqp18m0x8zehv616iv8etcxnh675h 2460021 2460020 2026-09-23T15:36:01Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2460021 wikitext text/x-wiki l3w8ow1qzxae83dtald8nrbm709q5un 2460022 2460021 2026-09-23T15:36:24Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2460022 wikitext text/x-wiki ma6i77375u2xusr1dq0z41hnlsqp7sq 2460023 2460022 2026-09-23T15:37:59Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" 2460023 wikitext text/x-wiki hz8jexhthzp1oakaxnrmnukcymoa2nw 2460024 2460023 2026-09-23T15:38:22Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" 2460024 wikitext text/x-wiki 3ndgkg54971qshmd034u9rms1zc2d0z 2460025 2460024 2026-09-23T15:38:23Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2460025 wikitext text/x-wiki h31nhefokyeeaix30o6dai66ix7e7ec 2460026 2460025 2026-09-23T15:38:26Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors 2460026 wikitext text/x-wiki 64gjn30883d0o4fk6arthrpx9qqtcmp 2460027 2460026 2026-09-23T15:38:28Z Stashbot 7414 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ T438961 2460027 wikitext text/x-wiki g8c6y4emcb8ab3l6yhokubp7myetphx 2460028 2460027 2026-09-23T15:38:29Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors 2460028 wikitext text/x-wiki ixjuz2huxziy8qfq1l4ra8lv27ldqry 2460029 2460028 2026-09-23T15:38:30Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059 2460029 wikitext text/x-wiki 9mtywdyx1ghp1dajogwp5oxzg1yjya3 2460030 2460029 2026-09-23T15:38:43Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059 2460030 wikitext text/x-wiki g7yqlv7qj0pw8r4juiuoyijnofazkde 2460031 2460030 2026-09-23T15:39:19Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059 2460031 wikitext text/x-wiki qig044xove1gvqaqik9jnfe95ktioi8 2460032 2460031 2026-09-23T15:41:40Z Stashbot 7414 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 T438961 2460032 wikitext text/x-wiki om4loy1g3aljii3m5qyncydb85x2eq7 2460033 2460032 2026-09-23T15:41:59Z Stashbot 7414 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook' 2460033 wikitext text/x-wiki pnol6qkfl5fc06f98mhbzco28720lec 2460034 2460033 2026-09-23T15:42:12Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2460034 wikitext text/x-wiki sbk0iaesr1n5qaef5vryets6b5r870s 2460035 2460034 2026-09-23T15:42:22Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie 2460035 wikitext text/x-wiki i613q6u1jja4vctgl6s17wdj4dxdvib 2460036 2460035 2026-09-23T15:43:48Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie 2460036 wikitext text/x-wiki 56xvt2q70i287qjxbtoy5dlgmrol7vn 2460037 2460036 2026-09-23T15:46:11Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: testing 2460037 wikitext text/x-wiki 2r4n4t9udlyofko9yswq4ldhwo3ol5l 2460038 2460037 2026-09-23T15:50:19Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2460038 wikitext text/x-wiki 8vrxn4oncup5t9rmz034iqm6d1x8rla 2460039 2460038 2026-09-23T15:50:36Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2460039 wikitext text/x-wiki sd9nslrymb8xuaqwjopck8otdnjnhm4 2460041 2460039 2026-09-23T15:51:31Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2460041 wikitext text/x-wiki rokab6291linzwm1a4za26jvjgd5h0a 2460042 2460041 2026-09-23T15:52:31Z Stashbot 7414 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie 2460042 wikitext text/x-wiki 38amuub7r9zx1h6rrni2mgjtam56l4z 2460043 2460042 2026-09-23T15:52:43Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie 2460043 wikitext text/x-wiki 0l1ygqiosfsrcfu0t77gacs7yozc9zx 2460044 2460043 2026-09-23T15:53:17Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s) 2460044 wikitext text/x-wiki foybaskh2a3ktznpi5ou6zi4dptb5n4 2460045 2460044 2026-09-23T15:54:10Z Stashbot 7414 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.* 2460045 wikitext text/x-wiki d7kr2kgv6fcnxfm1vow6fx13sqviq9m 2460046 2460045 2026-09-23T15:54:54Z Stashbot 7414 moritzm: installing cjose security updates 2460046 wikitext text/x-wiki 0kuutrgdr7onlnrg9ehap5skju6bpsd 2460047 2460046 2026-09-23T15:55:56Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade (T438828) 2460047 wikitext text/x-wiki sohxqhf1iiza19aydakhvu02uk8yjnt 2460049 2460047 2026-09-23T16:10:33Z Stashbot 7414 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie 2460049 wikitext text/x-wiki eh9pkh8db3mww9x40hl8mer0kmpdhv4 2460050 2460049 2026-09-23T16:10:54Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie 2460050 wikitext text/x-wiki 3eh5z0yz6at737ji7h39ozps83d6dls 2460053 2460050 2026-09-23T16:29:25Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie 2460053 wikitext text/x-wiki gb17ubrbdnqkg8dit7k2ay8ynkdnh9m 2460054 2460053 2026-09-23T16:30:31Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2460054 wikitext text/x-wiki qzkk6hke61erghvpzyvjr8wownzsphr 2460055 2460054 2026-09-23T16:34:19Z Stashbot 7414 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie 2460055 wikitext text/x-wiki 2re10kuvuur13b0e90ouyitvph3wye9 2460056 2460055 2026-09-23T16:35:39Z Stashbot 7414 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie 2460056 wikitext text/x-wiki esdv1zbbnylshg4npsfgyhcqtg6jnmi 2460059 2460056 2026-09-23T16:39:59Z Stashbot 7414 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie 2460059 wikitext text/x-wiki t5jpv8j2zy7gu63bx70761fg74pyygl 2460064 2460059 2026-09-23T16:48:31Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie 2460064 wikitext text/x-wiki irltipj2ozlpalji58w59r4hxyytwkc 2460065 2460064 2026-09-23T16:51:13Z Stashbot 7414 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie 2460065 wikitext text/x-wiki m6n7xvuw8l7bmd1un4auhb8jocm5bqk 2460067 2460065 2026-09-23T16:51:34Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie 2460067 wikitext text/x-wiki fx0zsq4ncoz6lnyemb0dg7bcp4zj6z3 2460068 2460067 2026-09-23T16:59:15Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage 2460068 wikitext text/x-wiki 1b05pkl80l75s6xgd78vy76i69t1r7b 2460069 2460068 2026-09-23T17:02:20Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage 2460069 wikitext text/x-wiki 95poz4yl0yrvx4bzh07ip3m9wnpj3yu 2460070 2460069 2026-09-23T17:07:59Z Stashbot 7414 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354 2460070 wikitext text/x-wiki 9vuy7nnr98n4f1c5izzr4kvqc1tiq3n 2460071 2460070 2026-09-23T17:10:44Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage 2460071 wikitext text/x-wiki 1pu8up5vxawq1rpem8gjluv3jemvzt1 2460072 2460071 2026-09-23T17:11:29Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049 2460072 wikitext text/x-wiki d2z9tuolbznrp1krgozir2c9e7ctp9c 2460073 2460072 2026-09-23T17:11:35Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet 2460073 wikitext text/x-wiki feeanoaqp5js4g21wn0e9npk00ndxn1 2460074 2460073 2026-09-23T17:14:33Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage 2460074 wikitext text/x-wiki imwapjha3qsxz5jdtmyze1yba5l8j0p 2460075 2460074 2026-09-23T17:16:11Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2460075 wikitext text/x-wiki sw82p7wt6vhehdyzwziay4dh84pm8g8 2460076 2460075 2026-09-23T17:17:49Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2460076 wikitext text/x-wiki qzujlx4v8etshomgqds0hm5rpc62i9p 2460077 2460076 2026-09-23T17:18:27Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2460077 wikitext text/x-wiki tucpbwi0si6nuisoaprqg2amtuzib6n 2460078 2460077 2026-09-23T17:21:34Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2460078 wikitext text/x-wiki dz2jloii3p46yd2sz5vtszai7xt3dya 2460079 2460078 2026-09-23T17:22:36Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2460079 wikitext text/x-wiki np4v8e3dprnu5wykgcr7boufwtdxwzm 2460080 2460079 2026-09-23T17:23:29Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] 2460080 wikitext text/x-wiki 7r315uaz8e4aaem0r20vqtlfmlwzt7r 2460081 2460080 2026-09-23T17:24:47Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified] 2460081 wikitext text/x-wiki 6j4a2lyf5ql99kwjtutzr4fqv2karzd 2460082 2460081 2026-09-23T17:35:44Z Stashbot 7414 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] 2460082 wikitext text/x-wiki 737kn0c60h092hf03y7irff1rua5jqx 2460083 2460082 2026-09-23T17:35:45Z Stashbot 7414 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified] 2460083 wikitext text/x-wiki spm3dhbfncgw6yal1hkrjnw6kztg4e2 2460084 2460083 2026-09-23T17:35:49Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie 2460084 wikitext text/x-wiki p7h50ypn5trnvpcm5yseeo94r25anje 2460085 2460084 2026-09-23T17:35:51Z Stashbot 7414 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - T439010 2460085 wikitext text/x-wiki gkc70z45y6e7zfm2ot6n48sp3ftojor 2460086 2460085 2026-09-23T17:37:03Z Stashbot 7414 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie 2460086 wikitext text/x-wiki rm415qu0y92tdedf9chtxbsdky91jz4 2460087 2460086 2026-09-23T17:43:23Z Stashbot 7414 taavi@dns1004: START - running authdns-update 2460087 wikitext text/x-wiki s1l2mpu25bd4e703417lfalc2luu3je 2460088 2460087 2026-09-23T17:46:03Z Stashbot 7414 taavi@dns1004: END - running authdns-update 2460088 wikitext text/x-wiki faeclkatdwfb8h2fzeztpggcpnq65ce 2460090 2460088 2026-09-23T17:51:55Z Stashbot 7414 taavi@dns1004: START - running authdns-update 2460090 wikitext text/x-wiki ro768v1lecb4vecb3tet8mm9p0z311w 2460092 2460090 2026-09-23T17:53:10Z Stashbot 7414 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage 2460092 wikitext text/x-wiki 7hwweja8omyl3kmwl5k86cfw17j7cnr 2460093 2460092 2026-09-23T17:54:17Z Stashbot 7414 taavi@dns1004: END - running authdns-update 2460093 wikitext text/x-wiki 76hmt8dljvf5cbd9avgqmf3t1cw4wcr 2460094 2460093 2026-09-23T17:57:55Z Stashbot 7414 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage 2460094 wikitext text/x-wiki 3ioyl5la14o6whssd9wvcs6ale2gb7h 2460095 2460094 2026-09-23T17:58:41Z Stashbot 7414 cdanis@dns1005: START - running authdns-update 2460095 wikitext text/x-wiki b8w4m80badbag7wxgmovhq9v40he8vs 2460096 2460095 2026-09-23T18:01:05Z Stashbot 7414 cdanis@dns1005: END - running authdns-update 2460096 wikitext text/x-wiki eapial0t9lwgrg6fa4x2k5rqrowa4eh 2460097 2460096 2026-09-23T18:02:47Z Stashbot 7414 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors 2460097 wikitext text/x-wiki l6imv36u2jh0bykg9lm56dgvci3dcyj 2460098 2460097 2026-09-23T18:02:58Z Stashbot 7414 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors 2460098 wikitext text/x-wiki gpxlqr2t07qk0e12yhr7gbtfp7ekb22 2460099 2460098 2026-09-23T18:03:20Z Stashbot 7414 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors 2460099 wikitext text/x-wiki t8d0sn795o9zjstx3t8n1rh47y6yzmu 2460100 2460099 2026-09-23T18:03:34Z Stashbot 7414 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors 2460100 wikitext text/x-wiki rjxem2e4hl3ieboiatk50g7t68852yz 2460101 2460100 2026-09-23T18:05:31Z Stashbot 7414 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors 2460101 wikitext text/x-wiki 7cbg5268898zdeerua6lm2vxdtm90rk 2460102 2460101 2026-09-23T18:05:41Z Stashbot 7414 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors 2460102 wikitext text/x-wiki bu54a4l3k6gpy9zr0i6s349x9ofpswz 2460103 2460102 2026-09-23T18:08:39Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors 2460103 wikitext text/x-wiki q3xnouzew4a2922celrakhjxh9r5g94 2460104 2460103 2026-09-23T18:08:49Z Stashbot 7414 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors 2460104 wikitext text/x-wiki ouhcyrutmpeshtf5lheve9eqj7pvfmv 2460106 2460104 2026-09-23T18:15:01Z Stashbot 7414 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie 2460106 wikitext text/x-wiki dshy9rhsf10phzon22tx1u4ki77kl6v 2460108 2460106 2026-09-23T18:28:53Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2460108 wikitext text/x-wiki syuk0hlmfyvuzo50ju82fbbxb7f8tky 2460109 2460108 2026-09-23T18:29:07Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2460109 wikitext text/x-wiki hcso3vbi1hkt2odquv3v2ufgkpua5f6 2460110 2460109 2026-09-23T18:30:53Z Stashbot 7414 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org 2460110 wikitext text/x-wiki dip6kpafo4wxyf8hipbl3py2am4rfk1 2460111 2460110 2026-09-23T18:36:36Z Stashbot 7414 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org 2460111 wikitext text/x-wiki 77my9dkqhnrdlct2jvwz2nshmwowg5k 2460112 2460111 2026-09-23T18:36:39Z Stashbot 7414 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org 2460112 wikitext text/x-wiki rv32b0pzzc4gaf0se48jya2p7tz1aw8 2460113 2460112 2026-09-23T18:36:44Z Stashbot 7414 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org 2460113 wikitext text/x-wiki 80911ari0et641a7qee1jsqcgp7r1sw 2460114 2460113 2026-09-23T18:36:45Z Stashbot 7414 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org 2460114 wikitext text/x-wiki o0lkhm3fwjec60x4vlz1tn4vama0bi8 2460115 2460114 2026-09-23T18:38:43Z Stashbot 7414 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors 2460115 wikitext text/x-wiki tmn0cyjjnmzfh1e604jz45daoy1v2ex 2460116 2460115 2026-09-23T18:38:48Z Stashbot 7414 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors 2460116 wikitext text/x-wiki mkkhr9b4jkmvxfrq0rmpmsgf80hr7sr 2460117 2460116 2026-09-23T18:40:24Z Stashbot 7414 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org 2460117 wikitext text/x-wiki 9lfsqk2qs0vlaiv5yy66gnr9n5uxvh9 2460118 2460117 2026-09-23T18:40:34Z Stashbot 7414 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org 2460118 wikitext text/x-wiki snpkj67q87p5oy2p78gy3dtm15oywlu 2460119 2460118 2026-09-23T18:40:34Z Stashbot 7414 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org 2460119 wikitext text/x-wiki nxupl1h06sfjvsnpxe0tcxi8fil9jl3 2460120 2460119 2026-09-23T18:40:40Z Stashbot 7414 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org 2460120 wikitext text/x-wiki 40zzla1c6sihfqhlczv7k12ecu372wl 2460121 2460120 2026-09-23T18:40:41Z Stashbot 7414 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org 2460121 wikitext text/x-wiki nlxaxw7xa42mj2oft5ye0d3udmga3j7 2460122 2460121 2026-09-23T18:42:44Z Stashbot 7414 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org 2460122 wikitext text/x-wiki b5s9wgbyjupx45pwvxe07xsimp1rs1d 2460123 2460122 2026-09-23T18:42:45Z Stashbot 7414 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org 2460123 wikitext text/x-wiki cc2s8wykekuxvmdqig7nm56l81s7pwt 2460124 2460123 2026-09-23T18:42:55Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update 2460124 wikitext text/x-wiki 2aiugta79vpis1w2gx93n546e9heed1 2460125 2460124 2026-09-23T18:43:02Z Stashbot 7414 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org 2460125 wikitext text/x-wiki pduizklpe6bs7g2owwg7zt2ete8920a 2460126 2460125 2026-09-23T18:43:09Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2460126 wikitext text/x-wiki h22gdl9h05bx82gr556wqi7g7wesr45 2460127 2460126 2026-09-23T18:45:28Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2460127 wikitext text/x-wiki dnbd7kmz8pvmcrwvsmyovnic1smoha7 2460128 2460127 2026-09-23T18:50:08Z Stashbot 7414 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org 2460128 wikitext text/x-wiki kw0hcsbd6irci42mtzmay39qijfv6rh 2460130 2460128 2026-09-23T18:58:13Z Stashbot 7414 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors 2460130 wikitext text/x-wiki otj27fi05b5r0ao6wl7a93lbhh00g32 2460131 2460130 2026-09-23T18:58:17Z Stashbot 7414 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors 2460131 wikitext text/x-wiki juvv58wk8dc4h8f5afc69swj0q3ncjk 2460132 2460131 2026-09-23T18:59:42Z Stashbot 7414 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on 2460132 wikitext text/x-wiki kjioxqmqpziky0575nx65op7eurha9x 2460133 2460132 2026-09-23T19:08:08Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update 2460133 wikitext text/x-wiki 5gcfdrv3tazerccqn116o36qnb9u28n 2460134 2460133 2026-09-23T19:10:49Z Stashbot 7414 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.* 2460134 wikitext text/x-wiki 9cq79jzskrrripommgyr9xobjs352b3 2460135 2460134 2026-09-23T19:10:51Z Stashbot 7414 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org 2460135 wikitext text/x-wiki 8e5zulrygfvfu0qmg48sasyl1zkqyhs 2460136 2460135 2026-09-23T19:11:00Z Stashbot 7414 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org 2460136 wikitext text/x-wiki 6cldushm09h460q7zf2o6xm3nfgextb 2460138 2460136 2026-09-23T19:14:50Z Stashbot 7414 taavi@dns1004: START - running authdns-update 2460138 wikitext text/x-wiki 3hmd8zzxhseut67alt1osrraqkfiazw 2460141 2460138 2026-09-23T19:17:16Z Stashbot 7414 taavi@dns1004: END - running authdns-update 2460141 wikitext text/x-wiki pi52ht1isgqdn0o8htz4gfibbniqu0r 2460144 2460141 2026-09-23T19:20:13Z Stashbot 7414 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on 2460144 wikitext text/x-wiki 28t5vvevrtqq3fkcxesmczokl4w00xi 2460147 2460144 2026-09-23T19:32:54Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2460147 wikitext text/x-wiki 7u4y8nf07uuv93jhrqd08m505l750j6 2460148 2460147 2026-09-23T19:33:04Z Stashbot 7414 volans: rebooting arclamp2001.codfw.wmnet 2460148 wikitext text/x-wiki msn2iqwvs7zr9wero9zlwj7mz2368mz 2460149 2460148 2026-09-23T19:33:50Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2460149 wikitext text/x-wiki sxhslox433eyikl4h9ltr58dbpbcgy2 2460150 2460149 2026-09-23T19:34:18Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie 2460150 wikitext text/x-wiki fm8d2duxosf8fvh1sytb66dm82w5v8x 2460156 2460150 2026-09-23T19:52:17Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie 2460156 wikitext text/x-wiki d2ghclv6qf37oyi8f0c0t581ozvadp6 2460161 2460156 2026-09-23T20:04:00Z Stashbot 7414 dzahn@dns1004: START - running authdns-update 2460161 wikitext text/x-wiki c5ll9d9e6zafve6lrd74wc6hem57ka3 2460163 2460161 2026-09-23T20:06:27Z Stashbot 7414 dzahn@dns1004: END - running authdns-update 2460163 wikitext text/x-wiki 1ki6uoim7uqrdbadv3jwir1ambq4rjy 2460173 2460163 2026-09-23T20:27:13Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2460173 wikitext text/x-wiki 78ttk42x40h583kv1r1w7qd5xwbg2bj 2460174 2460173 2026-09-23T20:27:25Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2460174 wikitext text/x-wiki hoob6c9k9bww89lrc1qj1t0596lc7dj 2460178 2460174 2026-09-23T20:30:49Z Stashbot 7414 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress T439010 2460178 wikitext text/x-wiki 8jtt9w1lyx3nl3bnb895ujgn7dfd4ef 2460180 2460178 2026-09-23T20:38:29Z Stashbot 7414 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie 2460180 wikitext text/x-wiki sstfzde73ee0aaavg0p4ujhzf9trw39 2460181 2460180 2026-09-23T20:39:38Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie 2460181 wikitext text/x-wiki 0nfyrcvpwr3etkx70ulwjqwau0zib3s 2460182 2460181 2026-09-23T20:41:33Z Stashbot 7414 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally) 2460182 wikitext text/x-wiki fatppvwonq0loqisxte5js6z79upmm9 2460183 2460182 2026-09-23T20:46:42Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2460183 wikitext text/x-wiki f6t7r5cfbsm9wd7rypk1fvva08yfzu8 2460184 2460183 2026-09-23T20:49:21Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2460184 wikitext text/x-wiki 6wwdxmfkjyrshmqopvtr0hd63zgynwu 2460185 2460184 2026-09-23T20:55:01Z Stashbot 7414 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster 2460185 wikitext text/x-wiki 0vs7txoo79fvgepbnmnwm93s0ojsihp 2460186 2460185 2026-09-23T20:57:27Z Stashbot 7414 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% 2460186 wikitext text/x-wiki 8drjdzumtlotuu5clnukr09q78avfi3 2460187 2460186 2026-09-23T21:07:48Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing 2460187 wikitext text/x-wiki h23g63akhkeia3dovkrersbr5w6o21u 2460188 2460187 2026-09-23T21:07:58Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing 2460188 wikitext text/x-wiki 5tebannxwvmnbfjt3bznvbqk4lzquq0 2460189 2460188 2026-09-23T21:08:08Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing 2460189 wikitext text/x-wiki lzttmd2i7hgbijkbcg47wa2lw74ywrf 2460190 2460189 2026-09-23T21:08:18Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing 2460190 wikitext text/x-wiki lg937b2n0c1rtl1l9jnsefzwn9xnliq 2460191 2460190 2026-09-23T21:18:44Z Stashbot 7414 Emperor: reset-failed then restart ceph-mon on moss-be2003 2460191 wikitext text/x-wiki ebaoom8pb4xh77rz7lhhaxvhems1ka4 2460192 2460191 2026-09-23T21:18:56Z Stashbot 7414 Emperor: ceph mgr fail on apus-be2005 2460192 wikitext text/x-wiki q56pu72clt79uehku89q7zhwefub8u7 2460193 2460192 2026-09-23T21:22:00Z Stashbot 7414 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie 2460193 wikitext text/x-wiki 9q06jp7f3xmftkukuay7e4j7ss0a8c9 2460194 2460193 2026-09-23T21:22:19Z Stashbot 7414 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress T439010 (duration: 51m 29s) 2460194 wikitext text/x-wiki sw132wad8yq2m1yvol0d40odrwm8qq7 2460195 2460194 2026-09-23T21:22:20Z Stashbot 7414 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress T439010 2460195 wikitext text/x-wiki 8auejdgmh625qgmnojpl95cjjviowos 2460198 2460195 2026-09-23T21:49:13Z Stashbot 7414 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie 2460198 wikitext text/x-wiki ab58bnxrp808q2rhrmmbz4eimaxj2dj 2460201 2460198 2026-09-23T22:08:46Z Stashbot 7414 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status 2460201 wikitext text/x-wiki sdvzl8pnihlnaeotfry5gjfnrfcqa1k 2460202 2460201 2026-09-23T22:08:58Z Stashbot 7414 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will) 2460202 wikitext text/x-wiki duuuwfjott52md1u9m5s87wq1nn19va 2460203 2460202 2026-09-23T22:23:36Z Stashbot 7414 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel 2460203 wikitext text/x-wiki bmhxx6e4k5flwj9fk3tedv5vpsq69s2 2460204 2460203 2026-09-23T22:24:01Z Stashbot 7414 ryankemper: [Cirrus] s/expected/expect 2460204 wikitext text/x-wiki k1ed01zov17kmaahx1s85uzwtuodoor 2460205 2460204 2026-09-23T22:28:13Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected (T439010) 2460205 wikitext text/x-wiki 681z6lur5vkd5lw55q5yt2xefym3uy3 2460206 2460205 2026-09-23T22:29:12Z Stashbot 7414 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see 2460206 wikitext text/x-wiki f0l96t3gjv0ppos9p9xcs2vkejp00he 2460207 2460206 2026-09-23T22:44:59Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected (T439010) 2460207 wikitext text/x-wiki fb37s14947mqliwo8luj5l14nw9llcc 2460209 2460207 2026-09-23T22:59:31Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts (T439010) 2460209 wikitext text/x-wiki oxmg7xd44giwol375tm38p25sjbw59f 2460210 2460209 2026-09-23T23:01:05Z Stashbot 7414 ryankemper: [Cirrus] Doing cirrussearch2114 next 2460210 wikitext text/x-wiki 0k32el49ofi92ok9i4r0h17boozvwhr 2460211 2460210 2026-09-23T23:08:49Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts (T439010) 2460211 wikitext text/x-wiki r9sg9uaa9e1r8ph0reoidm3ttkph2cd 2460212 2460211 2026-09-23T23:09:26Z Stashbot 7414 ryankemper: [Cirrus] rolling cirrussearch2086 next 2460212 wikitext text/x-wiki 6o4p2kap9ohftwzuhl8h3lj61krnimb 2460213 2460212 2026-09-23T23:20:58Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts (T439010) 2460213 wikitext text/x-wiki lq4d1vh8v26xf52whssjqtubl3e8oju 2460214 2460213 2026-09-23T23:23:56Z Stashbot 7414 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress T439010 (duration: 121m 40s) 2460214 wikitext text/x-wiki mkn9ka9p8isltb4tue2u479kmg8szgh 2460215 2460214 2026-09-23T23:23:57Z Stashbot 7414 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall. 2460215 wikitext text/x-wiki 6oa7i9jpwt71eh0yfnbyfq54s7n4ckf 2460216 2460215 2026-09-23T23:27:45Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts (T439010) 2460216 wikitext text/x-wiki lb2yxu6t2p7m36w0bw2xtibik8bt5mn 2460217 2460216 2026-09-23T23:34:07Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts (T439010) 2460217 wikitext text/x-wiki iaik8d0hg9nq8uu9kbzxihk9cxf30x8 2460218 2460217 2026-09-23T23:35:01Z Stashbot 7414 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia (T438293) 2460218 wikitext text/x-wiki l84vrk2m6jcrmv7ii3id3b5x56gcvaa 2460219 2460218 2026-09-23T23:38:16Z Stashbot 7414 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () 2460219 wikitext text/x-wiki 8uuyleb40q7r4qusa5b25xlj557ylif 2460220 2460219 2026-09-23T23:38:45Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts (T439010) 2460220 wikitext text/x-wiki fc3wxa28ifrigrfziawpf5budmfun63 2460221 2460220 2026-09-23T23:48:23Z Stashbot 7414 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones 2460221 wikitext text/x-wiki dr4g7592ol8wqw6cvk5z8da8mpusvhs 2460222 2460221 2026-09-23T23:48:43Z Stashbot 7414 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () 2460222 wikitext text/x-wiki nyh1yc5ptx1dv1wdoc87ga4cuq4t6m1 2460223 2460222 2026-09-23T23:49:51Z Stashbot 7414 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () 2460223 wikitext text/x-wiki a3lz4tygpukpdbixtudbghy8hwmacdc 2460224 2460223 2026-09-23T23:54:12Z Stashbot 7414 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () 2460224 wikitext text/x-wiki 22gztj1n3izvmtpqx9yxqi7r0q175yt 2460225 2460224 2026-09-23T23:56:48Z Stashbot 7414 brett: Switching acme-chief primary from codfw to eqiad - T439010 2460225 wikitext text/x-wiki 2umjg5l6rr3kh0ltcao7sgoi9b355n0 2460227 2460225 2026-09-23T23:58:54Z Stashbot 7414 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie 2460227 wikitext text/x-wiki q2hbxx2yd20h23bludmkhmgy6os1m36 2460228 2460227 2026-09-24T00:00:19Z Stashbot 7414 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () 2460228 wikitext text/x-wiki 404nf79zj3m8sg8l600wa469pzq9vyj 2460230 2460228 2026-09-24T00:05:40Z Stashbot 7414 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () 2460230 wikitext text/x-wiki 7a8txsn7q9g1pwzadrvz2kfh3ov8hj1 2460231 2460230 2026-09-24T00:10:01Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.* 2460231 wikitext text/x-wiki 6f3859x0qhsqor2o39nve2urtpuv0ck 2460232 2460231 2026-09-24T00:35:10Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state (T439010) 2460232 wikitext text/x-wiki on4cdou9b9p0eohcnq82fyv0sbldy5b 2460233 2460232 2026-09-24T00:38:17Z Stashbot 7414 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling) 2460233 wikitext text/x-wiki d3eviqtb80c53330carzx66z0ywvfra 2460234 2460233 2026-09-24T00:40:09Z Stashbot 7414 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though 2460234 wikitext text/x-wiki oqmet7eajh01f3k7g9psj6bizx7utlg 2460235 2460234 2026-09-24T01:03:59Z Stashbot 7414 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts 2460235 wikitext text/x-wiki s2u2k4x4in0sjtu6wemfnza3xx66geb 2460236 2460235 2026-09-24T01:04:28Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected (T439010) 2460236 wikitext text/x-wiki fcz2dz4o7pya5qkxcmldgkfgrog42ee 2460237 2460236 2026-09-24T01:10:19Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected (T439010) 2460237 wikitext text/x-wiki pjamoeawrapxtdp8evbi9kmxwigmqro 2460238 2460237 2026-09-24T01:11:53Z Stashbot 7414 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet 2460238 wikitext text/x-wiki ehibo6zrqr8ecsh1vozvpkoq2robmyi 2460239 2460238 2026-09-24T01:14:31Z Stashbot 7414 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected (T439010) 2460239 wikitext text/x-wiki ev572yuh4bozicipjwz7m8l7wu5bkh1 2460240 2460239 2026-09-24T01:16:12Z Stashbot 7414 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here 2460240 wikitext text/x-wiki 5adm1z824w63qewnq2xfougica9x5ma 2460241 2460240 2026-09-24T01:19:14Z Stashbot 7414 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green 2460241 wikitext text/x-wiki pvyamtp6jmcbohen6tv4c43f0766b26 2460242 2460241 2026-09-24T01:21:49Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - T439010 2460242 wikitext text/x-wiki odhvspb78izp9dpkqb38xdfli2kfyeq 2460252 2460242 2026-09-24T05:07:35Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - T439010 2460252 wikitext text/x-wiki twc6m5gxozct5semfqzfzsdrbb0xw89 2460254 2460252 2026-09-24T06:06:50Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw 2460254 wikitext text/x-wiki d6x92qklhlot9ou5hxegsh5fvnc08fo 2460255 2460254 2026-09-24T06:07:46Z Stashbot 7414 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw 2460255 wikitext text/x-wiki 70wuhtjn9zd2swmymhw1p30bpyo8xgb 2460256 2460255 2026-09-24T06:38:30Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply 2460256 wikitext text/x-wiki 912fciwj3kkth4wszafk3zbfaohq8z4 2460257 2460256 2026-09-24T06:39:35Z Stashbot 7414 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production 2460257 wikitext text/x-wiki hplzi8y9svhaxo07ell1vowt4shxfkz 2460258 2460257 2026-09-24T06:41:31Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply 2460258 wikitext text/x-wiki 7el5gr8evrcgs039bbee5bm8mh1dgvh 2460259 2460258 2026-09-24T06:41:51Z Stashbot 7414 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production (T437213) 2460259 wikitext text/x-wiki lhvnwcu8ng4jdelx6cim0sa9p79w3lx 2460260 2460259 2026-09-24T06:51:15Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade 2460260 wikitext text/x-wiki dlauylfpm008rar47sg81jevpll7izh 2460261 2460260 2026-09-24T07:02:18Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade 2460261 wikitext text/x-wiki 3uid14u04pr6nrgqnku4clhzndaupua 2460262 2460261 2026-09-24T07:06:05Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade 2460262 wikitext text/x-wiki ed0u1v9x90nqv4hk44xby98g20a8e5a 2460263 2460262 2026-09-24T07:16:57Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade 2460263 wikitext text/x-wiki h6ztroeutcxswn9sm5awyv1ypgp95it 2460264 2460263 2026-09-24T07:19:40Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade 2460264 wikitext text/x-wiki hr4ktq8co5rwnylkmo2tg8koucw93du 2460266 2460264 2026-09-24T07:48:12Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade 2460266 wikitext text/x-wiki 43k2d9xt472bwr8lecz3ip45mx9dooo 2460267 2460266 2026-09-24T07:52:15Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, T437984] 2460267 wikitext text/x-wiki 75hs0i9n48j1ft3p8o22as9p51a1hea 2460268 2460267 2026-09-24T07:52:19Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, T437984] 2460268 wikitext text/x-wiki td0ydsharkivy0uspf4n0f8pplchsf1 2460269 2460268 2026-09-24T07:53:50Z Stashbot 7414 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade 2460269 wikitext text/x-wiki rahgpziwjxg23p12tdfn85o75w840pg 2460270 2460269 2026-09-24T08:02:53Z Stashbot 7414 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade 2460270 wikitext text/x-wiki n5uouev2fhuewrhhbe0n0qi59mgbmkc 2460272 2460270 2026-09-24T08:06:59Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner 2460272 wikitext text/x-wiki bbph4fcep5eejcexgqxcmsf16x6q803 2460273 2460272 2026-09-24T08:13:08Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12 2460273 wikitext text/x-wiki mzm7h7deq1dtjniozsjcuv9mqbu2lcq 2460274 2460273 2026-09-24T08:20:32Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12 2460274 wikitext text/x-wiki ax38gmhxxjbvu6he5xkz4ck0zhp9pkn 2460275 2460274 2026-09-24T08:22:10Z Stashbot 7414 XioNoX: asw1-b12-drmrs> request system reboot - T437984 2460275 wikitext text/x-wiki c9n88dfpr3cievy7t7u9rzsar7124h9 2460277 2460275 2026-09-24T08:31:00Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1015.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) 2460277 wikitext text/x-wiki qubxg63mugp07txc7zjrz6nma4hqdm3 2460278 2460277 2026-09-24T08:31:04Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet 2460278 wikitext text/x-wiki cly1tpqfeghkh5k2pca8dvdi7qkmzyr 2460279 2460278 2026-09-24T08:31:05Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet 2460279 wikitext text/x-wiki 6nwe97rkagkhb3jyqi7p4w6ikxnmi3n 2460283 2460279 2026-09-24T08:37:04Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet 2460283 wikitext text/x-wiki 0tllrezuolx49rnktqfk4x5t46xvjvy 2460284 2460283 2026-09-24T08:37:54Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet 2460284 wikitext text/x-wiki ffb8v0xb4ym4una0hc54f1mwng3u1a4 2460285 2460284 2026-09-24T08:37:55Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1015.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) 2460285 wikitext text/x-wiki ndz3qweb2p6j1s5crz6r9xhzqnyc5g1 2460287 2460285 2026-09-24T08:45:01Z Stashbot 7414 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade 2460287 wikitext text/x-wiki ml4uw0wnmj82wld4tmold215fdt640r 2460289 2460287 2026-09-24T08:45:25Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt 2460289 wikitext text/x-wiki l5lofztpg6f0bjgeck0h390cbbf7rrh 2460290 2460289 2026-09-24T08:45:26Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt 2460290 wikitext text/x-wiki l0r7mzm3zs4rthbsh1g58pwuua7w4fj 2460291 2460290 2026-09-24T08:46:35Z Stashbot 7414 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866 2460291 wikitext text/x-wiki gwnf80ac67ism889r61eh9emz7qyr4b 2460292 2460291 2026-09-24T08:46:40Z Stashbot 7414 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade 2460292 wikitext text/x-wiki eja5dz6mmujk6dppu934lxm659oe9tp 2460293 2460292 2026-09-24T08:47:55Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12 2460293 wikitext text/x-wiki a55o6cgfkyjb297kbdoh400vji33z26 2460294 2460293 2026-09-24T08:49:22Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12 2460294 wikitext text/x-wiki opqllyycjrhro16ucmdvkau4h0k278h 2460296 2460294 2026-09-24T08:54:40Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2460296 wikitext text/x-wiki 0wb9pp7aos77cec5u3lb9050yy0osce 2460299 2460296 2026-09-24T08:55:48Z Stashbot 7414 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision 2460299 wikitext text/x-wiki mfv6x3mpuvfe9kc0hmkcsutxd7mkws4 2460300 2460299 2026-09-24T08:56:06Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13 2460300 wikitext text/x-wiki 3kzsatqr58dexsg4qba3qi5bjr676th 2460303 2460300 2026-09-24T08:58:54Z Stashbot 7414 moritzm: installing node-lodash security updates 2460303 wikitext text/x-wiki 7c7enfjsd9nzudfvf0l74aj1ajqjhyp 2460305 2460303 2026-09-24T08:59:46Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2460305 wikitext text/x-wiki to4xm1iuw3kbmn5djxggvckxwb29s1i 2460306 2460305 2026-09-24T09:00:14Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13 2460306 wikitext text/x-wiki j5joxefwg0crhwgfuwj4ik5gy5f34a7 2460307 2460306 2026-09-24T09:00:42Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner 2460307 wikitext text/x-wiki jqbc9wl1fhnhkxpt1i6r389ki92jzpx 2460308 2460307 2026-09-24T09:01:22Z Stashbot 7414 XioNoX: asw1-b13-drmrs> request system reboot - T437984 2460308 wikitext text/x-wiki r9erbqyjeies1j65c3zvlkizqeebzpg 2460311 2460308 2026-09-24T09:04:23Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet 2460311 wikitext text/x-wiki lt760mgkn5n4n3wus7jwxoc7l8t454p 2460312 2460311 2026-09-24T09:05:45Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828) 2460312 wikitext text/x-wiki 76hd8tni61df9n769gtq80lx2cwdfka 2460314 2460312 2026-09-24T09:10:31Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet 2460314 wikitext text/x-wiki o2m4r9cidabii6cbm3k0u0u6k5n2n2h 2460316 2460314 2026-09-24T09:21:29Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2460316 wikitext text/x-wiki 2bawf785jzx43s5tcygocp2wypyh2dg 2460317 2460316 2026-09-24T09:21:58Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2460317 wikitext text/x-wiki tdvsf8st4os1w6jpkgpmpru7y3qg67e 2460318 2460317 2026-09-24T09:22:12Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2460318 wikitext text/x-wiki iey19nfk6o6cn6ckp1htbkq7eb1e6qa 2460319 2460318 2026-09-24T09:22:38Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2460319 wikitext text/x-wiki 28vpm95a7urcfbioxp9wukyeknxl0tq 2460320 2460319 2026-09-24T09:22:50Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2460320 wikitext text/x-wiki o87bxek5lpu5gkvi9g5ksjitzylpnf0 2460321 2460320 2026-09-24T09:23:12Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2460321 wikitext text/x-wiki 8e0wips7dixux0lgv355xvutg1v806g 2460322 2460321 2026-09-24T09:25:28Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13 2460322 wikitext text/x-wiki hplqm4pmgje57yaiwpghymrn1flsj7k 2460323 2460322 2026-09-24T09:26:28Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13 2460323 wikitext text/x-wiki ji9jlamr00io9tb95keb7moks7s31al 2460324 2460323 2026-09-24T09:27:13Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, T437984] 2460324 wikitext text/x-wiki 1aq33it9nayezt7vxwrwujzygbzqei3 2460325 2460324 2026-09-24T09:27:16Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, T437984] 2460325 wikitext text/x-wiki 9xfwx6y8v6dfudevpbxjs3w9mzr2jte 2460327 2460325 2026-09-24T09:40:31Z Stashbot 7414 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet 2460327 wikitext text/x-wiki f1hajfw2ekyc4d266mlqnsfpw3pft8o 2460328 2460327 2026-09-24T09:46:55Z Stashbot 7414 moritzm: installing bind9 security updates (client-side tools/libs only) 2460328 wikitext text/x-wiki m8b3n8rh485ptj2s4phquvyb8i4m336 2460332 2460328 2026-09-24T10:03:03Z Stashbot 7414 moritzm: installing apr-util security updates 2460332 wikitext text/x-wiki 9c5dpkynvlixbd01pfjknfi5d8swgpf 2460333 2460332 2026-09-24T10:08:16Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828) 2460333 wikitext text/x-wiki q56l8r6pau8osv0sl91nl5t4h1dc6vv 2460334 2460333 2026-09-24T10:11:57Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828) 2460334 wikitext text/x-wiki 3v6jdyjix36i4g61vmtqbeo5k7eqw6x 2460335 2460334 2026-09-24T10:12:05Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply 2460335 wikitext text/x-wiki 0dkzxkez6rxmg7bsq0uzjvn76crec1i 2460336 2460335 2026-09-24T10:17:25Z Stashbot 7414 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw 2460336 wikitext text/x-wiki isra6rtljqvyr9rb5piwdcsdigvesiw 2460337 2460336 2026-09-24T10:17:33Z Stashbot 7414 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s) 2460337 wikitext text/x-wiki l4tggwzasn0hy9yczc9w4yqpuhyvgb1 2460339 2460337 2026-09-24T10:21:30Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] 2460339 wikitext text/x-wiki bn7bwc331aieqljsp21xjw8uttn2fyn 2460341 2460339 2026-09-24T10:22:15Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply 2460341 wikitext text/x-wiki rh9g6uur8l442rjyxbrqaj5qre4db29 2460342 2460341 2026-09-24T10:23:23Z Stashbot 7414 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision 2460342 wikitext text/x-wiki rpjv9uyk6kyeks7gayjw33vavrgb2rr 2460344 2460342 2026-09-24T10:25:44Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2460344 wikitext text/x-wiki 6ln1b80ce6o4qtyknyfle3658za37f7 2460345 2460344 2026-09-24T10:26:38Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2460345 wikitext text/x-wiki 1addnh2zc85ta39hdl5q2ptadulqg95 2460346 2460345 2026-09-24T10:26:39Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2460346 wikitext text/x-wiki j6b8t1fcq10rb2yjp4e3uqittzj822p 2460347 2460346 2026-09-24T10:29:02Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2460347 wikitext text/x-wiki aq5t3wd9g3uecnnah61qa6g4jfnlvwh 2460351 2460347 2026-09-24T10:32:04Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s) 2460351 wikitext text/x-wiki qc6o7jvjhlv9rzrwixbk9b844i5jwwh 2460352 2460351 2026-09-24T10:34:20Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply 2460352 wikitext text/x-wiki 2h8el0mwckcr9t0ry6stj2x2haws65d 2460353 2460352 2026-09-24T10:35:25Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] 2460353 wikitext text/x-wiki 1uj6is2k68cc4ecry74la3zcq23td8a 2460354 2460353 2026-09-24T10:36:26Z Stashbot 7414 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision 2460354 wikitext text/x-wiki qeozhu7srhwuw9533zh7ljtrjcdhpmf 2460355 2460354 2026-09-24T10:39:39Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2460355 wikitext text/x-wiki 7q3kwozp6zmbjsioym5fyguqt4t4qp5 2460356 2460355 2026-09-24T10:40:14Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2460356 wikitext text/x-wiki eur0vyxyyuafggc0xm98vmcfz366zlm 2460359 2460356 2026-09-24T10:42:36Z Stashbot 7414 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828) 2460359 wikitext text/x-wiki nlv4zxgduqzuuqfg076dcx7um4jof5v 2460360 2460359 2026-09-24T10:44:35Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply 2460360 wikitext text/x-wiki ctismt2e7d5hq71mbl032fvl60us4yb 2460361 2460360 2026-09-24T10:45:35Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s) 2460361 wikitext text/x-wiki 0a2wke27mc35ybkeesj8vnrk9acuv63 2460362 2460361 2026-09-24T10:50:35Z Stashbot 7414 hnowlan: deleting stuck mw-web pods in eqiad 2460362 wikitext text/x-wiki tsr62u1uwi07vb0nhybs0580m1p2g3e 2460366 2460362 2026-09-24T11:04:10Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828) 2460366 wikitext text/x-wiki pn538wc9mpbqn7g9dcpyqsc5i7aome3 2460367 2460366 2026-09-24T11:13:36Z Stashbot 7414 hnowlan: restarted restbase on restbase2029 2460367 wikitext text/x-wiki p7a4zpy80gj5afkwo7xummw78376mzi 2460368 2460367 2026-09-24T11:21:08Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828) 2460368 wikitext text/x-wiki nzg3llqmg2elw1vakh2e2wmtfrs1aji 498 9925 2460197 2459668 2026-09-23T21:46:57Z Stashbot 7414 Southparkfan: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344336 to Puppetserver - T435393 2460197 wikitext text/x-wiki 4ol2jqd02pk01tyni366ke09vufwtwh 0 17290 2459827 2459667 2026-09-23T12:56:31Z Stashbot 7414 awight: manually run puppet agent 2459827 wikitext text/x-wiki 1rfon2339rl66rfva1vu1hh8gtnz3p7 2459830 2459827 2026-09-23T12:58:48Z Stashbot 7414 awight: add seanleong-wmde to deployment-prep 2459830 wikitext text/x-wiki 82ykajvhskpo5wlarnroerxju1ni77i 2460151 2459830 2026-09-23T19:36:03Z Stashbot 7414 hashar: gerrit: force online reindexing of the change index over ssh hitting gerrit1003 (switch over). `gerrit index start changes --force` 2460151 wikitext text/x-wiki i6yyfco0igvh8twoka3hjvmapg6gbki 2460152 2460151 2026-09-23T19:38:05Z Stashbot 7414 hashar: gerrit: started reindexing for accounts, group and projects indices 2460152 wikitext text/x-wiki g2qke2t20dxhejxf2nmo4tveqlydw7u 2460165 2460152 2026-09-23T20:13:31Z Stashbot 7414 hashar: gerrit: reindexing has been completed 2460165 wikitext text/x-wiki 84hwucn787znwy3i2xvwf8bvmxl2ddu 2460172 2460165 2026-09-23T20:26:54Z Stashbot 7414 dancy: Upgrading istio to 1.30.5 on gitlab-cloud-runners (staging) (T439035) 2460172 wikitext text/x-wiki 1t929ef2ezmdhtkyvu0m01gbwa7lgiq 2460179 2460172 2026-09-23T20:37:26Z Stashbot 7414 hashar: gerrit: deleted https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1344362 duplicate Change-Id I67dd7b3c879049a1c98586c365c297eac150b963 of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1329697 2460179 wikitext text/x-wiki a2g0inexnajqbou48y6fhd61nk2uoye 2460196 2460179 2026-09-23T21:46:54Z Stashbot 7414 Southparkfan: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344336 to Puppetserver - T435393 2460196 wikitext text/x-wiki hu2aecy5cs715ue2b5yzkyzvxdvxews 2460200 2460196 2026-09-23T22:06:54Z Stashbot 7414 Southparkfan: decommission Beta Cluster PHP 8.3 hosts deployment-mediawiki13, deployment-mediawiki14, deployment-jobrunner05, deployment-mwmaint03 - T435393 2460200 wikitext text/x-wiki osaon3gtald7v3yzt00i3ee1h2deiiz 0 20225 2460040 2451497 2026-09-23T15:50:40Z AQuhen (WMF) 29318 Remove thin deployment which is not needed anymore 2460040 wikitext text/x-wiki c8vfgkcpzykyd3m62yo1wjhlljvyu9q 0 24248 2460063 2456346 2026-09-23T16:47:40Z ADenisse-WMF 31462 /* Version upgrade */ 2460063 wikitext text/x-wiki ndhypejdkb9whbz41id8xg1qog31623 2460091 2460063 2026-09-23T17:53:02Z ADenisse-WMF 31462 /* Version upgrade */ 2460091 wikitext text/x-wiki fyle5j0yxlhds2rbj958frjf51uyopz 498 30942 2460010 2459738 2026-09-23T15:28:21Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2460010 wikitext text/x-wiki f037scxsxrk1a6obzqcfcqak6wtzri3 2460011 2460010 2026-09-23T15:28:49Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2460011 wikitext text/x-wiki loeid50tmb182ftumc2he24e292cqc8 2460139 2460011 2026-09-23T19:17:06Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services 2460139 wikitext text/x-wiki 7o5xsdp22eu517atyujd4qhhixygz8w 2460140 2460139 2026-09-23T19:17:11Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services 2460140 wikitext text/x-wiki pkfvbhizchxpnu14j16780j7bqeszr5 2460143 2460140 2026-09-23T19:18:31Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services 2460143 wikitext text/x-wiki sgri3tonh6fzvsx98enuohled44eii7 2460145 2460143 2026-09-23T19:21:52Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services 2460145 wikitext text/x-wiki nz4shbngchtc9n039sn1z8a740o37yb 2460276 2460145 2026-09-24T08:28:26Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1078' (T435537) 2460276 wikitext text/x-wiki 1ess4wea7570skw8ha5nzemvmmevb19 2460295 2460276 2026-09-24T08:51:25Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1078' (T435537) 2460295 wikitext text/x-wiki ozdhtit9ijj03d08dqm6mxrf6g84u3p 2460329 2460295 2026-09-24T09:54:27Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) 2460329 wikitext text/x-wiki jkv6qboxr2js74wra8x3mcwlemt78zu 2460330 2460329 2026-09-24T09:54:32Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) 2460330 wikitext text/x-wiki tt3i9th2rwcrbwx13fe0hqkwlobeolc 2460331 2460330 2026-09-24T09:54:58Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1079' (T435537) 2460331 wikitext text/x-wiki tt5z3t5tcd6l89so48fbll7v7xr2xcm 2460340 2460331 2026-09-24T10:21:55Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1079' (T435537) 2460340 wikitext text/x-wiki pq8cuzqdghv44bmxkfrcq11xtszp9mr 2460348 2460340 2026-09-24T10:30:11Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) 2460348 wikitext text/x-wiki 34epwzcp0u2r6146i1krzn5fgledd68 2460349 2460348 2026-09-24T10:30:19Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) 2460349 wikitext text/x-wiki adbsbeurq4bvywirk7rpxs42tx78v8n 2460350 2460349 2026-09-24T10:30:36Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1080' (T435537) 2460350 wikitext text/x-wiki onu0k6ocq4w5t0xhbju6qykyy7jwppn 2460365 2460350 2026-09-24T11:01:39Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1080' (T435537) 2460365 wikitext text/x-wiki is8mp06e6jngzhpd8jh564s7i2c7zms 0 443307 2460146 2330524 2026-09-23T19:26:09Z BCornwall-WMF 31521 /* Production environment */ Split out the instructions on switching primary instances 2460146 wikitext text/x-wiki 2x4nf315vjhrmrj7brzbkwrk6dntfrc 2460153 2460146 2026-09-23T19:45:15Z BCornwall-WMF 31521 /* Production environment */ ce: code block formatting 2460153 wikitext text/x-wiki gqolcrxxudbnrz089qelhpnjmbd8jg1 2460155 2460153 2026-09-23T19:48:51Z BCornwall-WMF 31521 /* Production environment */ reorg sections: rm "production environment", create "Architecture" and move cert info up to it, create "Management" section 2460155 wikitext text/x-wiki hcbckm5e8k01g4y6armke2ij6z6kt5y 2460157 2460155 2026-09-23T19:55:16Z BCornwall-WMF 31521 gitweb template 2460157 wikitext text/x-wiki rmf3o88po3fde7l6wutkootf9fn0uxr 2460158 2460157 2026-09-23T19:57:58Z BCornwall-WMF 31521 /* See also */ add historical bug task 2460158 wikitext text/x-wiki 5jjmlram7z65vx7znfeqrhgqb43ni44 2460159 2460158 2026-09-23T20:00:17Z BCornwall-WMF 31521 Put history below intro blurb 2460159 wikitext text/x-wiki cwl2dpz2vs9e0x738mdyvdn3t3rpqyo 2460160 2460159 2026-09-23T20:02:55Z BCornwall-WMF 31521 import history from [[Acme-chief/Cloud_VPS_setup]] 2460160 wikitext text/x-wiki 60p0lri7tgqfceusx4vi6xuejr0oskq 2460162 2460160 2026-09-23T20:04:50Z BCornwall-WMF 31521 /* See also */ clarify phab ticket is about cloud vps 2460162 wikitext text/x-wiki nhmbp9qehxogcju7crcuc10ufe65c6a 2460164 2460162 2026-09-23T20:08:14Z BCornwall-WMF 31521 ce: remove extra newline 2460164 wikitext text/x-wiki hfl6yym8a9ni1vags30fv54nzymo0m3 2460166 2460164 2026-09-23T20:23:30Z BCornwall-WMF 31521 Merge in useful contents of [[Acme-chief/Cloud VPS]] 2460166 wikitext text/x-wiki 2big7fzcrxactbzxenpzatl4lk7884m 2460169 2460166 2026-09-23T20:25:20Z BCornwall-WMF 31521 /* See also */ Remove dead link to vps setup 2460169 wikitext text/x-wiki t3nikhwrs84eq1kh6nnr8asehpww6nj 2460170 2460169 2026-09-23T20:26:26Z BCornwall-WMF 31521 Move [[#Monitoring]] down into [[#Management]] 2460170 wikitext text/x-wiki n7csteek46m45s6t4a08tpanae8v0ns 2460171 2460170 2026-09-23T20:26:39Z BCornwall-WMF 31521 /* Management */ ce: remove extra newline 2460171 wikitext text/x-wiki 768rwmyey6mgwoe4gha0k7jhsso4xnu 2460175 2460171 2026-09-23T20:28:54Z BCornwall-WMF 31521 Move monitoring section to [[#Troubleshooting]] 2460175 wikitext text/x-wiki dstrj6f45tjxxub919n3cmizql11ns1 2460176 2460175 2026-09-23T20:29:50Z BCornwall-WMF 31521 /* Troubleshooting */ Link to google pki status page 2460176 wikitext text/x-wiki am0bm3oatwb3819vj31soqd5zudkvi5 498 443948 2460107 2439779 2026-09-23T18:27:47Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed f1b263f216 (upgrade dependencies) and 32a27cf571 (fix README) 2460107 wikitext text/x-wiki kaaeveqon4on0140fw1pmudjg5296zw 2460129 2460107 2026-09-23T18:53:51Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 77c45c428a (Python 3.14) 2460129 wikitext text/x-wiki q1vtmdlquymzvdukhu5ojt0kblhfvpj 2460137 2460129 2026-09-23T19:11:04Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed fa3fbc988f (Toolforge Components Service, push-to-deploy) 2460137 wikitext text/x-wiki s0f7qbb9ajaie01eui4rt5hksx8vvoz 0 443981 2460048 2445244 2026-09-23T16:00:18Z Elilopes 37903 wikipedia 2460048 wikitext text/x-wiki lgbarityefhl9gtp2l3rkhrgmrow81k 0 444398 2460208 2454910 2026-09-23T22:50:54Z Krinkle 44 link to 2460208 wikitext text/x-wiki 3hnukz5kysgwwuhhjlgfcxdj8iowx6a 0 444533 2460014 2423924 2026-09-23T15:34:04Z Elilopes 37903 Generate key 2460014 wikitext text/x-wiki spffeirkbzkk9cz984n5hi6m1at2m4l 110 446078 2460167 2162397 2026-09-23T20:24:10Z BCornwall-WMF 31521 BCornwall-WMF moved page [[Acme-chief/Cloud VPS setup]] to [[Obsolete:Acme-chief/Cloud VPS setup]] without leaving a redirect: Merged in to [[Acme-chief]] 2162397 wikitext text/x-wiki 3zm9nedsm5e2ycab5la8s9bpt8tjjsf 0 447760 2460326 2427846 2026-09-24T09:38:03Z Hashar 32 some proofreading 2460326 wikitext text/x-wiki 6h6dvhosq7ph3x97lmfkk1dnzbgylog 0 447761 2460154 2434669 2026-09-23T19:45:55Z LSobanski (WMF) 18783 /* Failover (in case primary host is unavailable) */ 2460154 wikitext text/x-wiki sry6j6mfe589i0h4i70wud0k00zesim 2460177 2460154 2026-09-23T20:30:19Z LSobanski (WMF) 18783 /* Failover (in case primary host is unavailable) */ 2460177 wikitext text/x-wiki e2rbtw0s29yv0ahqtwdf2emq1pcore1 2460310 2460177 2026-09-24T09:02:56Z Taavi-WMF 41365 /* Failover (in case primary host is unavailable) */ minor clarifications 2460310 wikitext text/x-wiki euzlo7galy4w28air4bprk9selqaovm 0 449160 2460229 2459676 2026-09-24T00:02:36Z Dexbot 30554 Bot: Updating the report 2460229 wikitext text/x-wiki e7rwjmd3oxu8bvpv7ing0htexmf3x8o 0 449344 2460288 2457097 2026-09-24T08:45:03Z VGutiérrez (WMF) 11925 /* Expression grammar and inline patterns */ fix typo 2460288 wikitext text/x-wiki 40fxmkp0ul9kf9iafkr0f1idg8f2sy5 116 453906 2460089 2458695 2026-09-23T17:48:14Z Gitlabaccountapprovalbot 37332 mdh was rejected. 2460089 wikitext text/x-wiki phxju75z1icdf7ya81ocysczedmnqos 0 458585 2459797 2459795 2026-09-23T12:05:00Z BRouberol-WMF 39141 /* Replacing Failed Disks */ 2459797 wikitext text/x-wiki hk3xd2x005d7wkidact6kcwesoqz5sc 2459798 2459797 2026-09-23T12:05:31Z BRouberol-WMF 39141 /* Identifying the physical slot associated with an OSD or device */ 2459798 wikitext text/x-wiki bale9oixy2zzzsuyrm043qki06l0qjv 2459799 2459798 2026-09-23T12:06:19Z BRouberol-WMF 39141 /* Replacing Failed Disks */ 2459799 wikitext text/x-wiki ceabpp259ksu6a1f94g67bl0cu5clcb 2459819 2459799 2026-09-23T12:43:50Z BRouberol-WMF 39141 /* Replacing Failed Disks */ 2459819 wikitext text/x-wiki mfbjb7kj9fhfpmsnzm8lqwg2csuybzr 498 458783 2460199 2452919 2026-09-23T22:02:08Z Southparkfan 1843 2460199 wikitext text/x-wiki e3z17gp1nxwilkhf3nzuo8o6psac08g 0 459699 2460000 2459589 2026-09-23T15:19:01Z Stashbot 7414 ejegg: fundraising civicrm upgraded from d575ad3e to 4fc5d217 2460000 wikitext text/x-wiki okepsnpq18yoldjazpbeat0pduq6g8p 2460007 2460000 2026-09-23T15:21:54Z Stashbot 7414 ejegg: payments-wiki upgraded from 48ac74d9 to 375adf37 2460007 wikitext text/x-wiki 80vfybpgqn8v5m4gcmw62092hwioux0 2460226 2460007 2026-09-23T23:58:18Z Stashbot 7414 eileen: civicrm upgraded from 4fc5d217 to 11d9e123 2460226 wikitext text/x-wiki 9dt4la6cu22q4yzrbionmhjo3qvtha9 2460251 2460226 2026-09-24T04:18:08Z Stashbot 7414 eileen: civicrm upgraded from 11d9e123 to 879c1771 2460251 wikitext text/x-wiki 08e2us9d4spov4e6oq3gkbrjialu7r3 2460253 2460251 2026-09-24T05:22:05Z Stashbot 7414 eileen: civicrm upgraded from 879c1771 to 7545e2c4 2460253 wikitext text/x-wiki c8dxtfv86dtofihwuhxdixcqa7pa5x1 116 460070 2459869 2452775 2026-09-23T14:17:38Z EtherpadBackupBot 54504 wikicon26-wiki_loves_broadcast archived by Tkarcher 2459869 wikitext text/x-wiki 7sabclivgpa2uwhebim7a67raqbi2mu 2459872 2459869 2026-09-23T14:18:34Z EtherpadBackupBot 54504 wikicon26-redaktionstreffen_altertum archived by Tkarcher 2459872 wikitext text/x-wiki pgcr8n0p60eksm59gjj3li999qy517n 2459874 2459872 2026-09-23T14:20:01Z EtherpadBackupBot 54504 wikicon26-the_wikipedia_library archived by Tkarcher 2459874 wikitext text/x-wiki 38xxmro1sqmxpdl3zf6n7btcump19jn 2459875 2459874 2026-09-23T14:20:55Z EtherpadBackupBot 54504 wikicon26-wikiprojekt_nahostkonflikt archived by Tkarcher 2459875 wikitext text/x-wiki d2f8tc83twx6jno592vp258618m9opl 2459876 2459875 2026-09-23T14:21:11Z EtherpadBackupBot 54504 wikicon26-schwere_zeiten_fuer_informationen archived by Tkarcher 2459876 wikitext text/x-wiki k7ehtk3b1o6jnlihnsyoy65gheq5cor 2459877 2459876 2026-09-23T14:21:27Z EtherpadBackupBot 54504 wikicon26-quellen-wikidata archived by Tkarcher 2459877 wikitext text/x-wiki 2gbfq5b0o07tbtm8k2fu2przqz148d7 2459878 2459877 2026-09-23T14:21:45Z EtherpadBackupBot 54504 wikicon26-regionalwikis archived by Tkarcher 2459878 wikitext text/x-wiki mtsfju723cxfo94okspfcrf5tiwhh9w 2459882 2459878 2026-09-23T14:23:54Z EtherpadBackupBot 54504 wikicon26-zu_jung_fuer_die_wikipedia archived by Tkarcher 2459882 wikitext text/x-wiki pytz9frc7b9zt5o1uidb3syqnsjskew 2459884 2459882 2026-09-23T14:24:12Z EtherpadBackupBot 54504 wikicon26-unterstuetzung_durch_wikimedia archived by Tkarcher 2459884 wikitext text/x-wiki 6yugyqn7apx72u2hmzz57gm767i4p7g 2459885 2459884 2026-09-23T14:24:44Z EtherpadBackupBot 54504 wikicon26-graphit archived by Tkarcher 2459885 wikitext text/x-wiki 9nhi1uj1v9tw3rzl5qfigw1zysxprtg 2459886 2459885 2026-09-23T14:24:59Z EtherpadBackupBot 54504 wikicon26-wiki_loves_folklore archived by Tkarcher 2459886 wikitext text/x-wiki thl9y4x3rl3btpuudlfvubawfb1ug81 2459887 2459886 2026-09-23T14:25:13Z EtherpadBackupBot 54504 wikicon26-denkmalschutz archived by Tkarcher 2459887 wikitext text/x-wiki brqkmpialtszychqs40kc8vudsq08ck 2459888 2459887 2026-09-23T14:25:29Z EtherpadBackupBot 54504 wikicon26-admins_im_spannungsverhaeltnis archived by Tkarcher 2459888 wikitext text/x-wiki 3rfknqshmf6ummxqv7xmg93ykwnr6w2 2459889 2459888 2026-09-23T14:25:47Z EtherpadBackupBot 54504 wikicon26-technische_wuensche_wikimedia_commons archived by Tkarcher 2459889 wikitext text/x-wiki ipyb66udmvmsuggxt8i1qr8y7p358ep 2459890 2459889 2026-09-23T14:26:03Z EtherpadBackupBot 54504 wikicon26-lesende_verstehen archived by Tkarcher 2459890 wikitext text/x-wiki cfjznowe6c1bj8bvhe6hr9yv70o8mi9 2459893 2459890 2026-09-23T14:26:21Z EtherpadBackupBot 54504 wikicon26-das_unsichtbare_sichtbar_machen archived by Tkarcher 2459893 wikitext text/x-wiki bzjclvomu1m1mluqb9anju73lzt96aa 2459894 2459893 2026-09-23T14:26:39Z EtherpadBackupBot 54504 wikicon26-aufzaehlungen_kuerzen archived by Tkarcher 2459894 wikitext text/x-wiki 04qk1pnzx371ldsb5jpdytvba2jmpnb 2459895 2459894 2026-09-23T14:26:56Z EtherpadBackupBot 54504 wikicon26-administrieren archived by Tkarcher 2459895 wikitext text/x-wiki fli9qyszrp69vzij0akpgjol5nzz46d 2459930 2459895 2026-09-23T14:51:50Z EtherpadBackupBot 54504 wikicon26-mit_der_zeit_gehen archived by Tkarcher 2459930 wikitext text/x-wiki 4e62v1cwwv7kod902iuk6484qgsky64 2459931 2459930 2026-09-23T14:52:11Z EtherpadBackupBot 54504 wikicon26-aufzaehlungen_kuerzen archived by Tkarcher 2459931 wikitext text/x-wiki 5yg4qhieahla4i1phadt8guicpx2cja 2459932 2459931 2026-09-23T14:52:33Z EtherpadBackupBot 54504 wikicon26-support-team archived by Tkarcher 2459932 wikitext text/x-wiki 3uaapbml22uoh2vsbiu5m06wlszwdbx 2459933 2459932 2026-09-23T14:52:54Z EtherpadBackupBot 54504 wikicon26-wikiprojektqueer archived by Tkarcher 2459933 wikitext text/x-wiki rejspzs8ljxb08ksu0vv8lkfqivf0c5 2459935 2459933 2026-09-23T14:53:13Z EtherpadBackupBot 54504 wikicon26-wikiprojekt_stolpersteine_goes_wikidata archived by Tkarcher 2459935 wikitext text/x-wiki s2a9247xc0h1ce0hvab8kxhtlcziq3d 2459937 2459935 2026-09-23T14:53:27Z EtherpadBackupBot 54504 wikicon26-usefulqueries archived by Tkarcher 2459937 wikitext text/x-wiki jgfl4y0zxofhbbajr5l4lqdp8w1rvlp 2459938 2459937 2026-09-23T14:53:40Z EtherpadBackupBot 54504 wikicon26-juedische_frauen_im_widerstand archived by Tkarcher 2459938 wikitext text/x-wiki 6xbi55zyt1hmjla5r7z2ukvmczfkag9 2459940 2459938 2026-09-23T14:53:56Z EtherpadBackupBot 54504 wikicon26-neues_von_wikimedia_commons archived by Tkarcher 2459940 wikitext text/x-wiki fka14etwnz4ek7mo5n6na3ojkodplgq 2459941 2459940 2026-09-23T14:54:09Z EtherpadBackupBot 54504 wikicon26-editathon_museum archived by Tkarcher 2459941 wikitext text/x-wiki 6q7eoyl71b3plyd45vci9433ip3pnf7 2459944 2459941 2026-09-23T14:54:28Z EtherpadBackupBot 54504 wikicon26-fotodigitalisierung_dia archived by Tkarcher 2459944 wikitext text/x-wiki f51etiamtnxq1unq6y8ys2u5p89x3iv 2459947 2459944 2026-09-23T14:54:39Z EtherpadBackupBot 54504 wikicon26-spezialbibliotheken archived by Tkarcher 2459947 wikitext text/x-wiki s77y9qajxk6d6vd8zgeqv6n0n9ae969 2459948 2459947 2026-09-23T14:54:56Z EtherpadBackupBot 54504 wikicon26-wikipedia_trainerinnen archived by Tkarcher 2459948 wikitext text/x-wiki 2osen5ikepxyrk1sx6l56fovlycxsg3 2459950 2459948 2026-09-23T14:55:09Z EtherpadBackupBot 54504 wikicon26-open_data archived by Tkarcher 2459950 wikitext text/x-wiki 9u85wwhl0tlx6do9n5lzoxqh1cinf54 2459952 2459950 2026-09-23T14:55:27Z EtherpadBackupBot 54504 wikicon26-demokratiegeschichte archived by Tkarcher 2459952 wikitext text/x-wiki 7t4ysr2tld0zz5vamqvr6q39iyex4fs 2459953 2459952 2026-09-23T14:55:42Z EtherpadBackupBot 54504 wikicon26-kleine-laender archived by Tkarcher 2459953 wikitext text/x-wiki timdlg12nwm9ap0mt79mrtp3gr2zzoo 2459956 2459953 2026-09-23T14:55:53Z EtherpadBackupBot 54504 wikicon26-gotha archived by Tkarcher 2459956 wikitext text/x-wiki 5cndhdptapqlp34bnsfmx943k1ow408 2459957 2459956 2026-09-23T14:56:05Z EtherpadBackupBot 54504 wikicon26-koloniale_sprache archived by Tkarcher 2459957 wikitext text/x-wiki 9ysrmivvm8y170uxpm1yyha3kkb7wgr 2459958 2459957 2026-09-23T14:56:19Z EtherpadBackupBot 54504 wikicon26-politikteam_von_wmde archived by Tkarcher 2459958 wikitext text/x-wiki itiy9zkd5f9hm03okid873wtaz8sjuw 2459960 2459958 2026-09-23T14:56:45Z EtherpadBackupBot 54504 wikicon26-ki_und_urheberrecht archived by Tkarcher 2459960 wikitext text/x-wiki 8ki42rdojl1pkz4li68l1zu58emybpp 2459961 2459960 2026-09-23T14:56:56Z EtherpadBackupBot 54504 wikicon26-mit_marginalisiertem_wissen_arbeiten archived by Tkarcher 2459961 wikitext text/x-wiki hmzcvvzbbmmm41koxn1xgwdpze4ivoa 2459962 2459961 2026-09-23T14:57:11Z EtherpadBackupBot 54504 wikicon26-kuwiki archived by Tkarcher 2459962 wikitext text/x-wiki 77it1z7ccinresvf3r50lx7xnoo6rwj 2459963 2459962 2026-09-23T14:57:25Z EtherpadBackupBot 54504 wikicon26-werkstatt_fuer_bessere_einleitungen archived by Tkarcher 2459963 wikitext text/x-wiki a1c7ara9225i2cfqx08in3so5v3c7q4 2459964 2459963 2026-09-23T14:57:40Z EtherpadBackupBot 54504 wikicon26-inklusion_und_barrierefreiheit archived by Tkarcher 2459964 wikitext text/x-wiki i3zld9dn47hfdmqsg7y6nningjvtebz 2459969 2459964 2026-09-23T14:57:55Z EtherpadBackupBot 54504 wikicon26-wikivoyage archived by Tkarcher 2459969 wikitext text/x-wiki 5a9qlabvb89z021ny8m60be94t2d8il 2459972 2459969 2026-09-23T14:58:12Z EtherpadBackupBot 54504 wikicon26-wiki_loves_broadcast_arbeitstreffen archived by Tkarcher 2459972 wikitext text/x-wiki 31u2inxir4wok8ak7hmm7ec64ze3cyo 2459973 2459972 2026-09-23T14:58:27Z EtherpadBackupBot 54504 wikicon26-openrefine archived by Tkarcher 2459973 wikitext text/x-wiki 9t80r6uhf1uagd61f1gt19kgp6of9r9 2459974 2459973 2026-09-23T14:58:44Z EtherpadBackupBot 54504 wikicon26-handyfotos archived by Tkarcher 2459974 wikitext text/x-wiki rkastrspaxk4jxx81kn63ci898q1krx 2459975 2459974 2026-09-23T14:59:00Z EtherpadBackupBot 54504 wikicon26-wikipediaartikel_aktuell_halten archived by Tkarcher 2459975 wikitext text/x-wiki skuym86j1fpzj7a0c4dscwwpv2r87cj 2459976 2459975 2026-09-23T14:59:12Z EtherpadBackupBot 54504 wikicon26-technische_entwicklungen archived by Tkarcher 2459976 wikitext text/x-wiki t2eq7wkz1pa8eiha6xlakxp90haircs 2459977 2459976 2026-09-23T14:59:26Z EtherpadBackupBot 54504 wikicon26-wissensgeneriung_mit_ki archived by Tkarcher 2459977 wikitext text/x-wiki 6ger6qp2socfw47vnx770njz218pisd 2459978 2459977 2026-09-23T14:59:36Z EtherpadBackupBot 54504 wikicon26-wikipediabus archived by Tkarcher 2459978 wikitext text/x-wiki cgsiojiqyovlclsepj9miyt5wqpzk9f 2459979 2459978 2026-09-23T14:59:50Z EtherpadBackupBot 54504 wikicon26-was_ist_glam archived by Tkarcher 2459979 wikitext text/x-wiki qv6n1ycga8v4wto1fov2z80qmex32mq 2459980 2459979 2026-09-23T15:00:04Z EtherpadBackupBot 54504 wikicon26-quartalstreffen_lokale_raeume archived by Tkarcher 2459980 wikitext text/x-wiki 2y7a0bfwee4143sqhhbwvf6ba0g1m88 2459981 2459980 2026-09-23T15:00:18Z EtherpadBackupBot 54504 wikicon26-was_sind_eigentlich_daten archived by Tkarcher 2459981 wikitext text/x-wiki mmac6z5agyduuxjw85hxtnykev6us58 2459982 2459981 2026-09-23T15:00:30Z EtherpadBackupBot 54504 wikicon26-was_sind_eigentlich_daten archived by Tkarcher 2459982 wikitext text/x-wiki a38bkas6t8v4ei5nxthzfrvmpk81h1c 2459984 2459982 2026-09-23T15:00:54Z EtherpadBackupBot 54504 wikicon26-Wikipedia_in-aus-und_(Hoch-)Schulen archived by Tkarcher 2459984 wikitext text/x-wiki 983ahbanj33hemmdo933c27axi7biux 0 460376 2460051 2459406 2026-09-23T16:13:01Z AWesterinen-WMF 54189 2460051 wikitext text/x-wiki 1f1plj0jkdqhukk8i8qemey60rhtwie 2460052 2460051 2026-09-23T16:17:47Z AWesterinen-WMF 54189 2460052 wikitext text/x-wiki eh24jhufiq6cjigdpwkos9xjy7142n7 0 460621 2459857 2458437 2026-09-23T13:53:22Z JGiannelos (WMF) 17083 2459857 wikitext text/x-wiki hm19ct9yqhlqegy4xwo9aqdm6rfc666 2459859 2459857 2026-09-23T13:56:41Z JGiannelos (WMF) 17083 2459859 wikitext text/x-wiki cry0xolpj1eg4is6gkhsenevofylzd1 2459862 2459859 2026-09-23T14:01:23Z JGiannelos (WMF) 17083 2459862 wikitext text/x-wiki sryyyyy68t5ujvt8a3083m07bqk2pju 2459863 2459862 2026-09-23T14:04:58Z JGiannelos (WMF) 17083 /* Service Level Objectives */ 2459863 wikitext text/x-wiki 0wl5qn41ybs0sof6f0vgo02ysb8vwxk 2459868 2459863 2026-09-23T14:12:57Z JGiannelos (WMF) 17083 /* Service Level Objectives */ 2459868 wikitext text/x-wiki 4g8gbxk3cwcq6k6he3z65opb40kv56t 2459871 2459868 2026-09-23T14:18:01Z JGiannelos (WMF) 17083 /* Future targets - worker capacity modelling */ 2459871 wikitext text/x-wiki 8q029n7xnwwtrjky1p2j8xm30yoax78 2459873 2459871 2026-09-23T14:19:18Z JGiannelos (WMF) 17083 /* Future targets - worker capacity modelling */ 2459873 wikitext text/x-wiki 656yev6tn8kaikxosroa5lbgtcwmpha 2459879 2459873 2026-09-23T14:22:05Z JGiannelos (WMF) 17083 /* Future targets - worker capacity modelling */ 2459879 wikitext text/x-wiki 0vjnf5z9j0jj5f7jqm7gkhilj6rqs6g 2459918 2459879 2026-09-23T14:43:00Z JGiannelos (WMF) 17083 /* Service Level Objectives */ 2459918 wikitext text/x-wiki 35h8wwchlq9c8gylbbp1ka3hsz19c51 0 460667 2460058 2459690 2026-09-23T16:37:00Z AWesterinen-WMF 54189 2460058 wikitext text/x-wiki at73tijzavflf640s73c0ei7e1cz0nu 2460061 2460058 2026-09-23T16:43:19Z AWesterinen-WMF 54189 2460061 wikitext text/x-wiki jcsdrdfw7655dorgbhayakadd1w6v58 2460245 2460061 2026-09-24T02:33:59Z AWesterinen-WMF 54189 /* Mix of manual and automatic modes in different scopes */ 2460245 wikitext text/x-wiki 9jul1yssjbys7egrwx8mw9cki1snl6l 2460246 2460245 2026-09-24T02:34:23Z AWesterinen-WMF 54189 /* Mix of manual and automatic modes in different scopes */ 2460246 wikitext text/x-wiki loycb5043gtc4y9gahl0i4ou5mnlt8h 2460249 2460246 2026-09-24T03:34:02Z AWesterinen-WMF 54189 2460249 wikitext text/x-wiki tmi3c5y0bk92q3wbao7jg8ez8xzyk3q 12 460669 2460057 2459693 2026-09-23T16:36:00Z SSapaty (WMF) 28955 /* Test Push-to-Deploy */ 2460057 wikitext text/x-wiki lhe4r9imtd6mv30bb23qb4is4dxtat5 2460066 2460057 2026-09-23T16:51:23Z SSapaty (WMF) 28955 /* Add a GitLab CI configuration */ Use the include directive for push-to-deploy configuration 2460066 wikitext text/x-wiki 5va3zjrdr9xp1ekgbzvwmbw39nzaics 2 460677 2459834 2459415 2026-09-23T13:13:21Z JGiannelos (WMF) 17083 /* Discovery */ 2459834 wikitext text/x-wiki d0r98i4s2qr31gsnwstxzqwopidyhsh 2459836 2459834 2026-09-23T13:16:41Z JGiannelos (WMF) 17083 /* Kubernetes */ 2459836 wikitext text/x-wiki rnyguhnton69s5nbeciddn7idvyjp9v 2459838 2459836 2026-09-23T13:19:43Z JGiannelos (WMF) 17083 2459838 wikitext text/x-wiki 040l8p8i9hd7okxw6f63wn1b1po2wdf 2459839 2459838 2026-09-23T13:22:52Z JGiannelos (WMF) 17083 /* REST Gateway */ 2459839 wikitext text/x-wiki kzds8udiskn5y2k8fo4x61o2kfc1pjf 2459843 2459839 2026-09-23T13:25:41Z JGiannelos (WMF) 17083 /* Common tasks */ 2459843 wikitext text/x-wiki emgwave6yn9on69b0j3bac9t5buckl5 2459844 2459843 2026-09-23T13:26:01Z JGiannelos (WMF) 17083 /* Invalidate app cache manually */ 2459844 wikitext text/x-wiki j24rponn5qdhjlc72bfbi8epgz87gy8 2459847 2459844 2026-09-23T13:29:43Z JGiannelos (WMF) 17083 /* Common tasks */ 2459847 wikitext text/x-wiki 2kd508rsu6fpmm3b4gaqrc5xd5ev0ex 2459850 2459847 2026-09-23T13:34:32Z JGiannelos (WMF) 17083 /* Image building */ 2459850 wikitext text/x-wiki dsnvtkjnv1wclo5fuaclmv1bowtgjfi 0 460682 2460243 2458522 2026-09-24T01:41:31Z AndrewBogott (WMF) 51047 /* Magnum infra reports as down */ 2460243 wikitext text/x-wiki q6ev25ar7pzopgrvfs97j1q52pjldb9 2460244 2460243 2026-09-24T01:42:40Z AndrewBogott (WMF) 51047 /* Magnum infra reports as down */ 2460244 wikitext text/x-wiki nh14six3hare1gdkudia5pc8oh1zpkh 0 460683 2459896 2458297 2026-09-23T14:27:52Z CCiufo-WMF 37368 CCiufo-WMF moved page [[Wikimedia Cloud Services team/EnhancementProposals/Toolforge Alerting System]] to [[Wikimedia Cloud Services team/EnhancementProposals/Toolforge Alerting Service]] 2458297 wikitext text/x-wiki c6mpph4c4qnby06xgg7xxslvg2fco2p 2459899 2459896 2026-09-23T14:28:28Z CCiufo-WMF 37368 2459899 wikitext text/x-wiki 2synml9me81xb6ulxs98vrb1lu6b227 2 460699 2460105 2459686 2026-09-23T18:08:56Z SSapaty (WMF) 28955 /* Add a GitLab CI configuration */ 2460105 wikitext text/x-wiki 04p2y0qimig2e2h9jm1g6do46xb5483 0 460710 2460247 2459055 2026-09-24T03:24:00Z AWesterinen-WMF 54189 2460247 wikitext text/x-wiki agxno8w1jsf1r79bibsu3yagx1afks1 2460250 2460247 2026-09-24T03:35:03Z AWesterinen-WMF 54189 2460250 wikitext text/x-wiki av3fgim97quflhtkuglzar0skjqnbqd 0 460718 2459849 2459787 2026-09-23T13:33:22Z Ottomata 88 /* Details and caveats */ removed bullet point about canary events 2459849 wikitext text/x-wiki fbhgv3o151lfqpjq893lb08rij3zt9g 116 460723 2459860 2026-09-23T13:57:07Z Sikjunior 59828 Created page with "{{Tool |name = sw-sister-alerts |title = Swahili Sister Update Alerts |description = Shows Swahili Wikipedia editors when a linked English article was updated more recently, so they can check whether the Swahili page needs an update. Signal only; editors decide. |website = https://sw-sister-alerts.toolforge.org/ |keywords = swahili, wikipedia, wikidata, interwiki, editors, tanzania, toolforge, updates, alerts |author = Sikjunior Mrimi |username = Meetsik |maintainer = ..." 2459860 wikitext text/x-wiki e92w4xpfkz26v8oqkasc632tdnt6a88 2460060 2459860 2026-09-23T16:40:16Z JJMC89 7474 removed [[Category:Tools]] using [[w:Wikipedia:HotCat|HotCat]] 2460060 wikitext text/x-wiki pcnov29orgistx8ryt8euueibhxeehj 2460062 2460060 2026-09-23T16:45:18Z JJMC89 7474 fix 2460062 wikitext text/x-wiki 9hv5f24m5ioelu9zu021lt3i36c3b3v 0 460724 2459897 2026-09-23T14:27:52Z CCiufo-WMF 37368 CCiufo-WMF moved page [[Wikimedia Cloud Services team/EnhancementProposals/Toolforge Alerting System]] to [[Wikimedia Cloud Services team/EnhancementProposals/Toolforge Alerting Service]] 2459897 wikitext text/x-wiki 58wa8iqaiewlat5v2c7zeqjhw79e5fp 0 460725 2460168 2026-09-23T20:24:45Z BCornwall-WMF 31521 Redirect to [[Acme-chief#Cloud_VPS_setup]] 2460168 wikitext text/x-wiki 75qtsq0e8yqghypk12qk6h3uihwrelx 12 460726 2460248 2026-09-24T03:29:17Z SSapaty (WMF) 28955 Tutorial moved from draft 2460248 wikitext text/x-wiki 04p2y0qimig2e2h9jm1g6do46xb5483 2460265 2460248 2026-09-24T07:45:03Z JJMC89 7474 +standard templates; ce 2460265 wikitext text/x-wiki mfvos11215u54y0gj9ckbe6mpdzr43o