Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.16 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 1380 2449844 2297998 2026-08-19T17:08:24Z Stashbot 7414 James_F: Created integration-cumin-01 for T433592 2449844 wikitext text/x-wiki 8rnnpbp8i9cndgomcmta4pebz9rwn77 1 2515 2449942 2440899 2026-08-20T04:40:12Z WǂMǂFOffice 58783 /* */ 2449942 wikitext text/x-wiki s3y7od74uuzqloyec6j1uv88e5rpb30 2449944 2449942 2026-08-20T04:40:23Z JJMC89 7474 Reverted edit by [[Special:Contributions/WǂMǂFOffice|WǂMǂFOffice]] ([[User talk:WǂMǂFOffice|talk]]) to last revision by [[User:2600 etc|2600 etc]] 2440899 wikitext text/x-wiki hewmai4x79oac5qnzso0gde8ht6jscr 498 3086 2449989 2449601 2026-08-20T10:30:47Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2449989 wikitext text/x-wiki h1g7ttkim41ik4di7n417gzdw8jkl8q 2449999 2449989 2026-08-20T10:44:18Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2449999 wikitext text/x-wiki gv5qkiszzva0fdbsi41ishxde5u8hh1 0 3685 2449876 2304201 2026-08-19T18:35:41Z Novem Linguae 23682 /* IPv6 network status at Wikimedia */ wl 2449876 wikitext text/x-wiki g2r9ilau6r8tl2vu10oqf947q8uhx0g 0 4108 2449669 2449645 2026-08-19T12:53:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1327110]] to Wednesday, August 19 UTC afternoon backport window 2449669 wikitext text/x-wiki q8rgc239nktufii7a6byqc7dnp0gbp9 2449670 2449669 2026-08-19T12:53:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1327111]] to Wednesday, August 19 UTC afternoon backport window 2449670 wikitext text/x-wiki opcu92qtz92dmri6651bcbrxvan23ff 2449693 2449670 2026-08-19T13:18:14Z ScheduleDeploymentBot 37566 Add [[gerrit:1326342]] to Wednesday, August 19 UTC afternoon backport window 2449693 wikitext text/x-wiki luzhilrr99yqirvxfee1o6xrdgyez3r 2449711 2449693 2026-08-19T13:48:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1314962]] to Wednesday, August 19 UTC afternoon backport window 2449711 wikitext text/x-wiki qfauh74yq07xzus26frz3800cltlhaq 2449826 2449711 2026-08-19T16:19:01Z Brennen 54136 /* {{Deployment_day|date=2026-08-20}} */ Add Phabricator switchover window. 2449826 wikitext text/x-wiki 06crs3ngqp2riegv6oj88ot9r2mi9aq 2449840 2449826 2026-08-19T16:53:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1325878]] to Wednesday, August 19 UTC late backport window 2449840 wikitext text/x-wiki j8h58ikj1c6j43kxqbgd3ljeyr42yb8 2449857 2449840 2026-08-19T17:42:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1327162]] to Wednesday, August 19 UTC late backport window 2449857 wikitext text/x-wiki 9c46eohu5x5dmq9zx0ncrnuyyeev81t 2449882 2449857 2026-08-19T19:24:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1327178]] to Wednesday, August 19 UTC late backport window 2449882 wikitext text/x-wiki 61xyryhdqck07fb9b1onvpfnacdcrac 2449884 2449882 2026-08-19T19:48:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1327128]] to Wednesday, August 19 UTC late backport window 2449884 wikitext text/x-wiki r303hcvqbezg0nehl8jyt3a7r1s9q73 2449986 2449884 2026-08-20T10:25:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1326848]] to Thursday, August 20 UTC afternoon backport window 2449986 wikitext text/x-wiki 7sadkwxl1ki016os4u68b7a775qeler 2450004 2449986 2026-08-20T11:02:09Z ScheduleDeploymentBot 37566 Add [[gerrit:1327511]] to Thursday, August 20 UTC afternoon backport window 2450004 wikitext text/x-wiki b8mb9uf0ju0n86gns096tnuwzpe7ina 0 4575 2449875 2363779 2026-08-19T18:35:22Z Novem Linguae 23682 /* Present */ wl 2449875 wikitext text/x-wiki 04k6si43cbpg0gs7junzwi1lb8qzpj3 2449879 2449875 2026-08-19T18:37:02Z Novem Linguae 23682 /* Present */ wl 2449879 wikitext text/x-wiki fn3f04soifojrp0p25kwpzfjtoe67ct 0 6479 2450022 2448312 2026-08-20T11:52:40Z CGoubert-WMF 32775 Remove lockmanager from redis_misc following move to redis_lock 2450022 wikitext text/x-wiki kc5cyc6iggbfj46k5i9i2p835jnkg28 498 6726 2449975 2449598 2026-08-20T10:06:52Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2449975 wikitext text/x-wiki 0mg2wr0k30orvrfbsirptycr6o6a6s8 2449976 2449975 2026-08-20T10:06:53Z Stashbot 7414 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api 2449976 wikitext text/x-wiki bgokg1kfsc387m8jfdko63dytnazvsv 2449977 2449976 2026-08-20T10:09:54Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2449977 wikitext text/x-wiki 8pb7ln74vhjd0fdp3h840jmwt1odwk2 2449984 2449977 2026-08-20T10:23:14Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2449984 wikitext text/x-wiki dv5i7bcl3wy0utfcdezvu3ca1d06aq8 2450015 2449984 2026-08-20T11:43:20Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2450015 wikitext text/x-wiki tjhtbu8tadlcgpk450menp1ztiufd2o 2450016 2450015 2026-08-20T11:43:59Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2450016 wikitext text/x-wiki n5npuwov1ynvx514r2bth0z0znzgiti 2450017 2450016 2026-08-20T11:45:40Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2450017 wikitext text/x-wiki tvuxve8jvhhs0linmxghj75ldc1lkug 2450027 2450017 2026-08-20T11:58:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2450027 wikitext text/x-wiki meicwkx7eq5dwfgyryfc5qtsxmdh19x 0 7919 2449647 2449646 2026-08-19T12:04:27Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet 2449647 wikitext text/x-wiki 1lzuso9hjogfo4vyv415beyt68jofyo 2449652 2449647 2026-08-19T12:35:10Z Stashbot 7414 blake@deploy1003: Started scap sync-world: non-build deploy for T417800 2449652 wikitext text/x-wiki 4dkl0txxvi49suvrhzqo4jclqgvn2pg 2449653 2449652 2026-08-19T12:36:49Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2449653 wikitext text/x-wiki d0cqyfod1ppdg4h0se3uhczql3exyth 2449654 2449653 2026-08-19T12:38:26Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: non-build deploy for T417800 (duration: 03m 52s) 2449654 wikitext text/x-wiki n7ck4yhgdd2chr5hm56tprb053ndw0w 2449655 2449654 2026-08-19T12:38:51Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1327096|Switch to redis lock manager on s4 and s8 (T366938)]] 2449655 wikitext text/x-wiki 8o508zk6ann8sdtjnoafad65pj5llxp 2449656 2449655 2026-08-19T12:40:55Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1327096|Switch to redis lock manager on s4 and s8 (T366938)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449656 wikitext text/x-wiki pe0htqvcsn85kx1s0p5kaj3smrz37s6 2449657 2449656 2026-08-19T12:41:21Z Stashbot 7414 jelto: update cert-manager to 1.19.6 on wikikube staging-codfw - T427402 2449657 wikitext text/x-wiki otho70lxyee1l55fydqxanmyh35vh4z 2449658 2449657 2026-08-19T12:42:06Z Stashbot 7414 jelto@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. 2449658 wikitext text/x-wiki 7yv9fxhunssz1r86a8fek843qhtdmwm 2449659 2449658 2026-08-19T12:42:15Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqord cct - cmooney@cumin1003" 2449659 wikitext text/x-wiki ac9x2em7q381kpvh0ue20xk4bozdvai 2449660 2449659 2026-08-19T12:42:47Z Stashbot 7414 jelto@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. 2449660 wikitext text/x-wiki 9vl3sms9yh1ostuwbcqwuqgbfd21zfr 2449661 2449660 2026-08-19T12:43:43Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2449661 wikitext text/x-wiki mbstrx8ju8lw9ak7v8am0ejwzrwhb2n 2449662 2449661 2026-08-19T12:45:36Z Stashbot 7414 elukey: move the /v2/releng.* prefix on the Docker Registry to its new s3 backend - T432829 2449662 wikitext text/x-wiki 4soqqzypgpa392cbcpl22hhynjpvxxz 2449663 2449662 2026-08-19T12:45:40Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqord cct - cmooney@cumin1003" 2449663 wikitext text/x-wiki t3qju6a86jj3dkoq3pk6tpzt316d7fy 2449664 2449663 2026-08-19T12:45:43Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2449664 wikitext text/x-wiki lglfr5a2a1ioe5pscdfs3utvej4d81j 2449665 2449664 2026-08-19T12:46:07Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2449665 wikitext text/x-wiki d8rttmetzbh850v7r04k0cilcokp91p 2449666 2449665 2026-08-19T12:48:02Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327096|Switch to redis lock manager on s4 and s8 (T366938)]] (duration: 09m 11s) 2449666 wikitext text/x-wiki 4p3cljaosu0w4perarfpz3nkz5s366m 2449667 2449666 2026-08-19T12:48:23Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2449667 wikitext text/x-wiki 67q4qecnj2wmgvx2iodimxll9576lrt 2449668 2449667 2026-08-19T12:53:11Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2902.codfw.wmnet with reason: Cloning 2449668 wikitext text/x-wiki 3upy6qg0mxh90nvlzg04ydpvgkg1mq3 2449672 2449668 2026-08-19T12:57:12Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and A:durum 2449672 wikitext text/x-wiki gusjmd03pq011m34ckp586fvcx2koje 2449673 2449672 2026-08-19T12:59:10Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449673 wikitext text/x-wiki djx9n9lf04gj2beh5bmftfvkjjpndxr 2449674 2449673 2026-08-19T12:59:17Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449674 wikitext text/x-wiki 3bnkqae5cam529b5jfeueqxxfzjn7m2 2449675 2449674 2026-08-19T12:59:48Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449675 wikitext text/x-wiki j4kz11y04extuyuvv7xq4j6xq0rstoi 2449676 2449675 2026-08-19T12:59:55Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449676 wikitext text/x-wiki 5qrx2xutoxtofm359d9mdiy640bzdul 2449677 2449676 2026-08-19T13:00:52Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2449677 wikitext text/x-wiki e7gpv2ylujqbq3fcx6o6mx792rfkb93 2449678 2449677 2026-08-19T13:01:04Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host ncmonitor1001.eqiad.wmnet 2449678 wikitext text/x-wiki kwafqcqbpsikhe3coye5j4c2mbdr3a8 2449679 2449678 2026-08-19T13:01:35Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449679 wikitext text/x-wiki bj99weqp5h31ubn3b9mfmjeyolw8ze9 2449680 2449679 2026-08-19T13:01:42Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449680 wikitext text/x-wiki 8zdoe5543z1ydmxiiu2xx999patk4kh 2449681 2449680 2026-08-19T13:03:04Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2449681 wikitext text/x-wiki d8r3gji86daog1do4bi9odsixd3a06r 2449683 2449681 2026-08-19T13:05:42Z Stashbot 7414 cjd91: sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-eqiad' --task-id T434478 --reason '9.2.15 upgrade' 2449683 wikitext text/x-wiki f372nw6ccdi6u4dpbmrsakn8lauwipe 2449684 2449683 2026-08-19T13:05:59Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478) 2449684 wikitext text/x-wiki 8gj805bz94kkdngdaeigp8oqydkh1qb 2449685 2449684 2026-08-19T13:06:33Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478) 2449685 wikitext text/x-wiki cif9mmg9co8wyuiy9g7henoy0m4yvdx 2449686 2449685 2026-08-19T13:06:38Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1327111|Special:AbuseReview: Add "no further action needed" review action (T435020)]], [[gerrit:1327110|AbuseReview: Take the review verdict as a REST path parameter (T435020)]] 2449686 wikitext text/x-wiki 7kcojaz4ymyrl62m6sgisk1bbp14rbe 2449687 2449686 2026-08-19T13:10:25Z Stashbot 7414 cdobbins@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=97) Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478) 2449687 wikitext text/x-wiki n101o23lweiriz0bi0bgtnqr8slsunp 2449688 2449687 2026-08-19T13:10:29Z Stashbot 7414 cdobbins@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=97) Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478) 2449688 wikitext text/x-wiki oshiodf26vt1l5o187d2wvzzspcpvry 2449689 2449688 2026-08-19T13:12:35Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449689 wikitext text/x-wiki ri95we8s2dhtc1kukwo20o6mo7t1ujj 2449690 2449689 2026-08-19T13:12:43Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449690 wikitext text/x-wiki boptdgnz4yc8fhhtpg13fqwcwems5qb 2449691 2449690 2026-08-19T13:16:52Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449691 wikitext text/x-wiki 887czz9sfbww0dphklc907wcwl7cge5 2449692 2449691 2026-08-19T13:16:59Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449692 wikitext text/x-wiki t1gwt7a28otzx8itvf8r1ve1s9pvzqy 2449694 2449692 2026-08-19T13:23:59Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1327111|Special:AbuseReview: Add "no further action needed" review action (T435020)]], [[gerrit:1327110|AbuseReview: Take the review verdict as a REST path parameter (T435020)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449694 wikitext text/x-wiki ckq64mr2jgw5ezp0w81m00wz5u27czi 2449695 2449694 2026-08-19T13:24:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1147 2449695 wikitext text/x-wiki 8byuhwxnaxygctu12x50uzperj68aos 2449696 2449695 2026-08-19T13:24:52Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1147 2449696 wikitext text/x-wiki tae0aynr7hvvqhq91j3dutjsda52tj5 2449697 2449696 2026-08-19T13:24:55Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2449697 wikitext text/x-wiki nsp702zzquabb66zq4oil5ty1nhiwoi 2449698 2449697 2026-08-19T13:25:36Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=97) rolling reboot on A:durum and A:durum 2449698 wikitext text/x-wiki eojbvqcp2xazei5ep7imgg8fwhdt3nc 2449699 2449698 2026-08-19T13:28:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449699 wikitext text/x-wiki it7gc7rjksx192b5c1qfuqmqzyf1r9f 2449700 2449699 2026-08-19T13:28:10Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449700 wikitext text/x-wiki pkyzzanvp2f81fjkm4kerh242d27j8o 2449701 2449700 2026-08-19T13:31:34Z Stashbot 7414 swfrench-wmf: starting etcd-main codfw -> eqiad switchover - T435103 2449701 wikitext text/x-wiki emyu2dp529yiyvg0njkeymni7mta0fm 2449702 2449701 2026-08-19T13:37:52Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327111|Special:AbuseReview: Add "no further action needed" review action (T435020)]], [[gerrit:1327110|AbuseReview: Take the review verdict as a REST path parameter (T435020)]] (duration: 31m 43s) 2449702 wikitext text/x-wiki h76xo3m3nu1035oqjk6puxgk3g909ko 2449703 2449702 2026-08-19T13:39:08Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2449703 wikitext text/x-wiki 9mkhfsaxqj0gr9i3er16sos7atd2q9v 2449704 2449703 2026-08-19T13:39:54Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1325532|srwiki: Allow bureaucrats to add and remove event-organizer group (T434748)]] 2449704 wikitext text/x-wiki kecct9r3yxx9ppenffjpwpc1s7jtvxv 2449705 2449704 2026-08-19T13:41:16Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2449705 wikitext text/x-wiki 681ln0m1mb6n8egkw5obyb7at735z9u 2449706 2449705 2026-08-19T13:44:10Z Stashbot 7414 swfrench@cumin2003: conftool action : set/pooled=no; selector: name=wikikube-worker2330.codfw.wmnet 2449706 wikitext text/x-wiki d1u0a7q501vbpwu1emj8nd09x96i3b6 2449707 2449706 2026-08-19T13:44:15Z Stashbot 7414 kharlan@deploy1003: kharlan, danielyepezgarces: Backport for [[gerrit:1325532|srwiki: Allow bureaucrats to add and remove event-organizer group (T434748)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449707 wikitext text/x-wiki 5c3es6e0h0wtadq3beim3tmd6vslq3e 2449708 2449707 2026-08-19T13:44:19Z Stashbot 7414 swfrench-wmf: finished etcd-main codfw -> eqiad switchover - T435103 2449708 wikitext text/x-wiki 11f2v5bwcksbqvz2xnlv4c51of3679c 2449709 2449708 2026-08-19T13:45:07Z Stashbot 7414 swfrench@cumin2003: conftool action : set/pooled=yes; selector: name=wikikube-worker2330.codfw.wmnet 2449709 wikitext text/x-wiki fzef8or7wejr2mfhsgl5pi30da4af20 2449710 2449709 2026-08-19T13:48:02Z Stashbot 7414 kharlan@deploy1003: kharlan, danielyepezgarces: Continuing with deployment 2449710 wikitext text/x-wiki 4ms3dgge28igu8tfryeb4ipfc8mbcu3 2449712 2449710 2026-08-19T13:51:31Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449712 wikitext text/x-wiki r5q6sg776uz5gs0l96krqw17dd9uz30 2449713 2449712 2026-08-19T13:51:39Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449713 wikitext text/x-wiki njcdua5garobh1df6to6r8bamdpq6em 2449714 2449713 2026-08-19T13:52:16Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2449714 wikitext text/x-wiki rh7s5m2j5234kl7abyfqet6u2kp4dqx 2449715 2449714 2026-08-19T13:54:26Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2449715 wikitext text/x-wiki pkxut97wxt0v32vai76bansv63v58y8 2449716 2449715 2026-08-19T13:54:51Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325532|srwiki: Allow bureaucrats to add and remove event-organizer group (T434748)]] (duration: 14m 56s) 2449716 wikitext text/x-wiki o77oba2tkke1qsh04svfi20tcgqp3by 2449717 2449716 2026-08-19T13:56:04Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2449717 wikitext text/x-wiki 3gshkzon0930pmhhj3zkjsu4oevjp3p 2449718 2449717 2026-08-19T13:56:22Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1326342|Revert^2 "wmf-config/ProductionServices: set URL for urldownloader to service record"]] 2449718 wikitext text/x-wiki ltqn5afnn6yn4wfc57iebss9amjeekb 2449719 2449718 2026-08-19T13:57:39Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host stat1008.eqiad.wmnet with OS bookworm 2449719 wikitext text/x-wiki 3o15ommzc2smlmk0yncg2xcscs9nef0 2449720 2449719 2026-08-19T13:58:26Z Stashbot 7414 kharlan@deploy1003: kharlan, sukhe: Backport for [[gerrit:1326342|Revert^2 "wmf-config/ProductionServices: set URL for urldownloader to service record"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449720 wikitext text/x-wiki 1n51v5e86uj5fm6ilntd2sg9ylefna6 2449721 2449720 2026-08-19T13:59:46Z Stashbot 7414 kharlan@deploy1003: kharlan, sukhe: Continuing with deployment 2449721 wikitext text/x-wiki ed0iq5cfgskcv7lxyt6aet6h3xttw92 2449722 2449721 2026-08-19T14:04:04Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1326342|Revert^2 "wmf-config/ProductionServices: set URL for urldownloader to service record"]] (duration: 07m 40s) 2449722 wikitext text/x-wiki 90bln281m5oz617os79nv70s6ms23on 2449723 2449722 2026-08-19T14:04:20Z Stashbot 7414 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2449723 wikitext text/x-wiki 9swjc4z5jbsiqzf39klj6wcr4wl5d9a 2449724 2449723 2026-08-19T14:05:23Z Stashbot 7414 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2449724 wikitext text/x-wiki i0snsdcokez93oe0r4lu9wwge6kta2t 2449725 2449724 2026-08-19T14:05:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet 2449725 wikitext text/x-wiki 03itnpk8agz18vkbn1ikor6w3wlf9wr 2449726 2449725 2026-08-19T14:05:58Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1314962|static: add new dir bimi/ for BIMI SVG and PEM file (T311685)]] 2449726 wikitext text/x-wiki kms3fct8pjsgz3iclyp3bm63z5i9imv 2449727 2449726 2026-08-19T14:06:02Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2449727 wikitext text/x-wiki r7xjcj4rwkink239e2jpzjiahpr3d12 2449728 2449727 2026-08-19T14:06:37Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449728 wikitext text/x-wiki axy6w3fcph3ynkzs9r4r631d7xx1qpu 2449729 2449728 2026-08-19T14:06:44Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449729 wikitext text/x-wiki pbst8wqjlhd4miqa0taho3zmvohbuo0 2449730 2449729 2026-08-19T14:08:02Z Stashbot 7414 kharlan@deploy1003: kharlan, sukhe: Backport for [[gerrit:1314962|static: add new dir bimi/ for BIMI SVG and PEM file (T311685)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449730 wikitext text/x-wiki 68opahgusj68fv8ooo58m4p5lvqoqrn 2449732 2449730 2026-08-19T14:09:26Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2449732 wikitext text/x-wiki fb3dpivct8nbrydxmb9wbgmhqmvzzhc 2449733 2449732 2026-08-19T14:09:36Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2449733 wikitext text/x-wiki kwrzgalngb6wragzcbpve5c0cjhrx9x 2449734 2449733 2026-08-19T14:11:06Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2449734 wikitext text/x-wiki 5xu793kfly7r31oj9sm6kfludxsjirx 2449735 2449734 2026-08-19T14:11:45Z Stashbot 7414 kharlan@deploy1003: kharlan, sukhe: Continuing with deployment 2449735 wikitext text/x-wiki nay5hgkuqfyfaqqrprwirxeibk3l8s8 2449737 2449735 2026-08-19T14:15:37Z Stashbot 7414 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2449737 wikitext text/x-wiki nyvx91jc70iibjm2nzocvv6q1kk7j3y 2449738 2449737 2026-08-19T14:15:58Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1314962|static: add new dir bimi/ for BIMI SVG and PEM file (T311685)]] (duration: 10m 00s) 2449738 wikitext text/x-wiki plpxrnnlay0iv1luvfbat8yta9l5sng 2449739 2449738 2026-08-19T14:16:02Z Stashbot 7414 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2449739 wikitext text/x-wiki 5l5j7uswbo5nedk07v8g66dxqo1d4q1 2449740 2449739 2026-08-19T14:16:37Z Stashbot 7414 elukey: upgrade spicerack on cumin1003 and cumin2003 2449740 wikitext text/x-wiki 1uwiusraja3sgk8vs34t8n6at4yiu44 2449741 2449740 2026-08-19T14:16:58Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2449741 wikitext text/x-wiki kflqh9osts6c9blrnw5xr46uqfnyewp 2449742 2449741 2026-08-19T14:17:36Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2449742 wikitext text/x-wiki p01fadcxx8bkej9shq9dax7phk45xbp 2449743 2449742 2026-08-19T14:17:41Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2449743 wikitext text/x-wiki oktmt0630pnvkwa0x34p6ded9xmlmu7 2449744 2449743 2026-08-19T14:17:44Z Stashbot 7414 topranks: disable puppet on hosts running BIRD BGP to test merge of patch to systemd healtchcheck service 2449744 wikitext text/x-wiki 7rw81oqv1948hf266juqn3roifpjch7 2449745 2449744 2026-08-19T14:18:20Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2449745 wikitext text/x-wiki 6c9sneplyatvwm0ld6rr294n4t49kzh 2449746 2449745 2026-08-19T14:23:35Z Stashbot 7414 tgr@deploy1003: Started scap sync-world: Backport for [[gerrit:1327094|SpecialMWOAuthListConsumers: Handle newFromMWUser returning null in addNavigationSubtitle (T435167)]] 2449746 wikitext text/x-wiki rtet6ghefozk7011nk89tpel4tgksys 2449747 2449746 2026-08-19T14:23:57Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.hosts.reboot-single for host durum3005.esams.wmnet 2449747 wikitext text/x-wiki qallah9vgy813wm73fw4ufw0lcaafn1 2449748 2449747 2026-08-19T14:25:39Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2449748 wikitext text/x-wiki tooqmqdm00jnghc827cg9rao1u95rco 2449749 2449748 2026-08-19T14:28:08Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host durum3005.esams.wmnet 2449749 wikitext text/x-wiki n41v7pmy7dx3danzloa3qcd69u5p81s 2449750 2449749 2026-08-19T14:28:42Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449750 wikitext text/x-wiki 9yeakouciyrqy9enbl7wwogv7lugjn0 2449751 2449750 2026-08-19T14:28:49Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449751 wikitext text/x-wiki aunx6vjepwxhu3vkvm2cer6j12dc6cp 2449753 2449751 2026-08-19T14:36:58Z Stashbot 7414 tgr@deploy1003: tgr: Backport for [[gerrit:1327094|SpecialMWOAuthListConsumers: Handle newFromMWUser returning null in addNavigationSubtitle (T435167)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449753 wikitext text/x-wiki ptivkpudngzl5xhz0cr9703j9vlg5kt 2449754 2449753 2026-08-19T14:38:41Z Stashbot 7414 tgr@deploy1003: tgr: Continuing with deployment 2449754 wikitext text/x-wiki 2d41q72rzy1qebyhmxr7nete1j93vx6 2449755 2449754 2026-08-19T14:41:16Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2449755 wikitext text/x-wiki 0tt1vyaikvwdss52wmv6s85zdqrivpu 2449756 2449755 2026-08-19T14:42:13Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage 2449756 wikitext text/x-wiki r9nckz5aq7n5jkauocgt6e5bv0x0b66 2449757 2449756 2026-08-19T14:42:20Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.hosts.reboot-single for host durum1001.eqiad.wmnet 2449757 wikitext text/x-wiki d7xm808mfgnzshxqpru1xxmvxs2rocs 2449758 2449757 2026-08-19T14:43:10Z Stashbot 7414 tgr@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327094|SpecialMWOAuthListConsumers: Handle newFromMWUser returning null in addNavigationSubtitle (T435167)]] (duration: 19m 25s) 2449758 wikitext text/x-wiki 9evrnehcurnhdg4p21zd3a4e360oagk 2449759 2449758 2026-08-19T14:43:34Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1147.eqiad.wmnet with OS bookworm 2449759 wikitext text/x-wiki 0cobs5o25f1hjlusrev52bvbykyhh1x 2449760 2449759 2026-08-19T14:46:11Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Delete db2902 ipv6 addr - fceratto@cumin1003" 2449760 wikitext text/x-wiki 751o9wmpm1iovulxc11kl0vu33s3p2v 2449761 2449760 2026-08-19T14:46:15Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Delete db2902 ipv6 addr - fceratto@cumin1003" 2449761 wikitext text/x-wiki 3q3hsougeepyjy8aa3tiz8zjvuffpqb 2449762 2449761 2026-08-19T14:46:17Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2449762 wikitext text/x-wiki e1n1flhbbfq5npser71gt6r05umhxu5 2449764 2449762 2026-08-19T14:48:19Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host durum1001.eqiad.wmnet 2449764 wikitext text/x-wiki 15wpul3v10vhvr4yg38h3erctoaqboa 2449765 2449764 2026-08-19T14:49:03Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage 2449765 wikitext text/x-wiki 9knsci6a7gwbxkegjlbhmh9h63srydo 2449767 2449765 2026-08-19T14:55:16Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2449767 wikitext text/x-wiki onmt63qnxw6hf8ujjrcaj6jopgbqmfo 2449768 2449767 2026-08-19T14:55:54Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2449768 wikitext text/x-wiki pw0iwmgc9ssiwjffuyr9a9wyneqoci0 2449769 2449768 2026-08-19T14:58:15Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1147.eqiad.wmnet with reason: host reimage 2449769 wikitext text/x-wiki th503g0zzaf6ybhbillteq3mu3byqcb 2449770 2449769 2026-08-19T14:59:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449770 wikitext text/x-wiki cjnfm7ae4tz6dfmteu3qrb37dwhn4dp 2449771 2449770 2026-08-19T14:59:44Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449771 wikitext text/x-wiki embd3amo9qdz2yil3wk5zonez5xcbgs 2449772 2449771 2026-08-19T15:04:39Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1147.eqiad.wmnet with reason: host reimage 2449772 wikitext text/x-wiki c9yq08su05qjzktp46vmgp582kd6ewi 2449773 2449772 2026-08-19T15:05:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449773 wikitext text/x-wiki gojepgmlnin1askibfnglh5thjuuwqm 2449774 2449773 2026-08-19T15:05:40Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449774 wikitext text/x-wiki bz6ipaxnso5mqvp8613kjxlozzg6gxw 2449775 2449774 2026-08-19T15:06:19Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie 2449775 wikitext text/x-wiki i5kgmzr0i5avvjrj9694aobdat9zo9m 2449776 2449775 2026-08-19T15:10:04Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1327098|UserRightsNotificationHandler: Use correct database for Echo (T435263)]], [[gerrit:1327101|UserRightsNotificationHandler: Use correct database for Echo (T435263)]] 2449776 wikitext text/x-wiki 7xv6kr4320nv64qsfyiizdkwxfsmc1c 2449777 2449776 2026-08-19T15:12:25Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1327098|UserRightsNotificationHandler: Use correct database for Echo (T435263)]], [[gerrit:1327101|UserRightsNotificationHandler: Use correct database for Echo (T435263)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449777 wikitext text/x-wiki je34nlpzj0cste7hkorxa4ymgjgca3s 2449778 2449777 2026-08-19T15:13:00Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2449778 wikitext text/x-wiki aik76qak91ry8qhaxr30kuzhwcmf62w 2449779 2449778 2026-08-19T15:17:16Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327098|UserRightsNotificationHandler: Use correct database for Echo (T435263)]], [[gerrit:1327101|UserRightsNotificationHandler: Use correct database for Echo (T435263)]] (duration: 07m 13s) 2449779 wikitext text/x-wiki begvfmt0r22b64t6o2ehdpf9lscz58f 2449780 2449779 2026-08-19T15:19:05Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Host crashed 2449780 wikitext text/x-wiki p6t82hkytl62fyyfp52gkt5akxxb4a2 2449781 2449780 2026-08-19T15:24:36Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host krb1002.eqiad.wmnet 2449781 wikitext text/x-wiki jc4ad7kytnysurb95rhdnjvakrby7lv 2449782 2449781 2026-08-19T15:25:36Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1327138|Enable redis lock manager everywhere (T366938)]] 2449782 wikitext text/x-wiki iy27og39ubn49uwhordb0901y3sptq9 2449783 2449782 2026-08-19T15:28:03Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1327138|Enable redis lock manager everywhere (T366938)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449783 wikitext text/x-wiki 1j2kuyn1u0c00qf8oo58q4cyh6o2mdf 2449784 2449783 2026-08-19T15:28:46Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1147.eqiad.wmnet with OS bookworm 2449784 wikitext text/x-wiki 0089v2whuuqacbc2k0ddwf83v37shb8 2449785 2449784 2026-08-19T15:29:54Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2449785 wikitext text/x-wiki saaw5fqqosh2obm3rr50o9wuarp2gk8 2449786 2449785 2026-08-19T15:30:21Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2449786 wikitext text/x-wiki pb78od31oeeca7wb0lx0jjk05s75vsv 2449787 2449786 2026-08-19T15:33:28Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqsin - cmooney@cumin1003" 2449787 wikitext text/x-wiki 2u8scu7uqlqlivlcrlh28xqnpwaozwd 2449788 2449787 2026-08-19T15:34:13Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327138|Enable redis lock manager everywhere (T366938)]] (duration: 08m 36s) 2449788 wikitext text/x-wiki 858k605trj6i3ig8oj6ieimt7aosvlh 2449789 2449788 2026-08-19T15:36:02Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqsin - cmooney@cumin1003" 2449789 wikitext text/x-wiki a2rata69mukb7e2h5v9nj1isya8s7ea 2449790 2449789 2026-08-19T15:36:05Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2449790 wikitext text/x-wiki osa180wg6i74hrzesvwy0wr0g49y3zg 2449791 2449790 2026-08-19T15:36:27Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2449791 wikitext text/x-wiki 140insyczs1uwzf2kad3nxbzhctv407 2449793 2449791 2026-08-19T15:38:45Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2449793 wikitext text/x-wiki 2bpwfu3pvasm7kw4gf4blhq7i8k085q 2449794 2449793 2026-08-19T15:42:07Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449794 wikitext text/x-wiki p61yilh0ibtsbukxqsexxeroaxb26xk 2449795 2449794 2026-08-19T15:42:18Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2449795 wikitext text/x-wiki 5366xzyh5ulcpub4zin7a139q55szri 2449796 2449795 2026-08-19T15:43:22Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449796 wikitext text/x-wiki khm8rh7x5cha9mt7yby0t8yvqskrrls 2449797 2449796 2026-08-19T15:44:27Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449797 wikitext text/x-wiki 9dy9768gai4r76p2t6cdiipwtywp9gz 2449798 2449797 2026-08-19T15:44:52Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2449798 wikitext text/x-wiki opvpxd7jmsx0bkpx2w7itf6uzyfa6ar 2449799 2449798 2026-08-19T15:45:10Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003" 2449799 wikitext text/x-wiki ku3761pv6l0lw8hr977j8853ak40eli 2449800 2449799 2026-08-19T15:48:55Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003" 2449800 wikitext text/x-wiki 4712z75228lesc3b7nngszr6y46ohna 2449801 2449800 2026-08-19T15:48:57Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2449801 wikitext text/x-wiki bpldzehts1ozzjnxfyfi8o2d0j7tj5m 2449802 2449801 2026-08-19T15:49:11Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2449802 wikitext text/x-wiki dosx0pskvmn2kjvekxnoiwqsak5pg7b 2449803 2449802 2026-08-19T15:51:34Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2449803 wikitext text/x-wiki s4mquurzuydm23o49llnpvo5f1hkkom 2449804 2449803 2026-08-19T15:52:28Z Stashbot 7414 urandom: beginning sessionstore Cassandra/JVM upgrade — T432944 2449804 wikitext text/x-wiki kso4kmvfqkhtt7dopvjds4vnxw9qgw2 2449805 2449804 2026-08-19T15:53:15Z Stashbot 7414 urandom: beginning sessionstore Cassandra/JVM upgrade — T435154 2449805 wikitext text/x-wiki pdqqwk9q1302ye40zbej2l5x6v6zxjf 2449806 2449805 2026-08-19T15:54:17Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore2004.codfw.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 2449806 wikitext text/x-wiki 6ni7j42x29mcjo4lslo947b1psfxwu3 2449807 2449806 2026-08-19T15:56:26Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore2004.codfw.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 2449807 wikitext text/x-wiki ozfw1lb12nz8gq88e0jjw1266iejxk2 2449808 2449807 2026-08-19T16:01:38Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore1004.eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 2449808 wikitext text/x-wiki 3vqkws5c9trrvt7q4xd9mrjv23acn2x 2449809 2449808 2026-08-19T16:03:31Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore1004.eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 2449809 wikitext text/x-wiki 6n821xbjynf9b2mxc2e348bjinxzqk7 2449810 2449809 2026-08-19T16:06:01Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2449810 wikitext text/x-wiki 9mgmgtytafrb0savb66ymler86n3te7 2449812 2449810 2026-08-19T16:06:44Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1327123|Revert^2 "Migrate database access to virtual domains" (T435305)]], [[gerrit:1327124|Pass the mapped domain of virtual-echo-shared to the push NameTableStores (T435305)]] 2449812 wikitext text/x-wiki 0kdiphicxbnw1wra16qjup9yfv0ac3i 2449813 2449812 2026-08-19T16:06:48Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2449813 wikitext text/x-wiki gaankj7arhuavba8w6aovkebc3gh27y 2449814 2449813 2026-08-19T16:06:58Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2449814 wikitext text/x-wiki 97o47xqalrgnewz3qmibqgb6aw56243 2449815 2449814 2026-08-19T16:09:00Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1327123|Revert^2 "Migrate database access to virtual domains" (T435305)]], [[gerrit:1327124|Pass the mapped domain of virtual-echo-shared to the push NameTableStores (T435305)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449815 wikitext text/x-wiki p1z0kyh69xuvwz1s3axapymagw1fq9p 2449816 2449815 2026-08-19T16:10:09Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Continuing with deployment 2449816 wikitext text/x-wiki k10k6ie3m15437jexzcr2w06chnd9y6 2449817 2449816 2026-08-19T16:11:43Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003" 2449817 wikitext text/x-wiki owscubsyqb00xodp4y92xv31fv1uf4i 2449818 2449817 2026-08-19T16:13:19Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore[2005-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 2449818 wikitext text/x-wiki m0f607bh5od4dh8um5klil7bhw7b65q 2449821 2449818 2026-08-19T16:14:26Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327123|Revert^2 "Migrate database access to virtual domains" (T435305)]], [[gerrit:1327124|Pass the mapped domain of virtual-echo-shared to the push NameTableStores (T435305)]] (duration: 07m 42s) 2449821 wikitext text/x-wiki bsf2e2lpp86xhfrh62t8zdfuydr78be 2449823 2449821 2026-08-19T16:14:49Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1326881|Echo: Start using virtual domains (T380385)]] 2449823 wikitext text/x-wiki 35pqllec3biofsthyhgbqyaxc5tge5i 2449824 2449823 2026-08-19T16:15:02Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003" 2449824 wikitext text/x-wiki 86lue7pm3r834nsjcu2onrcsu1rkxxy 2449825 2449824 2026-08-19T16:15:05Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2449825 wikitext text/x-wiki 9lmnrvr3t194g22htyj744q6sa186hc 2449827 2449825 2026-08-19T16:19:14Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1326881|Echo: Start using virtual domains (T380385)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449827 wikitext text/x-wiki 6dvimoatrc739v45lszwo08n5wxc7ea 2449828 2449827 2026-08-19T16:21:02Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore[2005-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 2449828 wikitext text/x-wiki b9j4n7srvxqi5obpd2r5olyvlru49h4 2449829 2449828 2026-08-19T16:21:50Z Stashbot 7414 urandom: Completed sessionstore Cassandra/JVM upgrade — T435154 2449829 wikitext text/x-wiki i4xklwb7grrx48ky6emf88nkc4yaj66 2449830 2449829 2026-08-19T16:23:08Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Continuing with deployment 2449830 wikitext text/x-wiki 25ldsi0wytotqz94tbq55z3m62pbq3d 2449831 2449830 2026-08-19T16:28:02Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1326881|Echo: Start using virtual domains (T380385)]] (duration: 13m 12s) 2449831 wikitext text/x-wiki lvrgnqmzdzkfrbwkmhccrgycc3b8t56 2449834 2449831 2026-08-19T16:40:53Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and A:durum 2449834 wikitext text/x-wiki ra6s3vxn6pe8dn0ev0dqsin5wg6nx5b 2449835 2449834 2026-08-19T16:41:06Z Stashbot 7414 cjd91: sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-eqiad' --task-id T434478 --reason '9.2.15 upgrade' 2449835 wikitext text/x-wiki grpi9puo26kg84hoj5hqpotbk9oty16 2449836 2449835 2026-08-19T16:41:16Z Stashbot 7414 cjd91: sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-codfw' --task-id T434478 --reason '9.2.15 upgrade' 2449836 wikitext text/x-wiki 51ik6ls6mfs4he5237i1hn32g8ryhjs 2449837 2449836 2026-08-19T16:41:21Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478) 2449837 wikitext text/x-wiki ccln800s7yap89p2smsbkq3hsexe99m 2449838 2449837 2026-08-19T16:41:28Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478) 2449838 wikitext text/x-wiki 5behwczf5othpzgkpvv6aym48nycoa1 2449839 2449838 2026-08-19T16:41:46Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting A:liberica-esams and A:liberica 2449839 wikitext text/x-wiki 2xltbc6jqr1hianq2hode2vwogvjd8j 2449841 2449839 2026-08-19T16:53:37Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-esams and A:liberica 2449841 wikitext text/x-wiki atgroctqf7u1xbf43m1qbj5j5f5mw48 2449842 2449841 2026-08-19T16:58:13Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2449842 wikitext text/x-wiki 3766j8nmdn7cko5xhzbki9jlh4vetbc 2449843 2449842 2026-08-19T17:00:21Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2449843 wikitext text/x-wiki 7ggmrx3jw7w4dsrmcpt7w0v1g4qvx71 2449846 2449843 2026-08-19T17:22:03Z Stashbot 7414 sbassett: Deployed security fix for T435210 (wmf.15) 2449846 wikitext text/x-wiki jz4vo3sczeqmjp1h72znxhirmn6ingb 2449847 2449846 2026-08-19T17:26:47Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2449847 wikitext text/x-wiki 2mt6rkummezqp1vy53mnw7bjnod7tgw 2449848 2449847 2026-08-19T17:29:45Z Stashbot 7414 sbassett: Deployed security fix for T435210 (wmf.16) 2449848 wikitext text/x-wiki ifoho6wvqihkeau6wmf9lwoj0z70ov0 2449849 2449848 2026-08-19T17:32:22Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on GTT VPLS - cmooney@cumin1003" 2449849 wikitext text/x-wiki 9p0yj3gfr2a10fxup1ba5ee5zo3ylbg 2449851 2449849 2026-08-19T17:37:48Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --verbose --scoreLessThan=0.7 --exceptDatasetChecksums=T434319-enwiki-models.txt # T434319 2449851 wikitext text/x-wiki 2e9o3qicu9rki0f1dqadg3vdyumucae 2449852 2449851 2026-08-19T17:37:59Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on GTT VPLS - cmooney@cumin1003" 2449852 wikitext text/x-wiki iuuf0g50va0z4jgkck366kdzyymn2wi 2449853 2449852 2026-08-19T17:38:02Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2449853 wikitext text/x-wiki gv2rgrc7s72jlj1fpipad5014cmqbml 2449854 2449853 2026-08-19T17:40:31Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.283.0" for 3 host(s) 2449854 wikitext text/x-wiki 2oe660goe5riibqrmzwi6fkecekrwji 2449855 2449854 2026-08-19T17:40:36Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2449855 wikitext text/x-wiki pdnbm2d91shpv6mh4re0l2vrrv9rzep 2449856 2449855 2026-08-19T17:41:48Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum and A:durum 2449856 wikitext text/x-wiki dhzryyrhitw05uze2llg8b6nwtinoot 2449858 2449856 2026-08-19T17:42:57Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.283.0" completed for 3 hosts 2449858 wikitext text/x-wiki 2tzzfx9qj6qyjkglfsu7tay4lr6xl06 2449861 2449858 2026-08-19T17:50:43Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2449861 wikitext text/x-wiki fbk60ckawlb872bqnn692wj3gf271go 2449862 2449861 2026-08-19T17:56:52Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478) 2449862 wikitext text/x-wiki noffrsol20bt3nqpy8awbcxjl3tev8q 2449863 2449862 2026-08-19T18:03:34Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478) 2449863 wikitext text/x-wiki idjafdqd9dyielqe3k9h8w4sws5r9c2 2449865 2449863 2026-08-19T18:09:26Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1319920|Allow setting a separate thumbUrl in production (T427465)]] 2449865 wikitext text/x-wiki ndan7wx0tlwo0ywidzqgjvf84by21c0 2449867 2449865 2026-08-19T18:24:03Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2449867 wikitext text/x-wiki 47cgc4ezx086b03ouy94lyzt9kdsocy 2449868 2449867 2026-08-19T18:26:11Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2449868 wikitext text/x-wiki 0l9twptuhzo0qgbucg8zme8tpxvl5of 2449869 2449868 2026-08-19T18:28:43Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1319920|Allow setting a separate thumbUrl in production (T427465)]], [[gerrit:1327167|Fix wmgThumbUrl config (T427465)]] 2449869 wikitext text/x-wiki kb7ar30nq13y1ognww9i5gvvniynqa2 2449870 2449869 2026-08-19T18:31:00Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1319920|Allow setting a separate thumbUrl in production (T427465)]], [[gerrit:1327167|Fix wmgThumbUrl config (T427465)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449870 wikitext text/x-wiki m4lolbhos535382mlpg8jvst72n212k 2449880 2449870 2026-08-19T18:43:21Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2449880 wikitext text/x-wiki inqmokf10zrgkfiihxs9adwz6km3vaq 2449881 2449880 2026-08-19T18:47:36Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319920|Allow setting a separate thumbUrl in production (T427465)]], [[gerrit:1327167|Fix wmgThumbUrl config (T427465)]] (duration: 18m 53s) 2449881 wikitext text/x-wiki btb0pks9k4bv3nhw9cr31hc9o5et8bg 2449883 2449881 2026-08-19T19:43:47Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance 2449883 wikitext text/x-wiki 8b94lgkp9ocghrilii3sve5culgiwfw 2449885 2449883 2026-08-19T19:56:32Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bookworm 2449885 wikitext text/x-wiki 0y9mt5y869766x2gycnulb988wzapss 2449886 2449885 2026-08-19T19:56:53Z Stashbot 7414 eevans@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cassandra-dev2001.codfw.wmnet with OS bookworm 2449886 wikitext text/x-wiki 5nww5rbxkkkty27o783amc95m16hlhq 2449887 2449886 2026-08-19T19:59:45Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bookworm 2449887 wikitext text/x-wiki 50rwa1m3ezk77blfqdwsxaqpuo2cfqi 2449888 2449887 2026-08-19T20:06:37Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1325878|[arwiki] Enable restricted user page editing and grant edit permissions (T434878)]] 2449888 wikitext text/x-wiki 6fguv0hf2stqujf0uyy30wwsvsd4ocb 2449889 2449888 2026-08-19T20:08:59Z Stashbot 7414 catrope@deploy1003: catrope, gergesshamon: Backport for [[gerrit:1325878|[arwiki] Enable restricted user page editing and grant edit permissions (T434878)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449889 wikitext text/x-wiki im1xs9uim3r58bis9d953cevpkmge6o 2449890 2449889 2026-08-19T20:11:11Z Stashbot 7414 catrope@deploy1003: catrope, gergesshamon: Continuing with deployment 2449890 wikitext text/x-wiki iv9soiwh55fervjj2pzbf3sd9lt7kp6 2449891 2449890 2026-08-19T20:15:23Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325878|[arwiki] Enable restricted user page editing and grant edit permissions (T434878)]] (duration: 08m 46s) 2449891 wikitext text/x-wiki bycnxg7b30ay2hnfzw54wde7p8n01rm 2449892 2449891 2026-08-19T20:17:18Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage 2449892 wikitext text/x-wiki 84jgoqyfxjfiyvt8dqa53byktaemri3 2449894 2449892 2026-08-19T20:23:38Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage 2449894 wikitext text/x-wiki dznqdzcghfc7srcqspzbvpzjvat3kmz 2449895 2449894 2026-08-19T20:24:27Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1327128|Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags (T432547)]] 2449895 wikitext text/x-wiki 6xoxqsk139foj5p5pzihcwkfp17furl 2449896 2449895 2026-08-19T20:26:56Z Stashbot 7414 catrope@deploy1003: catrope, arlolra: Backport for [[gerrit:1327128|Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags (T432547)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449896 wikitext text/x-wiki 1034upuh1s0lldkzb1g3mz8f10frl22 2449897 2449896 2026-08-19T20:27:45Z Stashbot 7414 catrope@deploy1003: catrope, arlolra: Continuing with deployment 2449897 wikitext text/x-wiki j8ayaqt43a9fw4l8bmaxvecd7y6rmsf 2449898 2449897 2026-08-19T20:31:57Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327128|Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags (T432547)]] (duration: 07m 30s) 2449898 wikitext text/x-wiki 9bi0veif2emile3rlvssmjlwnkt47w6 2449899 2449898 2026-08-19T20:35:48Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1327178|AccountRecovery: Notify the email address of the on file of the request (T425799)]] 2449899 wikitext text/x-wiki 8opoekh5q11dvvf6qv8isdcww6jebay 2449900 2449899 2026-08-19T20:55:36Z Stashbot 7414 catrope@deploy1003: catrope: Backport for [[gerrit:1327178|AccountRecovery: Notify the email address of the on file of the request (T425799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449900 wikitext text/x-wiki 8tmef5evbctnozp9j8fyxbj7bmqlu53 2449901 2449900 2026-08-19T21:09:57Z Stashbot 7414 catrope@deploy1003: catrope: Continuing with deployment 2449901 wikitext text/x-wiki k8cliat1n30idgvl3v84xqtdoo34xq0 2449902 2449901 2026-08-19T21:14:00Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2001.codfw.wmnet with OS bookworm 2449902 wikitext text/x-wiki puydrtlv6u31ltlwv147h5n6ijuqjno 2449903 2449902 2026-08-19T21:22:51Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327178|AccountRecovery: Notify the email address of the on file of the request (T425799)]] (duration: 47m 02s) 2449903 wikitext text/x-wiki t58sdvqyl0yrwhbij40z9ok3v22ciuu 2449904 2449903 2026-08-19T21:28:42Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reboot-single for host cassandra-dev2001.codfw.wmnet 2449904 wikitext text/x-wiki ctbbxzkd5y6rq4jg7k3zhiv3bqwh7v8 2449905 2449904 2026-08-19T21:34:25Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cassandra-dev2001.codfw.wmnet 2449905 wikitext text/x-wiki oyebtlx1ut6suo0wkxo0yfcp9ao9ukk 2449906 2449905 2026-08-19T22:00:56Z Stashbot 7414 krinkle@deploy1003: Locking from deployment [ALL REPOSITORIES]: awaiting CI 2449906 wikitext text/x-wiki tuwh6730gbfm5izwvrd1wmcvgalnai2 2449907 2449906 2026-08-19T22:00:58Z Stashbot 7414 krinkle@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: awaiting CI (duration: 00m 01s) 2449907 wikitext text/x-wiki 3r6mghiay39qbw5pzvx651s4l9qg99w 2449908 2449907 2026-08-19T22:01:02Z Stashbot 7414 krinkle@deploy1003: Locking from deployment [ALL REPOSITORIES]: awaiting CI 2449908 wikitext text/x-wiki jk6p4j2717pbb9fk2wgs3deox9kgllt 2449909 2449908 2026-08-19T22:04:09Z Stashbot 7414 krinkle@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: awaiting CI (duration: 03m 06s) 2449909 wikitext text/x-wiki tpcrr6u1didcw5t0368b7qeh8a4xm1j 2449910 2449909 2026-08-19T22:04:52Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1327162|Article: Split subjectpageheader by model and disable for wikitext]], [[gerrit:1327169|Make uppercase greek letters normal (non-italic) font (T434686 T434428)]], [[gerrit:1327176|Skin: Avoid DB lookup for pagecategorieslink message (T347123)]] 2449910 wikitext text/x-wiki 7sjmkkahc17od1x6upgp1asmmspjmyk 2449915 2449910 2026-08-19T22:25:03Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1327162|Article: Split subjectpageheader by model and disable for wikitext]], [[gerrit:1327169|Make uppercase greek letters normal (non-italic) font (T434686 T434428)]], [[gerrit:1327176|Skin: Avoid DB lookup for pagecategorieslink message (T347123)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449915 wikitext text/x-wiki aibo1h80undzthadpuv78ze1c4mzh0j 2449916 2449915 2026-08-19T22:29:26Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2449916 wikitext text/x-wiki bbwc3b279cqpi1si1z3fxrzm7dt3rof 2449917 2449916 2026-08-19T22:42:43Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327162|Article: Split subjectpageheader by model and disable for wikitext]], [[gerrit:1327169|Make uppercase greek letters normal (non-italic) font (T434686 T434428)]], [[gerrit:1327176|Skin: Avoid DB lookup for pagecategorieslink message (T347123)]] (duration: 37m 52s) 2449917 wikitext text/x-wiki 2b3vo7p6ia73nwzh8zfcuwh90ctf6lb 2449918 2449917 2026-08-19T22:46:43Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1327201|Enable ReadingLists for all logged in users on test wiki (T435258)]] 2449918 wikitext text/x-wiki ekadl0ov11e08yqfsm53y5bc2u2y9ph 2449919 2449918 2026-08-19T22:49:15Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1327201|Enable ReadingLists for all logged in users on test wiki (T435258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449919 wikitext text/x-wiki 4lnrsqadjtfj32qsi82fx9jcs45klge 2449920 2449919 2026-08-19T22:50:49Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2449920 wikitext text/x-wiki lxe6lfil6g13clrrldgonaey41g80ux 2449921 2449920 2026-08-19T22:58:03Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327201|Enable ReadingLists for all logged in users on test wiki (T435258)]] (duration: 11m 20s) 2449921 wikitext text/x-wiki 7yj3s1uu37kqc0eiu7wj25hu3b0fxbk 2449922 2449921 2026-08-19T23:08:34Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1327197|Enable thumb.wikimedia.org on mediawiki.org (T427465)]] 2449922 wikitext text/x-wiki 5ml5q8ftfkihdvmi6j37zwozlxnwqna 2449923 2449922 2026-08-19T23:09:02Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2449923 wikitext text/x-wiki tgr47lr4qo17eu7vyezs29d2a414rl9 2449924 2449923 2026-08-19T23:09:08Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2449924 wikitext text/x-wiki 55ddwg1fn6kt31er3n12nvrxkkntxu0 2449925 2449924 2026-08-19T23:09:26Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2449925 wikitext text/x-wiki d7xqpp13dyhvif3475woybypvx6ej55 2449926 2449925 2026-08-19T23:10:45Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2449926 wikitext text/x-wiki 2vro3joxyieikeatape6m2xyaawup9t 2449927 2449926 2026-08-19T23:10:59Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1327197|Enable thumb.wikimedia.org on mediawiki.org (T427465)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2449927 wikitext text/x-wiki ec1rg1p3fyen4ny1423mnp2rn3m7o6y 2449928 2449927 2026-08-19T23:15:13Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2449928 wikitext text/x-wiki pyedos3pwoz2chioesa4e7jz8f9axhp 2449929 2449928 2026-08-19T23:16:10Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2449929 wikitext text/x-wiki gmfpbsj15ck3wweskxok2rq97050ew9 2449931 2449929 2026-08-19T23:18:44Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2449931 wikitext text/x-wiki t7mfuozgeo7dm9q4oxo9o54tfibdrf0 2449932 2449931 2026-08-19T23:19:24Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327197|Enable thumb.wikimedia.org on mediawiki.org (T427465)]] (duration: 10m 50s) 2449932 wikitext text/x-wiki 8kgakx93s0dl6zeb3cllwga95qrjq3h 2449938 2449932 2026-08-20T02:00:44Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2449938 wikitext text/x-wiki pvsvjvedvsy339qeziy0c6fw49w5ylh 2449939 2449938 2026-08-20T02:08:34Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 49s) 2449939 wikitext text/x-wiki rulctzzmkorrg3b2m6tr3elpkbidlcp 2449947 2449939 2026-08-20T06:18:38Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@6b9b6ff]: deploy phab1004 for to pick up config values for T435087 2449947 wikitext text/x-wiki hq098d18rid59ukqh8olsvzwqlzozt9 2449948 2449947 2026-08-20T06:20:24Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@6b9b6ff]: deploy phab1004 for to pick up config values for T435087 (duration: 01m 46s) 2449948 wikitext text/x-wiki 8mdc78ux9ifwb06y91mcu75u6jbwo35 2449949 2449948 2026-08-20T06:21:40Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@6b9b6ff]: deploy phab1005 for T435087 2449949 wikitext text/x-wiki 4t4wa7podkycvzwmon9xe1rt39ek6qc 2449950 2449949 2026-08-20T06:22:20Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@6b9b6ff]: deploy phab1005 for T435087 (duration: 00m 39s) 2449950 wikitext text/x-wiki 0mu1vcbs0f0p7txpruxrhpuy53t4r4r 2449951 2449950 2026-08-20T06:25:29Z Stashbot 7414 aokoth@dns1004: START - running authdns-update 2449951 wikitext text/x-wiki 2j26ytd15du349p8uekekj2gm2e3v0k 2449952 2449951 2026-08-20T06:27:43Z Stashbot 7414 aokoth@dns1004: END - running authdns-update 2449952 wikitext text/x-wiki 7mz76pcwh07nw360w0ge1v4ph7hfer7 2449956 2449952 2026-08-20T08:18:18Z Stashbot 7414 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.16 refs T430835 2449956 wikitext text/x-wiki ae586qg367dcs946i0gnnutlbnzqy97 2449957 2449956 2026-08-20T08:21:38Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2449957 wikitext text/x-wiki fk01enb4vg4yxis8qwe4zt29orj7t7z 2449958 2449957 2026-08-20T08:23:52Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2449958 wikitext text/x-wiki ktr6od9sopp5qxf0275xg2aqgswtu37 2449960 2449958 2026-08-20T09:04:23Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2449960 wikitext text/x-wiki mbnvhzjya1mjvuoxqii1357hziev0l0 2449961 2449960 2026-08-20T09:04:54Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2449961 wikitext text/x-wiki bf1qiddb0puev44g16m0hvndwx91zoa 2449962 2449961 2026-08-20T09:07:56Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2449962 wikitext text/x-wiki 0xekoy9s7xisojo8w48lc7hixvypk81 2449963 2449962 2026-08-20T09:08:17Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2449963 wikitext text/x-wiki 8s2ocl7t3rb0vrujqnlacj76bcerza7 2449964 2449963 2026-08-20T09:08:25Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2449964 wikitext text/x-wiki a8td10flqu6ljkn2eoccdgquakk97rz 2449965 2449964 2026-08-20T09:08:45Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2449965 wikitext text/x-wiki eothtvk9mqwsnya8gzd2evtn3hx1umj 2449966 2449965 2026-08-20T09:20:41Z Stashbot 7414 moritzm: imported squid 7.6-2.1for trixie-wikimedia/main T427282 2449966 wikitext text/x-wiki to429j1p5nz8sxx1x1hgf114i1pb641 2449967 2449966 2026-08-20T09:22:20Z Stashbot 7414 jelto: update cert-manager to 1.19.6 on wikikube staging-eqiad - T427402 2449967 wikitext text/x-wiki 3irrpmb7sxx9s36z17qg1wf35562ine 2449968 2449967 2026-08-20T09:22:53Z Stashbot 7414 jelto@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2449968 wikitext text/x-wiki szmj25p3i8a0jfc7xnp3cu2xtbc6rry 2449969 2449968 2026-08-20T09:23:27Z Stashbot 7414 jelto@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2449969 wikitext text/x-wiki a12zd7uerh037w2f40k4gzcaxqe9wtm 2449978 2449969 2026-08-20T10:13:20Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1901.eqiad.wmnet 2449978 wikitext text/x-wiki shdxk07wwh8uzzudueruj1s3stap1mw 2449979 2449978 2026-08-20T10:16:45Z Stashbot 7414 blake@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2449979 wikitext text/x-wiki 6drbzrcoljp27sej9fokola06zxjnxx 2449980 2449979 2026-08-20T10:17:48Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2449980 wikitext text/x-wiki 20jh5gmgw9v35wpb2zjws8dba3uobz3 2449981 2449980 2026-08-20T10:18:48Z Stashbot 7414 blake@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2449981 wikitext text/x-wiki lrhhiokpbf9a6hvk996sgrx9xj0g0sg 2449982 2449981 2026-08-20T10:20:37Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2449982 wikitext text/x-wiki 807ak6ekqkclcppwuj1upu3600utx71 2449983 2449982 2026-08-20T10:20:39Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts db1901.eqiad.wmnet 2449983 wikitext text/x-wiki jgcwubwy0zuwzrcpe8aks86843i1rmf 2449985 2449983 2026-08-20T10:23:34Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host stat1010.eqiad.wmnet with OS bookworm 2449985 wikitext text/x-wiki 3wb231w8xlqu5a869z9g5hqhw9fyfzw 2449987 2449985 2026-08-20T10:27:15Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.ganeti.makevm for new host db1901.eqiad.wmnet 2449987 wikitext text/x-wiki 0tnecr8dgmqkbuqwbvg5gexlbd84dnx 2449988 2449987 2026-08-20T10:27:18Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2449988 wikitext text/x-wiki tbxxsoehoqvp67gqeb11978fob9uhcq 2449990 2449988 2026-08-20T10:31:53Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003" 2449990 wikitext text/x-wiki 7z45kewulixvjmauonfk35g4ph6ms68 2449991 2449990 2026-08-20T10:31:57Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003" 2449991 wikitext text/x-wiki tcct2autuc08yle4ot6wdhv8p5p2x1u 2449992 2449991 2026-08-20T10:32:00Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2449992 wikitext text/x-wiki i7u7noxcdkfpmf09ucabx2oocp13uui 2449993 2449992 2026-08-20T10:32:02Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1901.eqiad.wmnet on all recursors 2449993 wikitext text/x-wiki 9wx22hktjrs37ryvs8qepxoh2f6hjll 2449994 2449993 2026-08-20T10:32:04Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1901.eqiad.wmnet on all recursors 2449994 wikitext text/x-wiki ekot1p2d7mbsfzyr3cpnri44lrihdit 2449995 2449994 2026-08-20T10:32:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1901.eqiad.wmnet - fceratto@cumin1003" 2449995 wikitext text/x-wiki esnbu6k8j5prfjpn1jart4nuf7uu54m 2449996 2449995 2026-08-20T10:32:36Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1901.eqiad.wmnet - fceratto@cumin1003" 2449996 wikitext text/x-wiki p219fzueyc279xttwrcd3ppxz2n5wj4 2449997 2449996 2026-08-20T10:35:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1901.eqiad.wmnet with OS trixie 2449997 wikitext text/x-wiki 7zt2ucz8tt7qa8ah0c8zdx0wn8hbznn 2450000 2449997 2026-08-20T10:47:41Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1901.eqiad.wmnet with reason: host reimage 2450000 wikitext text/x-wiki ieh79spwhc7spkm6t327zav7sehwa25 2450001 2450000 2026-08-20T10:47:53Z Stashbot 7414 tappof: bump space for prometheus k8s-dse in eqiad 2450001 wikitext text/x-wiki 5xf671nb2pqxqxakki1hvi0kar84su4 2450002 2450001 2026-08-20T10:47:59Z Stashbot 7414 tappof: bump space for prometheus k8s-aux in codfw 2450002 wikitext text/x-wiki o6xk2nl8zqoon677ln0j28dv18w1bom 2450003 2450002 2026-08-20T10:53:48Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1901.eqiad.wmnet with reason: host reimage 2450003 wikitext text/x-wiki esdusadkasf306n2v2ejpdufub7n2pi 2450005 2450003 2026-08-20T11:07:38Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1901.eqiad.wmnet with OS trixie 2450005 wikitext text/x-wiki j52dcswkwh5wntvj8nfh0bxekvke9lk 2450006 2450005 2026-08-20T11:07:40Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host db1901.eqiad.wmnet 2450006 wikitext text/x-wiki pmjya8ovy4syemworjdpl316bsd41v4 2450007 2450006 2026-08-20T11:24:12Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host stat1010.eqiad.wmnet with OS bookworm 2450007 wikitext text/x-wiki g3oyqgqoztiqc3770fjdkxfmh2zvcr8 2450008 2450007 2026-08-20T11:26:37Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1327518|Enable thumb.wikimedia.org on cswiki and fawiki (T427465)]] 2450008 wikitext text/x-wiki k0hyde67r62a64fuii6no49ga9vjsuk 2450009 2450008 2026-08-20T11:30:52Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1327518|Enable thumb.wikimedia.org on cswiki and fawiki (T427465)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450009 wikitext text/x-wiki 8xgi2v2mycv2sdvr5u1bhjwdjcgz6vb 2450010 2450009 2026-08-20T11:33:01Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2450010 wikitext text/x-wiki 3ev2shehofyla3tmzyfhl3hsvne64ih 2450011 2450010 2026-08-20T11:33:47Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host stat1010.eqiad.wmnet with OS bookworm 2450011 wikitext text/x-wiki 2e2gbtnzu2hbbgvgt60t2ddp3frk1dk 2450012 2450011 2026-08-20T11:35:05Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1902.eqiad.wmnet 2450012 wikitext text/x-wiki lw4uzztnii0fjidre1f4rkajat0fyqq 2450013 2450012 2026-08-20T11:37:17Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327518|Enable thumb.wikimedia.org on cswiki and fawiki (T427465)]] (duration: 10m 40s) 2450013 wikitext text/x-wiki 0rigzy6teo8y4ofojpti8mgvibept5l 2450014 2450013 2026-08-20T11:39:39Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2450014 wikitext text/x-wiki nr3np5ejle0v8zu0tfclel7zm0fozv2 2450018 2450014 2026-08-20T11:46:44Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2450018 wikitext text/x-wiki 0rnj9t0r4e6ot7hh3dofyjgri3i2ilh 2450019 2450018 2026-08-20T11:48:40Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2450019 wikitext text/x-wiki qnq1mf6rw9oc8pr6asnvxij6jy1a62d 2450020 2450019 2026-08-20T11:49:06Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2450020 wikitext text/x-wiki jmytf1t5fampdthw77tqj05k8tmouuj 2450021 2450020 2026-08-20T11:51:40Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2450021 wikitext text/x-wiki incjh1ke54andso6tdcm2bimv9vlel0 2450023 2450021 2026-08-20T11:57:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1902.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2450023 wikitext text/x-wiki 4rg2sehjwjtj6fo6nh689114f4mia2x 2450024 2450023 2026-08-20T11:57:40Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1902.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2450024 wikitext text/x-wiki 52h3w8ei68ishwxvrno4sd0j9fxa0oq 2450025 2450024 2026-08-20T11:57:42Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2450025 wikitext text/x-wiki je3ywci1m5gy3xwkx59nplvsh7ay8uf 2450026 2450025 2026-08-20T11:57:45Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1902.eqiad.wmnet 2450026 wikitext text/x-wiki fuaii66xg4plx07x6k24d2vhdm3sj8a 0 13333 2449833 2401121 2026-08-19T16:29:14Z AKlapper (WMF) 421 Fix 404 URL 2449833 wikitext text/x-wiki 4r2n81kopqi3wruypkelii30exufc0d 0 17290 2449651 2449258 2026-08-19T12:30:28Z Stashbot 7414 James_F: Updating development images on contint primary for “fundraising: Drop libc-client-dev from the bookworm PHP 8.2 image” 2449651 wikitext text/x-wiki f5m7mh5n0mj7lmhcmesu5bla28ydm50 2449763 2449651 2026-08-19T14:48:09Z Stashbot 7414 hashar: integration: updated castor save job to have rsync emit statistics | https://gerrit.wikimedia.org/r/c/integration/config/+/1321049 | T432685 2449763 wikitext text/x-wiki onuahsiin488ytboxtjy7bgpzpzto2z 2449959 2449763 2026-08-20T08:54:07Z Stashbot 7414 hashar: deployment-prep: hard reboot deployment-cache-test-08 # T435421 2449959 wikitext text/x-wiki cpyppqrv0waq2t5su7yls64oosgmuv6 0 22139 2449864 2406534 2026-08-19T18:06:38Z XCollazo-WMF 31846 Add Spark 3.5.8 to list 2449864 wikitext text/x-wiki atrq4uxrmwpoh5n604s9gipqtiryb14 0 22777 2449878 2448211 2026-08-19T18:36:38Z Novem Linguae 23682 /* Architecture Overview */ wl 2449878 wikitext text/x-wiki 67b95z1w8tfv59bjtn7l2q4cyv1w0kx 0 26586 2449832 2449300 2026-08-19T16:28:05Z AKlapper (WMF) 421 Fix 404 URL 2449832 wikitext text/x-wiki qkepmk05szjfgvo2nzsamojgagfbkl3 0 46350 2449941 2420846 2026-08-20T04:39:46Z WǂMǂFOffice 58783 /* */ 2449941 wikitext text/x-wiki 99mmjfdqdfb1gclhmdnz7hikxw2825f 2449943 2449941 2026-08-20T04:40:13Z JJMC89 7474 Reverted edit by [[Special:Contributions/WǂMǂFOffice|WǂMǂFOffice]] ([[User talk:WǂMǂFOffice|talk]]) to last revision by [[User:Ricert511|Ricert511]] 2420846 wikitext text/x-wiki celoh1n3ol67y2xza67z813twje8de9 2449945 2449943 2026-08-20T04:40:56Z WǂMǂFOffice 58783 2449945 wikitext text/x-wiki 99mmjfdqdfb1gclhmdnz7hikxw2825f 2449946 2449945 2026-08-20T04:42:49Z JJMC89 7474 Reverted edit by [[Special:Contributions/WǂMǂFOffice|WǂMǂFOffice]] ([[User talk:WǂMǂFOffice|talk]]) to last revision by [[User:JJMC89|JJMC89]] 2420846 wikitext text/x-wiki celoh1n3ol67y2xza67z813twje8de9 0 84340 2449877 2445217 2026-08-19T18:36:16Z Novem Linguae 23682 /* Infrastructure */ wl 2449877 wikitext text/x-wiki 94q08cx54b1ngal3sbclvfpltcus6ro 0 441014 2449736 2445248 2026-08-19T14:14:53Z SFrench-WMF 37362 /* Switchover */ Clarify what's essential in the replication host switch patch 2449736 wikitext text/x-wiki brr5frpmzadlectv0oj07rjjrz53pac 2449845 2449736 2026-08-19T17:19:42Z SFrench-WMF 37362 /* Primary cluster switchover */ Add reference to T435103 in intro 2449845 wikitext text/x-wiki 0yft8enlmic7gwlcjvtz5h9znqc89mv 498 443949 2449911 2448825 2026-08-19T22:11:58Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy a7ad288 (Update all SPARQL unit tests targeting Q41960 to use P10241 instead of P31) 2449911 wikitext text/x-wiki pcutusz81ydmbywyswj67lhtmy11f82 2449912 2449911 2026-08-19T22:11:59Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 1cd9f21 (Switch test fixtures from red pandas to lighthouses) 2449912 wikitext text/x-wiki 83w34lwtd0hbxnbckvq9kjkkclnw8nf 2449913 2449912 2026-08-19T22:12:01Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 59ccbe6 (Refactor: extract _build_drilldown_query helper) 2449913 wikitext text/x-wiki i7f3ec620mcfeb2hfquu1rtnbyh7cwc 2449914 2449913 2026-08-19T22:12:04Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy ef9b3fc (Wrap drilldown queries in subquery for label resolution) 2449914 wikitext text/x-wiki acmjjcc1h3az220rur3krwn0ah502f3 0 444523 2449930 2406539 2026-08-19T23:17:06Z Neil Shah-Quinn (WMF) 36601 Stat1008 has been updated to Debian Bookworm 2449930 wikitext text/x-wiki 32by01171jnonrltc48kwsiea49pj2s 3 444547 2449648 2449426 2026-08-19T12:13:42Z Taavi 13997 /* Question about block */ Reply 2449648 wikitext text/x-wiki 8b8mou9qwdup13zwwyrleek9uzygb1s 2449671 2449648 2026-08-19T12:56:33Z Sean.hoyland 37387 /* Question about block */ 2449671 wikitext text/x-wiki sf034dxqptb6o3cx88ojlnny358uptu 0 444747 2449970 2331114 2026-08-20T09:44:13Z Victus1144 58789 safisafi 2449970 wikitext text/x-wiki d6vxdq6flf6zj5zxjd4czfrxmaiupbw 2449971 2449970 2026-08-20T09:44:43Z Quinlan83 44942 Reverted edit by [[Special:Contributions/Victus1144|Victus1144]] ([[User talk:Victus1144|talk]]) to last revision by [[User:Novem Linguae|Novem Linguae]] 2331114 wikitext text/x-wiki tfs926l23k8vtkzx7wa97rn2059vywe 2449972 2449971 2026-08-20T09:47:08Z Victus1144 58789 2449972 wikitext text/x-wiki aked1yv166ut26ljzo52noq0y0724h2 2449973 2449972 2026-08-20T09:48:25Z Victus1144 58789 dasdasd 2449973 wikitext text/x-wiki 9q96jtp8eo3d890g77z2q7veupx5sbx 2449974 2449973 2026-08-20T09:48:39Z Quinlan83 44942 Reverted edits by [[Special:Contributions/Victus1144|Victus1144]] ([[User talk:Victus1144|talk]]) to last revision by [[User:Quinlan83|Quinlan83]] 2331114 wikitext text/x-wiki tfs926l23k8vtkzx7wa97rn2059vywe 0 445686 2449935 2425844 2026-08-19T23:42:31Z BryanDavis 1604 /* Adding tests */ Add some details about suites and a note about testwiki being a little weird 2449935 wikitext text/x-wiki bayck489ltqx3hdmra8askq55a2ia1h 0 446443 2449752 2250370 2026-08-19T14:31:42Z BKing (WMF) 29017 remove reference to OS 2449752 wikitext text/x-wiki lzedlj0ll6a5ts665tbbolvjgk1phza 0 448438 2449820 1925285 2026-08-19T16:13:46Z AKlapper (WMF) 421 Remove dead link to Differential 2449820 wikitext text/x-wiki 5sq4qpbz2kw9m404hcocwejyo3wm12v 0 449160 2449936 2449415 2026-08-20T00:02:14Z Dexbot 30554 Bot: Updating the report 2449936 wikitext text/x-wiki ijv3ggl2j93ntho5v93fy2ybhrpw0f6 498 450970 2449859 2379436 2026-08-19T17:46:52Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.delete_project for project onfire in eqiad1 2449859 wikitext text/x-wiki sepkmt355zp5jnsy2wj7eg7vbs7i5qo 2449860 2449859 2026-08-19T17:49:29Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project onfire in eqiad1 2449860 wikitext text/x-wiki q1s0xludeyu99uq7l5kdjd3ype9ff9d 0 451170 2449874 2043990 2026-08-19T18:34:57Z Novem Linguae 23682 /* 1. Executive Summary */ wl 2449874 wikitext text/x-wiki ou7xcocr2zvsh0ibkh778ch1n0lmz72 0 451360 2449893 2414323 2026-08-19T20:19:46Z MPopov (WMF) 5891 Added question about GrowthBook 2449893 wikitext text/x-wiki 1vb4i4ws8iq1a6k4t3h8qs5fl1y0qaa 498 451627 2449792 2196777 2026-08-19T15:37:25Z Stashbot 7414 James_F: Shut down all the instances, not used since we moved to Catalyst. 2449792 wikitext text/x-wiki do5eo8h7cusnzljyncdos1hwgap2mg7 3 453751 2449649 2449399 2026-08-19T12:21:35Z StrikerBot 8475 Welcome to Toolforge! 2449649 wikitext text/x-wiki d2yozoq2y39nx9zc6j9m3m122781gxs 0 455674 2449953 2227463 2026-08-20T06:36:46Z AKlapper (WMF) 421 {{outdated}} 2449953 wikitext text/x-wiki 0k0umzrk5wcuad8txj6twsogiqipmvk 2 455839 2449954 2436679 2026-08-20T07:09:12Z Meno25 1446 fix 2449954 wikitext text/x-wiki pfiknm5jbdn6swutnnkpq04rmeqt59z 3 455840 2449955 2436680 2026-08-20T07:09:18Z Meno25 1446 fix 2449955 wikitext text/x-wiki di6dc1hi375rekjkf3vlpmpqeylvbhw 0 455886 2449850 2333167 2026-08-19T17:36:53Z BKing (WMF) 29017 2449850 wikitext text/x-wiki t4gdws8y3n16wuc4eo3sipqj0gpdf9a 12 456551 2449650 2359256 2026-08-19T12:28:58Z Taavi-WMF 41365 new features 2449650 wikitext text/x-wiki g90ixq4rm5cqtusptfdu0jbeiaz1ork 0 459290 2449766 2446181 2026-08-19T14:54:20Z KCVelaga (WMF) 38150 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2449766 wikitext text/x-wiki ayq92ra5meezojalw17sv67lhcqzkcn 0 459699 2449933 2449428 2026-08-19T23:36:31Z Stashbot 7414 larssandergreen: civicrm upgraded from 4f83dccd to ba2bc5f4 2449933 wikitext text/x-wiki kepktkwuqtcw9c91usjm3xi7e9cx9sq 2449934 2449933 2026-08-19T23:40:21Z Stashbot 7414 larssandergreen: tools upgraded from 9f2c4da9 to 41162cd2 2449934 wikitext text/x-wiki 0m6n88h3veoso6luzgt807x21scklk9 2449937 2449934 2026-08-20T01:04:44Z Stashbot 7414 larssandergreen: civicrm upgraded from ba2bc5f4 to 225dc534 2449937 wikitext text/x-wiki iy2wbyk3dwmlsdsizvaucw7erly0398 2449940 2449937 2026-08-20T04:18:27Z Stashbot 7414 ejegg: donorwiki upgraded from a06ae998 to c56b3ad7 2449940 wikitext text/x-wiki bsgkplbrkb95vdg9lufkz53mpbfwf8b 0 459901 2449682 2388885 2026-08-19T13:04:06Z JMeybohm (WMF) 16709 Initial version 2449682 wikitext text/x-wiki gdgwh14w2lml5nsttmsg0a8jiibpxsl 0 460142 2449866 2432251 2026-08-19T18:22:58Z Novem Linguae 23682 /* Response */ ce 2449866 wikitext text/x-wiki 6oasob5d2ntssmdvott4t5t7hf634sv 2449873 2449866 2026-08-19T18:34:39Z Novem Linguae 23682 /* Impact */ wl 2449873 wikitext text/x-wiki dvdgivi4926iq7k99dhwyzihexg1h8j 0 460366 2449819 2444763 2026-08-19T16:13:20Z AWesterinen-WMF 54189 /* Breadth First Search */ 2449819 wikitext text/x-wiki 2rlfudnxv3noyjkb39papgbyfaki0gy 2449822 2449819 2026-08-19T16:14:30Z AWesterinen-WMF 54189 /* Breadth First Search */ 2449822 wikitext text/x-wiki suysy61agwx8xn5vu8n3axaxjpbybt5 2 460540 2449731 2449203 2026-08-19T14:09:15Z Ottomata 88 /* Usefulness */ 2449731 wikitext text/x-wiki 0il01jrad6qmu0by0lxkibxu2fn41ak 0 460556 2449998 2447592 2026-08-20T10:36:01Z Atsuko-WMF 54082 /* Why */ 2449998 wikitext text/x-wiki rxfqwzqvyrvkl1ddq1b7mt8wbc5xo2a 0 460559 2449811 2448545 2026-08-19T16:06:29Z Iackerman 41698 minor style consistency things 2449811 wikitext text/x-wiki ivwy78n71lhy8s2ujkliwncpyv2iu8y 0 460568 2449871 2026-08-19T18:33:13Z Novem Linguae 23682 create 2449871 wikitext text/x-wiki angffad04amef7fmfqv4unebhku0dxa 0 460569 2449872 2026-08-19T18:33:46Z Novem Linguae 23682 create 2449872 wikitext text/x-wiki ms6dc033ms74w2q4omcrofxadnxo1qx