Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.18 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 739 2454950 2433934 2026-09-07T07:26:41Z Stashbot 7414 dcaro: deleted stale ferm-realserver-mss.prom file that was triggering an alert 2454950 wikitext text/x-wiki m0o4qp9mwafxpx4espy240mbp88b1n4 1 2515 2454933 2452097 2026-09-06T17:00:01Z Lhnh067 59346 Đanh may 2454933 wikitext text/x-wiki koaygn0unk2cdyk6fyahsa69jwcifuj 2454936 2454933 2026-09-06T17:03:45Z Enbi 52304 Reverted 1 edit by [[Special:Contributions/Lhnh067|Lhnh067]] ([[User talk:Lhnh067|talk]]) (TwinkleGlobal) 2454936 wikitext text/x-wiki siv1uf1xtkisdnkqlv766k79ajz1z2r 0 4108 2454967 2454905 2026-09-07T08:03:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1337450]] to Monday, September 07 UTC afternoon backport window 2454967 wikitext text/x-wiki hi6s9186mw4vux6t8jhmwrczmr391ho 0 7919 2454944 2454917 2026-09-07T02:00:17Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2454944 wikitext text/x-wiki ryuqwfrqlomjoydbwcroviegdwgeqvz 2454945 2454944 2026-09-07T02:08:50Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 33s) 2454945 wikitext text/x-wiki autqm8i3n94cvb905yufzd43vafjfdp 2454947 2454945 2026-09-07T06:14:47Z Stashbot 7414 moritzm: installing Chromium security updates 2454947 wikitext text/x-wiki qitxdhszjbrtfx5sshqmrrib4qkymaf 2454948 2454947 2026-09-07T07:05:14Z Stashbot 7414 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1329314|ArticleGuidance: Configure feedback links to local talk pages (T433483)]] 2454948 wikitext text/x-wiki a3sbsnusmsfysjh6p9f6dmcwfpobm4p 2454949 2454948 2026-09-07T07:23:42Z Stashbot 7414 kartik@deploy1003: abi, kartik: Backport for [[gerrit:1329314|ArticleGuidance: Configure feedback links to local talk pages (T433483)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2454949 wikitext text/x-wiki e4rvs620yqcgbhf639fon4p80u3w1cy 2454951 2454949 2026-09-07T07:34:51Z Stashbot 7414 kartik@deploy1003: abi, kartik: Continuing with deployment 2454951 wikitext text/x-wiki 3k38hwtydzizys2tllo56r6r0zkr5ri 2454952 2454951 2026-09-07T07:45:44Z Stashbot 7414 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply 2454952 wikitext text/x-wiki 0jr4nst6vd5a28lraw1wnx6i8y4ys66 2454953 2454952 2026-09-07T07:46:38Z Stashbot 7414 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply 2454953 wikitext text/x-wiki 1osemf7dics65xb63ym9bky9ctsuyy4 2454954 2454953 2026-09-07T07:47:05Z Stashbot 7414 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1329314|ArticleGuidance: Configure feedback links to local talk pages (T433483)]] (duration: 41m 51s) 2454954 wikitext text/x-wiki emubandrcanfwddsxepve7ue7mdg94t 2454955 2454954 2026-09-07T07:50:45Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply 2454955 wikitext text/x-wiki a3qv2lg8i6wvi5ydzqjgke2u54meme9 2454956 2454955 2026-09-07T07:51:58Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply 2454956 wikitext text/x-wiki fw22l7yh4ct563sxenxc187bczq5af5 2454957 2454956 2026-09-07T07:54:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1180: Cloning 2454957 wikitext text/x-wiki q2yp11cy00ggn81trtk94sp23571vd6 2454958 2454957 2026-09-07T07:54:51Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1180: Cloning 2454958 wikitext text/x-wiki npjhgrtf4e4di5ssci4fijnzdyl56gp 2454959 2454958 2026-09-07T07:55:03Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1241: Cloning 2454959 wikitext text/x-wiki acgw8mot3s1a59u8lkdteffgof22byq 2454960 2454959 2026-09-07T07:55:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1241: Cloning 2454960 wikitext text/x-wiki mua1vq2e5t0tsdihdgh5dpzmuf547q5 2454961 2454960 2026-09-07T07:56:17Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Cloning 2454961 wikitext text/x-wiki l91fnzow9chdz0964uvfj26aq3aqsif 2454962 2454961 2026-09-07T07:57:50Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1180.eqiad.wmnet onto db1241.eqiad.wmnet 2454962 wikitext text/x-wiki bpbplwzmmltxk46mn27yqnzvzsqqzqf 2454963 2454962 2026-09-07T08:00:46Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply 2454963 wikitext text/x-wiki occytynqxn7l5lfggmb2g5r2lgcs6ip 2454964 2454963 2026-09-07T08:02:04Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. 2454964 wikitext text/x-wiki 4y64kmg54nwif29g7qukh4cyaqylnan 2454965 2454964 2026-09-07T08:02:07Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply 2454965 wikitext text/x-wiki fzsw6p7z50tbt8uq7bap23thf3jk5dy 2454966 2454965 2026-09-07T08:03:27Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2454966 wikitext text/x-wiki e9owvuu4xlbsf88g9bsr5tebovlsl5w 2454968 2454966 2026-09-07T08:03:59Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host aqs1022.eqiad.wmnet with OS bookworm 2454968 wikitext text/x-wiki hbr0l9x9ep5zbposy33azn88liw943z 2454969 2454968 2026-09-07T08:20:52Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. 2454969 wikitext text/x-wiki q626z7crtsen7mbxs205e7rfi9bty6r 2454970 2454969 2026-09-07T08:22:03Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2454970 wikitext text/x-wiki qwedriwu1xkdowvzfokzqw67krg5zmv 2454972 2454970 2026-09-07T08:28:22Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.ganeti.makevm for new host ldap-replica1005.wikimedia.org 2454972 wikitext text/x-wiki fg6m2tiq1298a72y5vti3jk1j6hxns1 2454973 2454972 2026-09-07T08:28:24Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.netbox 2454973 wikitext text/x-wiki kp9h6uvqzhtu9x2g9s3atnoyjyo2g3r 2454974 2454973 2026-09-07T08:34:37Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s5 T437188 2454974 wikitext text/x-wiki guxusf0x5te7en7xwljc0w7y3fvle3b 2454975 2454974 2026-09-07T08:34:51Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2157 with weight 0 T437188', diff saved to https://phabricator.wikimedia.org/P96353 and previous config saved to /var/cache/conftool/dbconfig/20260907-083448-marostegui.json 2454975 wikitext text/x-wiki 4kfu59iqxd1vz52aopd7kmizntsd2ef 2454977 2454975 2026-09-07T08:37:40Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica1005.wikimedia.org - jmm@cumin1004" 2454977 wikitext text/x-wiki 96419l9j56w4dowkn5bnyd9bz4gcsxj 2454978 2454977 2026-09-07T08:37:43Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica1005.wikimedia.org - jmm@cumin1004" 2454978 wikitext text/x-wiki ldxlckpkmvl8ryeg7y7kwcsizc5ctcx 2454979 2454978 2026-09-07T08:37:44Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2454979 wikitext text/x-wiki ja19wjp6bhskd2p7jr0skzw0p18a6nf 2454980 2454979 2026-09-07T08:37:45Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.wipe-cache ldap-replica1005.wikimedia.org on all recursors 2454980 wikitext text/x-wiki o0ywhbc56y8f10ts326u9f36muncmsj 2454981 2454980 2026-09-07T08:37:51Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ldap-replica1005.wikimedia.org on all recursors 2454981 wikitext text/x-wiki 9ntnc2cxa68wcrcy8n2g6e2ac6ufic5 2454982 2454981 2026-09-07T08:38:04Z Stashbot 7414 marostegui: Starting s5 codfw failover from db2213 to db2157 - T437188 2454982 wikitext text/x-wiki o6p4fp3n0ximfgquf69ipktzm75cm91 2454983 2454982 2026-09-07T08:38:18Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica1005.wikimedia.org - jmm@cumin1004" 2454983 wikitext text/x-wiki rs3oxlbcg2zogozg2v0gmbn3njpjpov 2454984 2454983 2026-09-07T08:38:21Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica1005.wikimedia.org - jmm@cumin1004" 2454984 wikitext text/x-wiki 79978wofwxn90xwldfhxcqmpiytoers 2454985 2454984 2026-09-07T08:38:28Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2157 to s5 primary T437188', diff saved to https://phabricator.wikimedia.org/P96354 and previous config saved to /var/cache/conftool/dbconfig/20260907-083825-marostegui.json 2454985 wikitext text/x-wiki dyf29z6xohfn7c2fonyxmfl110f78fv 2454986 2454985 2026-09-07T08:39:00Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.reimage for host ldap-replica1005.wikimedia.org with OS trixie 2454986 wikitext text/x-wiki h71jhqrp69z7ofuiwtepxg6bihmjhxc 2454987 2454986 2026-09-07T08:39:07Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2213 T437188', diff saved to https://phabricator.wikimedia.org/P96355 and previous config saved to /var/cache/conftool/dbconfig/20260907-083904-marostegui.json 2454987 wikitext text/x-wiki d20clzh5ugrre7umo3gyjsjes6wt3hj 2454988 2454987 2026-09-07T08:39:26Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover 2454988 wikitext text/x-wiki hd9vh03dkfnrjvea69n3js4k8jtf5jt 2454989 2454988 2026-09-07T08:40:25Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2213: Repooling after switchover 2454989 wikitext text/x-wiki 0hgn01hb7z4z1yqff4c1224thkpnnp8 2454990 2454989 2026-09-07T08:40:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover 2454990 wikitext text/x-wiki dzqczzga748x8m3vrwyypq2rvft0qdk 2454991 2454990 2026-09-07T08:41:01Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. 2454991 wikitext text/x-wiki 7rj5m2haf41dttkwq1whpfu1ptn2158 2454992 2454991 2026-09-07T08:42:15Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2454992 wikitext text/x-wiki ecwt1g98vq9bib16dphtqj1pyqjrhrp 2454993 2454992 2026-09-07T08:48:05Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1022.eqiad.wmnet with reason: host reimage 2454993 wikitext text/x-wiki almp6ttmcqbtb80bebpq1vzzz8aae28 2454994 2454993 2026-09-07T08:49:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pool db1180.eqiad.wmnet in after cloning 2454994 wikitext text/x-wiki accvdjkqkwle4xrwlgykvjwbmgieiin 2454995 2454994 2026-09-07T08:52:36Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-replica1005.wikimedia.org with reason: host reimage 2454995 wikitext text/x-wiki ppbx8m6t5onjjp75bap0ig1ue2bygbe 2454996 2454995 2026-09-07T08:52:42Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1022.eqiad.wmnet with reason: host reimage 2454996 wikitext text/x-wiki 2p8x79c1a9w5rwihbcb45z5bdejh4sq 2454997 2454996 2026-09-07T08:54:55Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1337455|Mentorship: Don't renew mentors' away status on every cleaner run (T436659)]], [[gerrit:1337458|Ignore PersonalDashboard stubs (T428679)]] 2454997 wikitext text/x-wiki c2e8cu385m60m20rr5hqw9gs7tvrbxj 2454998 2454997 2026-09-07T08:55:39Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync 2454998 wikitext text/x-wiki 08fc04sef5w62brlf7eag0u1f8s4ae2 2454999 2454998 2026-09-07T08:55:40Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync 2454999 wikitext text/x-wiki 4a8s7ubpgvrn0riqv7zjzbf2a2x1vmq 2455000 2454999 2026-09-07T08:56:24Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-replica1005.wikimedia.org with reason: host reimage 2455000 wikitext text/x-wiki f65xedn78vxdt9tpxunt61g3avw9l6j 2455001 2455000 2026-09-07T08:56:54Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.reboot-single for host maps-test2001.codfw.wmnet 2455001 wikitext text/x-wiki pkkit21obc71bo048wbtyu2hlu7jr8u 2455002 2455001 2026-09-07T08:57:45Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2455002 wikitext text/x-wiki tuw2s0lasmnom1ye01w1xvfgo7v8nis 2455003 2455002 2026-09-07T08:59:05Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2455003 wikitext text/x-wiki obqi9omlr6pttmxpth9uo8rkszz224x 2455004 2455003 2026-09-07T09:01:10Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1337455|Mentorship: Don't renew mentors' away status on every cleaner run (T436659)]], [[gerrit:1337458|Ignore PersonalDashboard stubs (T428679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455004 wikitext text/x-wiki kjpi859u1n5fb905qgpe1gbsps5ocyb 2455005 2455004 2026-09-07T09:02:07Z Stashbot 7414 moritzm: installing giflib security updates 2455005 wikitext text/x-wiki 4cdb0y0az7dwjanh54guwllz19q27ue 2455006 2455005 2026-09-07T09:03:16Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps-test2001.codfw.wmnet 2455006 wikitext text/x-wiki rof5dmo2h26qc4ioarsjitgue26j8h3 2455007 2455006 2026-09-07T09:04:06Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2455007 wikitext text/x-wiki h56ayre9y6f22d5f7i3xd34qs4387vi 2455008 2455007 2026-09-07T09:08:15Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Continuing with deployment 2455008 wikitext text/x-wiki f0p1yes3ge7olrimxgc8c5wrtahtoll 2455009 2455008 2026-09-07T09:10:02Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1022.eqiad.wmnet with OS bookworm 2455009 wikitext text/x-wiki 4y7wdyiduu0buc05t70aoglcbq7273b 2455010 2455009 2026-09-07T09:10:05Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 139009 2455010 wikitext text/x-wiki czs5zi38ztrjgk1wu4ap9sz6h4u2x5f 2455011 2455010 2026-09-07T09:10:21Z Stashbot 7414 moritzm: rebuild software RAID following disk replacement T437036 2455011 wikitext text/x-wiki 24ynstltmhdhitlkdr4l42z8qn3hq8g 2455012 2455011 2026-09-07T09:10:31Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-replica1005.wikimedia.org with OS trixie 2455012 wikitext text/x-wiki jdck442zp83zbtjw56lraxossmf6yey 2455013 2455012 2026-09-07T09:10:33Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ldap-replica1005.wikimedia.org 2455013 wikitext text/x-wiki dyd2vtsh3ckc7bsvkycb3rfxui4cy55 2455014 2455013 2026-09-07T09:12:16Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 139009 2455014 wikitext text/x-wiki 8kahg5wx8saatntjhxyl9mth5qojyc8 2455015 2455014 2026-09-07T09:15:07Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337455|Mentorship: Don't renew mentors' away status on every cleaner run (T436659)]], [[gerrit:1337458|Ignore PersonalDashboard stubs (T428679)]] (duration: 20m 12s) 2455015 wikitext text/x-wiki 8rg9eh04c5whxdptdbx42ckt4y9guor 2455016 2455015 2026-09-07T09:23:41Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-codfw 2455016 wikitext text/x-wiki nboi3f25jpp43vdjakwy91pc7t52g8i 2455017 2455016 2026-09-07T09:25:42Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2213: Repooling after switchover 2455017 wikitext text/x-wiki tpcbm80x2xbpsi019vifkuczwaqfz63 2455018 2455017 2026-09-07T09:32:20Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2455018 wikitext text/x-wiki o2musroic29hrrudwnuxf1ewz0e68mf 2455019 2455018 2026-09-07T09:32:34Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2455019 wikitext text/x-wiki 9usl6tjqycjiuvi6ku7kggbekh9atks 2455020 2455019 2026-09-07T09:34:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pool db1180.eqiad.wmnet in after cloning 2455020 wikitext text/x-wiki o0fce5klskbctenpnn3n8onj5icvhdi 2455021 2455020 2026-09-07T09:37:24Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2455021 wikitext text/x-wiki mwlbb6lnxvwo8izq7zczqbwz5cujkt6 2455022 2455021 2026-09-07T09:37:29Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2455022 wikitext text/x-wiki glv072a89h0i94qcdybduohwpz9zpoc 2455023 2455022 2026-09-07T09:38:59Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2455023 wikitext text/x-wiki 0bavb2txwvpj9sin8n9io84tno85rw2 2455024 2455023 2026-09-07T09:41:33Z Stashbot 7414 aokoth@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2455024 wikitext text/x-wiki ncml8hltn3pxr61fr0crp1e17gd2oe6 2455025 2455024 2026-09-07T09:42:26Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1337379|Enable thumb.wikimedia.org everywhere (T427465)]] 2455025 wikitext text/x-wiki j3syrwuz6qzisbpxjqakj01vkpiyj8w 2455026 2455025 2026-09-07T09:44:42Z Stashbot 7414 aokoth@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2455026 wikitext text/x-wiki ersrp1a80o3x993nozbp6vrl5sp0lok 2455027 2455026 2026-09-07T09:45:45Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2455027 wikitext text/x-wiki 5d8qet3hxka5i2c47wm3880tyyfsi5p 2455028 2455027 2026-09-07T09:46:58Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1337379|Enable thumb.wikimedia.org everywhere (T427465)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455028 wikitext text/x-wiki nn9ig1oicv0n9z2zdwfn7u639doseam 2455029 2455028 2026-09-07T09:47:57Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2455029 wikitext text/x-wiki 02sry0w27n5iwrke6r1256k9ji5ymn9 2455030 2455029 2026-09-07T09:48:02Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2455030 wikitext text/x-wiki eo4hcgds2orimflrj6674h5p8mhwsjl 2455031 2455030 2026-09-07T09:48:28Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2455031 wikitext text/x-wiki qzxrz3z55d7ixqi9iy1tphrmt8cdqyd 2455032 2455031 2026-09-07T09:48:50Z Stashbot 7414 aokoth@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2455032 wikitext text/x-wiki t9dyct03n41h1bj3edpbvqqhy8zliz2 2455033 2455032 2026-09-07T09:49:00Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.ganeti.makevm for new host ldap-replica1006.wikimedia.org 2455033 wikitext text/x-wiki jncf4n7cf52z69vss0oantful4o4qvf 2455034 2455033 2026-09-07T09:49:02Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.netbox 2455034 wikitext text/x-wiki hikpa1a6hz651pxnnw53rdmqrp4ckgi 2455035 2455034 2026-09-07T09:50:20Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw 2455035 wikitext text/x-wiki aal24dytrjja9j5ya4t6z3caakp8fqp 2455036 2455035 2026-09-07T09:51:26Z Stashbot 7414 aokoth@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2455036 wikitext text/x-wiki flc2dodwg8s3amr3g8i38b6ys900k7a 2455037 2455036 2026-09-07T09:52:38Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337379|Enable thumb.wikimedia.org everywhere (T427465)]] (duration: 10m 11s) 2455037 wikitext text/x-wiki kt5b40tqqrvqit4f55a66oebh2qqfff 2455038 2455037 2026-09-07T09:52:40Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2455038 wikitext text/x-wiki dj9ba9wdxgc9odiu7dl9ge3juzckvr3 2455039 2455038 2026-09-07T09:53:32Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica1006.wikimedia.org - jmm@cumin1004" 2455039 wikitext text/x-wiki 824o5rut22g3n6i71wr1om44gv66gmf 2455040 2455039 2026-09-07T09:53:43Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2455040 wikitext text/x-wiki ld5j51pj107a1200dhj4oitc1lz7byk 2455041 2455040 2026-09-07T09:54:40Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2455041 wikitext text/x-wiki mlluxliqt5e2nktd53slzfas6r3gg3q 2455042 2455041 2026-09-07T09:54:41Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw 2455042 wikitext text/x-wiki 9w19c70uvrrveza4bh34ajhjr8lirah 2455043 2455042 2026-09-07T09:56:19Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2455043 wikitext text/x-wiki gd4cormozsasc78w1kti4ka6ugr6jn3 2455044 2455043 2026-09-07T09:56:47Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2455044 wikitext text/x-wiki i21gru1msgja24p2xcz3hbpzl7dld6x 2455045 2455044 2026-09-07T09:57:02Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica1006.wikimedia.org - jmm@cumin1004" 2455045 wikitext text/x-wiki aso07tfcp8y5xjgaxuoz158s0rm9q2z 2455046 2455045 2026-09-07T09:57:03Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2455046 wikitext text/x-wiki h0ut1onssjqfvi84clcddkb6lywbkqx 2455047 2455046 2026-09-07T09:57:04Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.wipe-cache ldap-replica1006.wikimedia.org on all recursors 2455047 wikitext text/x-wiki nklz9pl9wh7o5ft4k3y010q882t71hk 2455048 2455047 2026-09-07T09:57:05Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ldap-replica1006.wikimedia.org on all recursors 2455048 wikitext text/x-wiki avrikc8x9puiblb49zkl70ojcph7mtq 2455049 2455048 2026-09-07T09:57:36Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica1006.wikimedia.org - jmm@cumin1004" 2455049 wikitext text/x-wiki evtuxer58sogw4d6jjhtxzsva0bqkbw 2455050 2455049 2026-09-07T09:57:40Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica1006.wikimedia.org - jmm@cumin1004" 2455050 wikitext text/x-wiki cs8gail95ucecwxclrn8pbjqbxxotqk 2455051 2455050 2026-09-07T09:57:45Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.reimage for host ldap-replica1006.wikimedia.org with OS trixie 2455051 wikitext text/x-wiki o7qv08i6o7gx9rpg03o17ubtcc9olte 2455052 2455051 2026-09-07T09:58:05Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2455052 wikitext text/x-wiki bqpozws0ykon1a4hypw0nfq2dgim03e 2455053 2455052 2026-09-07T09:58:51Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2455053 wikitext text/x-wiki guqkshfm1mwnaxhom0wmoubi8oln288 2455054 2455053 2026-09-07T09:59:48Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2455054 wikitext text/x-wiki qv5eu5drgkj4qgw9jm0b6kszkv5uayf 2455055 2455054 2026-09-07T10:00:36Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2455055 wikitext text/x-wiki qxtk7idsw6tu9vsps4n5jzm1qojcnfn 2455056 2455055 2026-09-07T10:02:26Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2001.codfw.wmnet 2455056 wikitext text/x-wiki hk0koy425fit3ax9aalox1fu4b9phqv 2455057 2455056 2026-09-07T10:04:22Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2455057 wikitext text/x-wiki 18rmiyv5o5vv63zroter40x2ypo09vb 2455058 2455057 2026-09-07T10:06:10Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2455058 wikitext text/x-wiki mop5d7i1xe0en2rug2q7rbna6ne9dqt 2455059 2455058 2026-09-07T10:06:35Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2455059 wikitext text/x-wiki 385gbikl04rywos7z4rp38vjw01xqn2 2455060 2455059 2026-09-07T10:07:33Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad 2455060 wikitext text/x-wiki l983qjfsgw6k7fh25v3ogaij5kjt9qj 2455061 2455060 2026-09-07T10:07:52Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2455061 wikitext text/x-wiki kmhjs0mzl1350m9qcbcldwcpch6096x 2455062 2455061 2026-09-07T10:08:57Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2455062 wikitext text/x-wiki 0jb0do9uqvlndt5kz30ina6re3g29lb 2455063 2455062 2026-09-07T10:09:55Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2455063 wikitext text/x-wiki ew6r6oq4497m87932jd285wk2o8x85f 2455064 2455063 2026-09-07T10:10:47Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2455064 wikitext text/x-wiki mvpzbjbl1ptc8cvcbncsn70zlifqi42 2455065 2455064 2026-09-07T10:11:43Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2455065 wikitext text/x-wiki m0kcanavpm795ulzk9kuo9nzrsq5skc 2455066 2455065 2026-09-07T10:11:44Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad 2455066 wikitext text/x-wiki 394s04qc8ge33bapfm9uklivowm3gxn 2455067 2455066 2026-09-07T10:12:51Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-replica1006.wikimedia.org with reason: host reimage 2455067 wikitext text/x-wiki tfxxddzqgbcfji5xw5v8cfhh58oa78m 2455068 2455067 2026-09-07T10:13:03Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-codfw 2455068 wikitext text/x-wiki blybxf83onodjjl2175s9v1ollagle1 2455069 2455068 2026-09-07T10:16:01Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host aqs2001.codfw.wmnet 2455069 wikitext text/x-wiki mqdifhbapv0s8371vxqe4jyg2qmewhm 2455070 2455069 2026-09-07T10:18:17Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-replica1006.wikimedia.org with reason: host reimage 2455070 wikitext text/x-wiki 7olrgzu7kruveeyet0jzas08kictday 2455071 2455070 2026-09-07T10:25:29Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2001.codfw.wmnet 2455071 wikitext text/x-wiki nblmzkgjam8by9lmhtzwaibeemf9fxq 2455072 2455071 2026-09-07T10:25:31Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2001.codfw.wmnet 2455072 wikitext text/x-wiki ttpp1jg42zw5r45o8nf26yyntcc96m7 2455073 2455072 2026-09-07T10:25:55Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host aqs2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2455073 wikitext text/x-wiki rb60y3w1dxrpuyvkufbup44vkaq1et2 2455074 2455073 2026-09-07T10:26:05Z Stashbot 7414 btullis@deploy1003: Started scap sync-world: Attempting to update mediawiki-cli for T436913 2455074 wikitext text/x-wiki kyest4220msb7n0enidaz3mn4k4fq9r 2455075 2455074 2026-09-07T10:26:30Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1340.eqiad.wmnet 2455075 wikitext text/x-wiki 6ecbyg03rpj8arvk4x58y82nfvuwz82 2455076 2455075 2026-09-07T10:26:34Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1340.eqiad.wmnet 2455076 wikitext text/x-wiki ne8z1pg8b7haj9pys8wg0q3c9p9r3ir 2455077 2455076 2026-09-07T10:27:10Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1340.eqiad.wmnet 2455077 wikitext text/x-wiki faw0hgjkdtznyvnr3125c2idslcokjy 2455078 2455077 2026-09-07T10:27:24Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host wikikube-worker1340.eqiad.wmnet with OS trixie 2455078 wikitext text/x-wiki o6ynda7uve4xf8r7xmd5abuhmtrn651 2455079 2455078 2026-09-07T10:27:40Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2455079 wikitext text/x-wiki b6erkdar7dwg52vhellkkxpt0wsz7kq 2455080 2455079 2026-09-07T10:27:54Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1340 2455080 wikitext text/x-wiki ncz0d8wis8ltxfd11hmfuwddy1tgyst 2455081 2455080 2026-09-07T10:29:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host aqs2001.codfw.wmnet with OS bookworm 2455081 wikitext text/x-wiki dpw5oqcqmtv4hcvxhu6dlgy3tygci67 2455082 2455081 2026-09-07T10:33:02Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2455082 wikitext text/x-wiki jmg7cilfp152a4aujskx343ixce3lyw 2455083 2455082 2026-09-07T10:33:28Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-replica1006.wikimedia.org with OS trixie 2455083 wikitext text/x-wiki 8r03k2g9xqhzekjk5zmbh6ihvzn85it 2455084 2455083 2026-09-07T10:33:32Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ldap-replica1006.wikimedia.org 2455084 wikitext text/x-wiki 04r7eda49dq4ieelkhzybnysfjsyk9t 2455086 2455084 2026-09-07T10:35:00Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1241: Pool db1241.eqiad.wmnet in after cloning 2455086 wikitext text/x-wiki pa2aceq6t3hofahhpsyow2ktt1gifl1 2455087 2455086 2026-09-07T10:37:11Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1340 - cgoubert@cumin2003" 2455087 wikitext text/x-wiki hclrdna3t818rdi00s5a7g9xbvjn6ra 2455088 2455087 2026-09-07T10:38:14Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1181.eqiad.wmnet onto db1190.eqiad.wmnet 2455088 wikitext text/x-wiki pje2vz8lnsd7p96bbolpq0j0fmhfg43 2455089 2455088 2026-09-07T10:38:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Depool db1181.eqiad.wmnet to then clone it to db1190.eqiad.wmnet - marostegui@cumin1003 2455089 wikitext text/x-wiki km3t9zbf7fhl4dmu5gb9wfgcs368x0r 2455091 2455089 2026-09-07T10:38:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Depool db1181.eqiad.wmnet to then clone it to db1190.eqiad.wmnet - marostegui@cumin1003 2455091 wikitext text/x-wiki 2g4roabbdpmhq6vplhjawug311bcbkj 2455092 2455091 2026-09-07T10:38:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1190: Depool db1190.eqiad.wmnet - marostegui@cumin1003 2455092 wikitext text/x-wiki 6d0t5wrlecb0xe103lsb0tc9ua17pst 2455093 2455092 2026-09-07T10:39:07Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1190: Depool db1190.eqiad.wmnet - marostegui@cumin1003 2455093 wikitext text/x-wiki pwm006r86cpnn497pw9v358raqrpwjz 2455094 2455093 2026-09-07T10:39:10Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1340 - cgoubert@cumin2003" 2455094 wikitext text/x-wiki prgeaaf9l7utfcos2bnib4jjr0n35sr 2455095 2455094 2026-09-07T10:39:11Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2455095 wikitext text/x-wiki shh5u5bj4n8345jycwn8xap33dnxo1d 2455096 2455095 2026-09-07T10:39:13Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache wikikube-worker1340.eqiad.wmnet 164.32.64.10.in-addr.arpa 4.6.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455096 wikitext text/x-wiki ftaluskshezri3dvukq5jurdnjx1wph 2455097 2455096 2026-09-07T10:39:17Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1340.eqiad.wmnet 164.32.64.10.in-addr.arpa 4.6.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2455097 wikitext text/x-wiki h8ohkz94wetdavumigtgnvgl8t53mdf 2455098 2455097 2026-09-07T10:39:18Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1340 2455098 wikitext text/x-wiki l6eqi63lyd445v1lqa12rtxzi5d0b6c 2455099 2455098 2026-09-07T10:39:50Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1340 2455099 wikitext text/x-wiki ozs9kg1tdbmg4kaem9li38frm7tjerc 2455100 2455099 2026-09-07T10:39:51Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1340 2455100 wikitext text/x-wiki n706c1efq68b15b7bl092j7j0rx2zo9 2455101 2455100 2026-09-07T10:40:54Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: extensions/GrowthExperiments/maintenance/cleanMentorList.php --wiki=hrwiki # T436659 2455101 wikitext text/x-wiki 0tnktebz7kpfknlaq65vvu5bley53ug 2455102 2455101 2026-09-07T10:47:06Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: extensions/GrowthExperiments/maintenance/cleanMentorList.php --wiki=frwiki # T436659 2455102 wikitext text/x-wiki 7zb74soz437htgbdiqzowhvx7zbkz4a 2455103 2455102 2026-09-07T10:50:39Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2455103 wikitext text/x-wiki 86k4ahnmfqkghwazuqshuabybddasz9 2455104 2455103 2026-09-07T10:51:56Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1340.eqiad.wmnet with reason: host reimage 2455104 wikitext text/x-wiki 5u47ptaus0gec4xcwrlxtufsm1fvlx4 2455105 2455104 2026-09-07T10:53:08Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2455105 wikitext text/x-wiki szczb3rv6vnotqi3mr8swtmgk5nyysy 2455106 2455105 2026-09-07T10:54:59Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1340.eqiad.wmnet with reason: host reimage 2455106 wikitext text/x-wiki g995gkrwkyrwczgyr3y23yuon7tguoc 2455107 2455106 2026-09-07T10:58:33Z Stashbot 7414 btullis@deploy1003: Finished scap sync-world: Attempting to update mediawiki-cli for T436913 (duration: 35m 20s) 2455107 wikitext text/x-wiki muz1bf3jn3000aj2pvt5uyj8z4dwz2l 2455108 2455107 2026-09-07T11:02:15Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2001.codfw.wmnet with reason: host reimage 2455108 wikitext text/x-wiki 3rzrqqpsbq591cj15h206hciox3pggv 2455109 2455108 2026-09-07T11:04:12Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1337550|rdbms: Discourage setting db domain to false for remote virtual domains (T422940)]], [[gerrit:1337310|Migrate querying categorylinks to virtual domain (T405812)]] 2455109 wikitext text/x-wiki gvay0z1yf49vbl4gqd39oiu5jpjl87x 2455111 2455109 2026-09-07T11:06:47Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges 2455111 wikitext text/x-wiki krkgqzd3t1cutjmx1au3e2wyyiqpw49 2455112 2455111 2026-09-07T11:06:49Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) 2455112 wikitext text/x-wiki orftazcj8nkaqx5217g9yhazva15nzo 2455114 2455112 2026-09-07T11:07:44Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1339.eqiad.wmnet 2455114 wikitext text/x-wiki rvyccz2wrvmokl4fozb1o73y11j55vc 2455115 2455114 2026-09-07T11:07:46Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1339.eqiad.wmnet 2455115 wikitext text/x-wiki hilj6hui0egn64skhqbtj1v6cqnb59l 2455116 2455115 2026-09-07T11:07:48Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet 2455116 wikitext text/x-wiki ss4ddzz2v6t0ewjuft4hxqc9gjbsf5o 2455117 2455116 2026-09-07T11:09:35Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2001.codfw.wmnet with reason: host reimage 2455117 wikitext text/x-wiki 2j01clsw4cu7y0tr8x082ncatlza09v 2455118 2455117 2026-09-07T11:10:22Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1337550|rdbms: Discourage setting db domain to false for remote virtual domains (T422940)]], [[gerrit:1337310|Migrate querying categorylinks to virtual domain (T405812)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2455118 wikitext text/x-wiki 6bko7j8tave6fmcew1busidqc99lbc6 2455119 2455118 2026-09-07T11:10:51Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with deployment 2455119 wikitext text/x-wiki shp9bhpby4zjvqa15fcoriiywnxm57i 2455120 2455119 2026-09-07T11:11:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges 2455120 wikitext text/x-wiki t7c95f8wejv298xvgyq72ggalfsnwwc 2455121 2455120 2026-09-07T11:11:31Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) 2455121 wikitext text/x-wiki gtj0odnlp6v2mi8njy06l21e3zbv48k 2455122 2455121 2026-09-07T11:13:35Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet 2455122 wikitext text/x-wiki 2g16g14qveuwh3dlj7xmbhwjk96pgqq 2455123 2455122 2026-09-07T11:14:16Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1340.eqiad.wmnet with OS trixie 2455123 wikitext text/x-wiki huliosdzrbu9a3g1bkq5ddxxm5m38oh 2455124 2455123 2026-09-07T11:18:19Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337550|rdbms: Discourage setting db domain to false for remote virtual domains (T422940)]], [[gerrit:1337310|Migrate querying categorylinks to virtual domain (T405812)]] (duration: 14m 08s) 2455124 wikitext text/x-wiki 72jo1thggwzzhkg02legy7xpfg7hget 2455125 2455124 2026-09-07T11:20:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1241: Pool db1241.eqiad.wmnet in after cloning 2455125 wikitext text/x-wiki 6sgd6cgxsh22be2dngjmlp0l3kfccve 2455126 2455125 2026-09-07T11:20:10Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1180.eqiad.wmnet onto db1241.eqiad.wmnet 2455126 wikitext text/x-wiki g7zmvs0awjdw39ljusam5r7b6wwtmq8 2455127 2455126 2026-09-07T11:28:03Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.decommission for hosts ldap-replica1006.wikimedia.org 2455127 wikitext text/x-wiki jte7xhpfnx5vb5x0hhlv9jq8za14rek 2455128 2455127 2026-09-07T11:28:41Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2001.codfw.wmnet with OS bookworm 2455128 wikitext text/x-wiki d7l0cvosoosolyq7j6xy4wr07ace8n6 2455129 2455128 2026-09-07T11:32:57Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.netbox 2455129 wikitext text/x-wiki ioxclbve9emvzn045bmp0iyd8g2vsoz 2455131 2455129 2026-09-07T11:35:12Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2002.codfw.wmnet 2455131 wikitext text/x-wiki lh7s5mcb1d67n1djygeakfgpze8iypo 2455132 2455131 2026-09-07T11:37:58Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ldap-replica1006.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1004" 2455132 wikitext text/x-wiki 0tqh494fslms2hfy57v8t2iv3y89nbu 2455133 2455132 2026-09-07T11:39:33Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ldap-replica1006.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1004" 2455133 wikitext text/x-wiki tdykm9ix6ih986nkyspgolyuheh15of 2455134 2455133 2026-09-07T11:39:35Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2455134 wikitext text/x-wiki 7s6gbb4qnmig8ux8inv6w6spvn9xl5f 2455135 2455134 2026-09-07T11:39:36Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ldap-replica1006.wikimedia.org 2455135 wikitext text/x-wiki r783lxmufz7l0ylh5o26bdkrdh9xzv7 2455136 2455135 2026-09-07T11:39:57Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.ganeti.makevm for new host ldap-replica2007.wikimedia.org 2455136 wikitext text/x-wiki pyoq72srpqb64uimsmi0jzqzzeq7j57 2455137 2455136 2026-09-07T11:39:59Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.netbox 2455137 wikitext text/x-wiki ehalxs1xfhpk6umvbfdpzaa701xhq5v 2455138 2455137 2026-09-07T11:41:53Z Stashbot 7414 moritzm: installing bash updates from bookworm point release 2455138 wikitext text/x-wiki 5risa2776bnvdpyn0qp5ba6nzb62cmw 2455139 2455138 2026-09-07T11:45:23Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica2007.wikimedia.org - jmm@cumin1004" 2455139 wikitext text/x-wiki fxkibiyqnjfj1vtawt2sugv0rfgxgkj 2455140 2455139 2026-09-07T11:45:26Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica2007.wikimedia.org - jmm@cumin1004" 2455140 wikitext text/x-wiki 36jexcfd9ccsd6abir9saqaaywhelyc 2455141 2455140 2026-09-07T11:45:27Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2455141 wikitext text/x-wiki 78w7usm8hcbtcjhcu6lwcaloogxd5i9 2455142 2455141 2026-09-07T11:45:28Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.dns.wipe-cache ldap-replica2007.wikimedia.org on all recursors 2455142 wikitext text/x-wiki 9babh4lcjbtqtf1fzpnncnicfeojzre 2455143 2455142 2026-09-07T11:45:29Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ldap-replica2007.wikimedia.org on all recursors 2455143 wikitext text/x-wiki lyd9m80f3jk9kib6504lw56od506pxv 2455144 2455143 2026-09-07T11:46:05Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica2007.wikimedia.org - jmm@cumin1004" 2455144 wikitext text/x-wiki 4c9mvsfi9wlxbr6ohhgrcs465970a42 2455145 2455144 2026-09-07T11:46:07Z Stashbot 7414 jmm@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica2007.wikimedia.org - jmm@cumin1004" 2455145 wikitext text/x-wiki dvuzu91wbf0uw136raselbr9l8wysxb 2455146 2455145 2026-09-07T11:46:12Z Stashbot 7414 jmm@cumin1004: START - Cookbook sre.hosts.reimage for host ldap-replica2007.wikimedia.org with OS trixie 2455146 wikitext text/x-wiki 3j9tocs7aumrhvw10ipny1n11scknnd 2455147 2455146 2026-09-07T11:46:59Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host aqs2002.codfw.wmnet 2455147 wikitext text/x-wiki fi8b9ciyat0gkuj4fq4ev6y5rg2oeq8 2455149 2455147 2026-09-07T11:52:20Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2455149 wikitext text/x-wiki dja477h0s9jdhjxa5yx0d5p0dkzvthm 2455152 2455149 2026-09-07T11:54:51Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2455152 wikitext text/x-wiki o2nxfm7oo0iod9kfcc5c5zgk06aqf8d 2455153 2455152 2026-09-07T11:57:08Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2002.codfw.wmnet 2455153 wikitext text/x-wiki hprdjwbpmujya8t4ravu2ccd00vsrpb 2455154 2455153 2026-09-07T11:57:10Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2002.codfw.wmnet 2455154 wikitext text/x-wiki aqtbu6miy4sr76y9qcyulcphibpcrub 0 46349 2454937 2247080 2026-09-06T17:04:22Z Lhnh067 59346 /* */ 2454937 wikitext text/x-wiki n89t9e9cqsxa9jinm5co7wejtjpbkws 2454938 2454937 2026-09-06T17:04:46Z Enbi 52304 Undid edits by [[Special:Contribs/Lhnh067|Lhnh067]] ([[User talk:Lhnh067|talk]]) to last version by Taavi 2454938 wikitext text/x-wiki gl1gwimf9oayli4lhx3yba6f1vz7ycd 2454939 2454938 2026-09-06T17:09:13Z Lhnh067 59346 Đanh may 2454939 wikitext text/x-wiki a0j01dqpjr06tm1hgnqyk8e9bjnq69j 2454940 2454939 2026-09-06T17:09:23Z Enbi 52304 Undid edits by [[Special:Contribs/Lhnh067|Lhnh067]] ([[User talk:Lhnh067|talk]]) to last version by Enbi 2454940 wikitext text/x-wiki gl1gwimf9oayli4lhx3yba6f1vz7ycd 498 441701 2455110 2152690 2026-09-07T11:05:56Z Stashbot 7414 taavi: migrate to push to deploy 2455110 wikitext text/x-wiki aw50u53awtuqlcqzsrno6478kyb355x 2455113 2455110 2026-09-07T11:07:23Z Stashbot 7414 taavi: add x4 to list of sections T437208 2455113 wikitext text/x-wiki q3q308ns0qbejn5x9nusa9ycf9ni286 12 441751 2455090 2388302 2026-09-07T10:38:18Z Taavi-WMF 41365 /* Extension databases */ document existence of links extensions (x4) 2455090 wikitext text/x-wiki n70qqinflz96f423v41y1hvucpj9h6n 0 442914 2455085 2452780 2026-09-07T10:34:31Z RAdin satoshi2008 59012 /* Dashboards */ 2455085 wikitext text/x-wiki r8qbq03gl4ry2lzv6n0lm5nw8rvnjw7 10 444620 2454934 2453783 2026-09-06T17:02:55Z Lhnh067 59346 Đanh may 2454934 wikitext text/x-wiki f9tusylj9e70f1yylbxm12zhbc24rit 2454935 2454934 2026-09-06T17:03:37Z Enbi 52304 Undid edits by [[Special:Contribs/Lhnh067|Lhnh067]] ([[User talk:Lhnh067|talk]]) to last version by Taavi: test edits, please use the sandbox 2454935 wikitext text/x-wiki l27pb9t9u4tehx9ijwmaegc6f0sgah5 0 447352 2454971 2454899 2026-09-07T08:24:36Z LToscano (WMF) 7191 /* Storage backends */ 2454971 wikitext text/x-wiki 57ieyrfbjcufx6s1a1v1b6xbk8u51in 0 449160 2454943 2454914 2026-09-07T00:02:18Z Dexbot 30554 Bot: Updating the report 2454943 wikitext text/x-wiki p10lnl3h3jvmfyyoronmx1mvui3re9t 2454976 2454943 2026-09-07T08:36:20Z Dexbot 30554 Bot: Updating the report 2454976 wikitext text/x-wiki dfijt1wldhl7sb6yhmqwqpoyjooc6g5 116 453906 2454942 2454932 2026-09-06T21:36:18Z Gitlabaccountapprovalbot 37332 rp238 was rejected. 2454942 wikitext text/x-wiki qv4to4ieyxabkn5c60ev5pgc8qoiu4q 2454946 2454942 2026-09-07T03:00:21Z Gitlabaccountapprovalbot 37332 jessica22664 was rejected. 2454946 wikitext text/x-wiki hm34om356h5jwesb73kjcjozy890hlx 2 455625 2454941 2222933 2026-09-06T17:26:12Z Martin Urbanec 6634 + 2454941 wikitext text/x-wiki 4ierhdtlcw4wa06blfxxz4t6da9nnls 0 460621 2455130 2454063 2026-09-07T11:34:14Z JGiannelos (WMF) 17083 /* Service Level Indicators (SLIs) */ 2455130 wikitext text/x-wiki bwaocd4t7th6jx7khjues8cwth5nqkn 2455148 2455130 2026-09-07T11:49:26Z JGiannelos (WMF) 17083 /* Service Level Objectives */ 2455148 wikitext text/x-wiki hoii4hcti7ozpgaaxmbhb979jgv17az 2455150 2455148 2026-09-07T11:53:32Z JGiannelos (WMF) 17083 /* Ideal targets */ 2455150 wikitext text/x-wiki 9u7ys4ov21teo0bvrrav5g5ai3nng4z 2455151 2455150 2026-09-07T11:53:46Z JGiannelos (WMF) 17083 2455151 wikitext text/x-wiki anthuc0b5lhk51p1u6u05hlu1ajifkd