Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.20 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2457992 2457627 2026-09-16T12:28:41Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2457992 wikitext text/x-wiki f4196tl7zkqprghjh4d5r42yu4xdrz4 2457994 2457992 2026-09-16T12:33:49Z Stashbot 7414 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) 2457994 wikitext text/x-wiki 5hhuxj14jfq04dn8jvsas5fgpg0iwgx 2458010 2457994 2026-09-16T12:59:31Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 2458010 wikitext text/x-wiki hs72dab0fn0i5bqazr63jmh8i9dqy2u 2458013 2458010 2026-09-16T13:02:08Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-16 2458013 wikitext text/x-wiki o11be7fbfaiyfojwulu7tn9jplvmox9 0 4108 2457983 2457881 2026-09-16T12:05:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1342231]] to Wednesday, September 16 UTC afternoon backport window 2457983 wikitext text/x-wiki o4094ee7znh0g5pxe5v20g53ykw4jg0 2457984 2457983 2026-09-16T12:05:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1342234]] to Wednesday, September 16 UTC afternoon backport window 2457984 wikitext text/x-wiki hs4p19fxga8avj8zr3vu8lnai4s2xwo 2458209 2457984 2026-09-16T18:58:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1342339]] to Wednesday, September 16 UTC late backport window 2458209 wikitext text/x-wiki 9tuoztsmgs44qdlykvau55e7nlvl4vv 2458262 2458209 2026-09-16T21:50:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1331830]] to Thursday, September 17 UTC afternoon backport window 2458262 wikitext text/x-wiki dbmz8k5k7m0kb3ehchml23om1187y7d 2458263 2458262 2026-09-16T21:51:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1340216]] to Thursday, September 17 UTC afternoon backport window 2458263 wikitext text/x-wiki 1zkr0kw7csy6ccqbwyfnlfmg9qo4ffs 2458264 2458263 2026-09-16T21:51:37Z ScheduleDeploymentBot 37566 Add [[gerrit:1341949]] to Thursday, September 17 UTC afternoon backport window 2458264 wikitext text/x-wiki q0ktch32caroxkfkdz0av41ibk3j5ag 2458298 2458264 2026-09-17T06:51:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1342398]] to Thursday, September 17 UTC morning backport window 2458298 wikitext text/x-wiki 8re72sn72rsenx5bafh6hg6q6s7jb1h 2458299 2458298 2026-09-17T06:51:25Z ScheduleDeploymentBot 37566 Add [[gerrit:1342396]] to Thursday, September 17 UTC morning backport window 2458299 wikitext text/x-wiki 4s3ab2bjz6o2o9u2k88bjqg64eo60kp 2458300 2458299 2026-09-17T06:51:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1342395]] to Thursday, September 17 UTC morning backport window 2458300 wikitext text/x-wiki mcrbz8cagxo0jpb6d69auhaary580l1 2458301 2458300 2026-09-17T06:51:45Z ScheduleDeploymentBot 37566 Add [[gerrit:1342545]] to Thursday, September 17 UTC morning backport window 2458301 wikitext text/x-wiki jkf00tq4w16cp5mqendgvxcqtu2y1ue 2458302 2458301 2026-09-17T06:51:55Z ScheduleDeploymentBot 37566 Add [[gerrit:1342546]] to Thursday, September 17 UTC morning backport window 2458302 wikitext text/x-wiki 7t1ly651cna3bw6j1dt9ff2vudan8nd 2458303 2458302 2026-09-17T06:52:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1342547]] to Thursday, September 17 UTC morning backport window 2458303 wikitext text/x-wiki 3xkuezu6pyqj9mja5s69kr9cvbmfurq 0 7919 2457981 2457980 2026-09-16T11:59:26Z Stashbot 7414 moritzm: pruned obsolete Bullseye image python3-bullseye from the docker registry T416452 2457981 wikitext text/x-wiki cxnay5rwilc0ma1hk7s3ctle13aqtno 2457982 2457981 2026-09-16T12:01:49Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.20 refs T430839 2457982 wikitext text/x-wiki l99zxvcbnnhm1hgvamqz9lu5yu9sd6a 2457986 2457982 2026-09-16T12:09:44Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a1-eqiad 2457986 wikitext text/x-wiki ge3qukzeyjq45psv0ynw9ixhsbiob17 2457987 2457986 2026-09-16T12:11:40Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a1-eqiad 2457987 wikitext text/x-wiki m9ynevgmhx1tlydxm3p4ppbnkhbebjh 2457988 2457987 2026-09-16T12:12:46Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2457988 wikitext text/x-wiki cg55ykusx22t8b846sm6wmfz910nbka 2457989 2457988 2026-09-16T12:13:15Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5025.eqsin.wmnet with OS trixie 2457989 wikitext text/x-wiki 2sci1zbwef87r6nypvfwf4gevitgktd 2457993 2457989 2026-09-16T12:33:09Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6002.drmrs.wmnet 2457993 wikitext text/x-wiki js5x7z4hjn2nllbj42sl4pwtb7ufjcj 2457995 2457993 2026-09-16T12:34:05Z Stashbot 7414 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp6002.drmrs.wmnet with reason: reimage 2457995 wikitext text/x-wiki mb0xnyentivf9cspmg9fi8c0bxnkz47 2457996 2457995 2026-09-16T12:34:50Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.provision for host cp6002.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457996 wikitext text/x-wiki g9c0bopnm93be3utm7bihrr21jyb75m 2457997 2457996 2026-09-16T12:38:15Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a1-eqiad 2457997 wikitext text/x-wiki rvsahvicwtvnf2jdeeznsyw4q9ux3pp 2457998 2457997 2026-09-16T12:38:56Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a1-eqiad 2457998 wikitext text/x-wiki nli0c5mull3jo0epfwjk6xhs3i0rawb 2457999 2457998 2026-09-16T12:41:02Z Stashbot 7414 dkertesz@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, T438052] 2457999 wikitext text/x-wiki ab7act4tbi80j4maixz14bu6xh9d0rq 2458000 2457999 2026-09-16T12:41:17Z Stashbot 7414 dkertesz@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, T438052] 2458000 wikitext text/x-wiki edebarxhil1ktidyl5egdn40kr4mln9 2458002 2458000 2026-09-16T12:43:55Z Stashbot 7414 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: cluster=dnsbox,dc=eqsin,service=authdns-update 2458002 wikitext text/x-wiki k8nz3xcjmp4wdcmlzbz9q9tqyr5t85w 2458003 2458002 2026-09-16T12:45:49Z Stashbot 7414 dkertesz@dns1004: START - running authdns-update 2458003 wikitext text/x-wiki 6bo11cbuxznu8x1xogwuydl9prz0nud 2458004 2458003 2026-09-16T12:48:00Z Stashbot 7414 dkertesz@dns1004: END - running authdns-update 2458004 wikitext text/x-wiki 0fkghut9v56yyrqxstx7fpmhov35lqj 2458005 2458004 2026-09-16T12:49:41Z Stashbot 7414 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: cluster=dnsbox,dc=eqsin 2458005 wikitext text/x-wiki bp5bc7cq66hklt514mx4u0dy05d2xc4 2458006 2458005 2026-09-16T12:51:41Z Stashbot 7414 dkertesz: eqsin pooled again (T438052) 2458006 wikitext text/x-wiki jkr3r8ls9bjrnresqw4xjx3rklf813x 2458007 2458006 2026-09-16T12:52:07Z Stashbot 7414 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp6002.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2458007 wikitext text/x-wiki hpbmw0lk3gw6kcd7up7re6fgfggst0p 2458008 2458007 2026-09-16T12:52:44Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6002.drmrs.wmnet 2458008 wikitext text/x-wiki hir47ivk3pdjuefcmysz5yx1qlery5k 2458011 2458008 2026-09-16T12:59:44Z Stashbot 7414 dkertesz@cumin1004: conftool action : set/weight=1; selector: name=cp5025.eqsin.wmnet 2458011 wikitext text/x-wiki emf2z2ic30td6p07ysj4kgv7pr7ulnj 2458012 2458011 2026-09-16T13:00:47Z Stashbot 7414 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp5025.eqsin.wmnet 2458012 wikitext text/x-wiki s0rj6q6q6m8wyhmf87aiopmfb2jog4n 2458014 2458012 2026-09-16T13:05:52Z Stashbot 7414 sukhe@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp6002.drmrs.wmnet 2458014 wikitext text/x-wiki hgmaqfcl9gwubady6d3h6d82g1u4m4w 2458015 2458014 2026-09-16T13:05:55Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6002.drmrs.wmnet 2458015 wikitext text/x-wiki mrqmtgyejamzfpls1z8xa0ylns1lwf3 2458016 2458015 2026-09-16T13:09:24Z Stashbot 7414 sukhe@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp6002.drmrs.wmnet 2458016 wikitext text/x-wiki o08nl8vadqg6pq3n896ydws6jlmq810 2458017 2458016 2026-09-16T13:09:27Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1342231|SI: Unset all filters on links to cases (T434530)]], [[gerrit:1342234|SI: Unset all filters on links to cases (T434530)]] 2458017 wikitext text/x-wiki lmlm9pfyj73ldk7a0e8lp6tyidkhoka 2458018 2458017 2026-09-16T13:11:38Z Stashbot 7414 slyngshede@cumin1003: conftool action : set/pooled=no; selector: name=cp5026.eqsin.wmnet 2458018 wikitext text/x-wiki cg6v013sgk3orqj8985gq360kso6p0m 2458019 2458018 2026-09-16T13:13:46Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1342231|SI: Unset all filters on links to cases (T434530)]], [[gerrit:1342234|SI: Unset all filters on links to cases (T434530)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2458019 wikitext text/x-wiki cfnjq8ji2ou7ijunxns8aq1qbrt8hza 2458020 2458019 2026-09-16T13:15:06Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply 2458020 wikitext text/x-wiki dvafddz7quca04ibhjr8e0ryzse5yp2 2458022 2458020 2026-09-16T13:15:35Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply 2458022 wikitext text/x-wiki eii38kzby90v6005j8yxcpv79zg81il 2458023 2458022 2026-09-16T13:18:04Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2458023 wikitext text/x-wiki 48c1zqw9be2oaqmv4p9fl82m94yt314 2458024 2458023 2026-09-16T13:19:13Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.provision for host cp6002.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2458024 wikitext text/x-wiki t9x1hc1rkcx8thvjtpwmoizx7mkntov 2458025 2458024 2026-09-16T13:19:25Z Stashbot 7414 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org 2458025 wikitext text/x-wiki ss6wj7yuhaoj5iob0eg6usv3po2x37o 2458026 2458025 2026-09-16T13:19:31Z Stashbot 7414 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org 2458026 wikitext text/x-wiki 6d3v9d9r8npsw76c8uhw8ug4e0f0kjz 2458027 2458026 2026-09-16T13:22:37Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342231|SI: Unset all filters on links to cases (T434530)]], [[gerrit:1342234|SI: Unset all filters on links to cases (T434530)]] (duration: 13m 10s) 2458027 wikitext text/x-wiki 751krlfvfsgcgig3ormklx8s8lzwx60 2458028 2458027 2026-09-16T13:22:56Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.hosts.provision for host cp5026.mgmt.eqsin.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2458028 wikitext text/x-wiki lsiwfifjdzpy0s47j2429788utzvml0 2458029 2458028 2026-09-16T13:24:06Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2458029 wikitext text/x-wiki r6zylcjdgy8t39n5s3bilfb6mgq8448 2458030 2458029 2026-09-16T13:24:36Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2458030 wikitext text/x-wiki 4mljfy2w26ncgd8srlvwpcttg2uk9rw 2458031 2458030 2026-09-16T13:25:32Z Stashbot 7414 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dse-k8s-etcd[1001-1003].eqiad.wmnet with reason: Maintenance to increase vCPUS T438084 2458031 wikitext text/x-wiki 9lgiodqaael1pmo2ev6iv9q7slitxoz 2458033 2458031 2026-09-16T13:29:45Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp6002.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2458033 wikitext text/x-wiki jdn8xdxf5glgdcv92rfbf4dw736e344 2458034 2458033 2026-09-16T13:33:35Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5026.mgmt.eqsin.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2458034 wikitext text/x-wiki mzjppkkxlzdwnxu1jlq0ggfss1erz4e 2458035 2458034 2026-09-16T13:33:47Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2458035 wikitext text/x-wiki j5sby0vgqhzngq7ei9vea2vtejv9n41 2458036 2458035 2026-09-16T13:34:13Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2458036 wikitext text/x-wiki k8ukmgcczrlz50p4bgihzg2u5b7lhzk 2458037 2458036 2026-09-16T13:34:18Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp5026.eqsin.wmnet with OS trixie 2458037 wikitext text/x-wiki pv2xwo0uzwzsqedp7klrfe5ttsomabv 2458039 2458037 2026-09-16T13:37:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-etcd1003.eqiad.wmnet 2458039 wikitext text/x-wiki dsklpw6suo4qcqi5ikjl8d4mxp7al03 2458040 2458039 2026-09-16T13:37:15Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM dse-k8s-etcd1003.eqiad.wmnet 2458040 wikitext text/x-wiki oklm02wqc0w703um9nde3exkyhy9t49 2458041 2458040 2026-09-16T13:37:27Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-etcd1003.eqiad.wmnet 2458041 wikitext text/x-wiki tstcmm2ijz46128ry6mtmkztfyvdu9o 2458042 2458041 2026-09-16T13:38:07Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp6002.drmrs.wmnet with OS trixie 2458042 wikitext text/x-wiki jelua096ha1jcpr5445fb83necieoso 2458044 2458042 2026-09-16T13:41:12Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM dse-k8s-etcd1003.eqiad.wmnet 2458044 wikitext text/x-wiki pcl5tkptrwcotlpevfcoyv49yecxvmd 2458045 2458044 2026-09-16T13:41:26Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-etcd1002.eqiad.wmnet 2458045 wikitext text/x-wiki f1xh1pfpkf6zptm6ecnsb9n3chhjius 2458047 2458045 2026-09-16T13:45:17Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM dse-k8s-etcd1002.eqiad.wmnet 2458047 wikitext text/x-wiki 3e71a7q8ui0qoctsqa4tilati3x1qqx 2458049 2458047 2026-09-16T13:46:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-etcd1001.eqiad.wmnet 2458049 wikitext text/x-wiki js5e6en0fk5o25iggoo4oipajym74wa 2458050 2458049 2026-09-16T13:46:57Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM dse-k8s-etcd1001.eqiad.wmnet 2458050 wikitext text/x-wiki ps1zk5rnyltecy8u3iavvjqv9wyh723 2458051 2458050 2026-09-16T13:48:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.remove-downtime for dse-k8s-etcd[1001-1003].eqiad.wmnet 2458051 wikitext text/x-wiki ecsl0miy14hguls53ja3k3lck1z6orb 2458052 2458051 2026-09-16T13:48:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dse-k8s-etcd[1001-1003].eqiad.wmnet 2458052 wikitext text/x-wiki bx8ui5k55nogn2amztveh4kk44moiax 2458053 2458052 2026-09-16T13:52:14Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2458053 wikitext text/x-wiki 3z7p1rr5t4es83bghhdtuf2zzptmwwu 2458055 2458053 2026-09-16T13:53:31Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2458055 wikitext text/x-wiki 5yudmhdih8cbmngij7j8kjxwnte022s 2458056 2458055 2026-09-16T13:54:39Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp6002.drmrs.wmnet with reason: host reimage 2458056 wikitext text/x-wiki rqfmmrlu3l7cxl59cru1kfnejx94zc7 2458057 2458056 2026-09-16T13:56:44Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5026.eqsin.wmnet with reason: host reimage 2458057 wikitext text/x-wiki csm2r3zx2ct8tatypvt53o19tcy19k4 2458058 2458057 2026-09-16T13:59:26Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6002.drmrs.wmnet with reason: host reimage 2458058 wikitext text/x-wiki qursgg1s8hqwkw5jul58hoo7hgxe0gq 2458059 2458058 2026-09-16T14:00:57Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2458059 wikitext text/x-wiki pfxgbags0hynu86hvr7eyxmcmthixni 2458060 2458059 2026-09-16T14:01:30Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2458060 wikitext text/x-wiki rih03dcop7at1ljc911zj2l3ss6d5fp 2458061 2458060 2026-09-16T14:02:08Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2458061 wikitext text/x-wiki p0841mnoq8b9rt3orcmes74kl2mqi5i 2458062 2458061 2026-09-16T14:02:19Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2458062 wikitext text/x-wiki hlgcvjap9sd4p561in1i5bjm70su8l0 2458063 2458062 2026-09-16T14:02:31Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2458063 wikitext text/x-wiki m3ravmidc1eadi298o6h8eucqimc6ws 2458065 2458063 2026-09-16T14:03:33Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5026.eqsin.wmnet with reason: host reimage 2458065 wikitext text/x-wiki pn9w5entm0mu1z8fpj412nce6pnpxr4 2458066 2458065 2026-09-16T14:07:34Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1342008|abstractwiki: Add three new articles per community advice to show off the feature (T434227)]] 2458066 wikitext text/x-wiki f122lx1czjqk271j8z1rqef188ko6qy 2458067 2458066 2026-09-16T14:08:24Z Stashbot 7414 Amir1: dropped links tables on db1238 (T437278) 2458067 wikitext text/x-wiki i9vsamtkhnntyj4icjlh6wiz6247vdq 2458068 2458067 2026-09-16T14:09:49Z Stashbot 7414 Amir1: dropped links tables on db2237 (T437278) 2458068 wikitext text/x-wiki aondsw35khah5neii7kygsmw9s5yrb4 2458069 2458068 2026-09-16T14:10:03Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync 2458069 wikitext text/x-wiki p9dpkufyv1ecvoy8asxeov8ehwy0bj0 2458070 2458069 2026-09-16T14:10:06Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync 2458070 wikitext text/x-wiki e3i3pbgjo192o7s98pv06pb6oq6lhk5 2458071 2458070 2026-09-16T14:10:17Z Stashbot 7414 jforrester@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.19,1.47.0-wmf.20,next --multiversion-image-basename docker-registry.discovery.wmnet/restricte 2458071 wikitext text/x-wiki ilv00egefqbn7w8p7isgannuzlslv1z 2458072 2458071 2026-09-16T14:10:33Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync 2458072 wikitext text/x-wiki h9427hnr4oin9qrwjz52gk286vss9y7 2458073 2458072 2026-09-16T14:10:36Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync 2458073 wikitext text/x-wiki 5n0675r9kjdj05tcvtv9iax2fxz9at5 2458074 2458073 2026-09-16T14:12:46Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1342008|abstractwiki: Add three new articles per community advice to show off the feature (T434227)]] 2458074 wikitext text/x-wiki idcsk5zcssspl9dejujgofstdgjiemz 2458075 2458074 2026-09-16T14:12:50Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2458075 wikitext text/x-wiki lhkqc4a385cvbxqodwevgw6kqxv9cxf 2458076 2458075 2026-09-16T14:13:01Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2458076 wikitext text/x-wiki irflwvkewejxfsp6q6ogjnd7xs9bcha 2458077 2458076 2026-09-16T14:14:10Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2458077 wikitext text/x-wiki ay6ztzzggc5mgqqbe4iv3jl2a8dcl1m 2458078 2458077 2026-09-16T14:14:20Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2458078 wikitext text/x-wiki r7sk6o3zjtjo56a2wmn7kvqm7c77axm 2458079 2458078 2026-09-16T14:14:25Z Stashbot 7414 jforrester@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.19,1.47.0-wmf.20,next --multiversion-image-basename docker-registry.discovery.wmnet/restricte 2458079 wikitext text/x-wiki gohwi34vvh1u7oo54yhfnfulcimlnii 2458080 2458079 2026-09-16T14:24:13Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync 2458080 wikitext text/x-wiki dz9lu7gjpesdu5phcvpdctwrjtd48qw 2458081 2458080 2026-09-16T14:24:29Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2458081 wikitext text/x-wiki bditnfun0wesmap05frfbx04t2m5sq0 2458082 2458081 2026-09-16T14:24:32Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2458082 wikitext text/x-wiki 3gkppkfetpe3ywbkqwurhb5pqxnmy7s 2458083 2458082 2026-09-16T14:24:40Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync 2458083 wikitext text/x-wiki jhan8d3difxadl5du5ugkfvpskiix6g 2458084 2458083 2026-09-16T14:29:57Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6002.drmrs.wmnet with OS trixie 2458084 wikitext text/x-wiki ih4f4tjokaiscd8hb9wqlcfqvnq6e9s 2458085 2458084 2026-09-16T14:32:17Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=1; selector: name=cp6002.drmrs.wmnet,service=cdn 2458085 wikitext text/x-wiki 7dkp47l6inljnfuup14810xqyfah8n6 2458086 2458085 2026-09-16T14:32:20Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=100; selector: name=cp6002.drmrs.wmnet,service=ats-be 2458086 wikitext text/x-wiki srcigc115jokqd7rvcyj1hwi0nl3slu 2458088 2458086 2026-09-16T14:33:47Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6002.drmrs.wmnet 2458088 wikitext text/x-wiki ce04za3la2m4r0h89ybde8yi5efx51l 2458089 2458088 2026-09-16T14:34:46Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2458089 wikitext text/x-wiki tdhmwa283t7ispts4s2g7e7z6nqskf0 2458090 2458089 2026-09-16T14:35:05Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2458090 wikitext text/x-wiki ml5l8wh977wv0ozfvr2dhbwczc9ya8r 2458091 2458090 2026-09-16T14:36:00Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2458091 wikitext text/x-wiki ma3gzqs9g80cdk0pac28dahqp0lk6uu 2458092 2458091 2026-09-16T14:42:04Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7003.magru.wmnet with OS trixie 2458092 wikitext text/x-wiki mdp51jrp3znjvnvm0iyz9j3nunx6rkv 2458093 2458092 2026-09-16T14:44:21Z Stashbot 7414 moritzm: installing python-filelock security updates 2458093 wikitext text/x-wiki 830d0527ldloxyjr7gh5tk79cwesadt 2458095 2458093 2026-09-16T14:45:58Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5026.eqsin.wmnet with OS trixie 2458095 wikitext text/x-wiki o8cky1q0y670iis5l95sxrniajm73k4 2458098 2458095 2026-09-16T14:47:24Z Stashbot 7414 slyngshede@cumin1003: conftool action : set/weight=1; selector: name=cp5026.eqsin.wmnet 2458098 wikitext text/x-wiki 2hnge1a4qtj6by90a8zs5ct7shw4pxl 2458099 2458098 2026-09-16T14:49:30Z Stashbot 7414 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp5026.eqsin.wmnet 2458099 wikitext text/x-wiki 5gpg3hbeqxeiv3ans7qa30hee1263qe 2458102 2458099 2026-09-16T14:50:52Z Stashbot 7414 moritzm: installing apache2 security updates 2458102 wikitext text/x-wiki c5lj4f278qbj4wbrfsqn740b3a199bo 2458109 2458102 2026-09-16T15:03:33Z Stashbot 7414 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1342279|Use parser output value instead of status (T438154)]] 2458109 wikitext text/x-wiki 6xlsu7q34j410e45t8yh88vbfrfeqhg 2458116 2458109 2026-09-16T15:14:23Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7003.magru.wmnet with reason: host reimage 2458116 wikitext text/x-wiki 9kmobw21mrna8j4khux9t4qfexwy0b1 2458117 2458116 2026-09-16T15:18:36Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7003.magru.wmnet with reason: host reimage 2458117 wikitext text/x-wiki gdeux8tvvdxgywipe67y87l7iflos16 2458118 2458117 2026-09-16T15:19:42Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2458118 wikitext text/x-wiki 3cwy08ruvtpp0mo1wnu4y58arujc0pn 2458120 2458118 2026-09-16T15:23:48Z Stashbot 7414 jnuche@deploy1003: jnuche, jforrester: Backport for [[gerrit:1342279|Use parser output value instead of status (T438154)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2458120 wikitext text/x-wiki bo6de920xhympbu1hah1cki5gw54k7y 2458121 2458120 2026-09-16T15:24:22Z Stashbot 7414 jnuche@deploy1003: jnuche, jforrester: Continuing with deployment 2458121 wikitext text/x-wiki dvn1aupvekam9k1alfq567e7mpb8tu7 2458123 2458121 2026-09-16T15:35:26Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-cron: apply 2458123 wikitext text/x-wiki kjamrru6x10kputd63ulx26ysd0zb8z 2458124 2458123 2026-09-16T15:35:39Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-cron: apply 2458124 wikitext text/x-wiki i8ice2rtm9c389iucg78cev96gfvf3n 2458125 2458124 2026-09-16T15:36:55Z Stashbot 7414 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342279|Use parser output value instead of status (T438154)]] (duration: 33m 21s) 2458125 wikitext text/x-wiki 6czmntcdd6igma8tn96n44kl5jiz92k 2458128 2458125 2026-09-16T15:41:50Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply 2458128 wikitext text/x-wiki tkmlsi5hxjtugh440ywzamkvhlwukib 2458129 2458128 2026-09-16T15:42:09Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7003.magru.wmnet with OS trixie 2458129 wikitext text/x-wiki mr7pq7k9nne6wacmolxyaq718zdyvgn 2458130 2458129 2026-09-16T15:42:16Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply 2458130 wikitext text/x-wiki te1ce3rs45wxbpodydarsn6p9mxeire 2458131 2458130 2026-09-16T15:42:26Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-cron: apply 2458131 wikitext text/x-wiki ok2ssmt81wtivr6w87puw74niirbr5i 2458132 2458131 2026-09-16T15:42:31Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-cron: apply 2458132 wikitext text/x-wiki lghoi4twe3ay4t6yrium48tdikrm1zp 2458135 2458132 2026-09-16T16:19:41Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2458135 wikitext text/x-wiki 3bqrmk8csatdzs786d3swxwcvdtlzor 2458141 2458135 2026-09-16T16:44:23Z Stashbot 7414 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7003 2458141 wikitext text/x-wiki 1mx45hnit2tutc1vqo4y1sj0e2sj51s 2458142 2458141 2026-09-16T16:46:31Z Stashbot 7414 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7003.magru.wmnet 2458142 wikitext text/x-wiki kf6t9qujt5ks19qdj0ap2lqqhea7jcr 2458143 2458142 2026-09-16T16:52:34Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2458143 wikitext text/x-wiki 62te89w2kipab7vkukd6e0n2tnz02v3 2458151 2458143 2026-09-16T17:05:51Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458151 wikitext text/x-wiki lcuhn6wioscl0k2ks6zcp08mp7tvzrc 2458152 2458151 2026-09-16T17:05:53Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458152 wikitext text/x-wiki 89o8ufefnspg8ahvvpiv154g2ds28jv 2458153 2458152 2026-09-16T17:07:15Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2458153 wikitext text/x-wiki qvool5kwfub9j80yj113ggd5fqscq1h 2458154 2458153 2026-09-16T17:07:17Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage 2458154 wikitext text/x-wiki kwyru9tfqbs11fozjcl503l6kavrzrl 2458157 2458154 2026-09-16T17:09:40Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2458157 wikitext text/x-wiki ho9vwsx6l3cz0v6ckxhlo2296x4hxkh 2458158 2458157 2026-09-16T17:10:01Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458158 wikitext text/x-wiki pyd2d41j436nx99p287dqu42kfvil0x 2458159 2458158 2026-09-16T17:10:03Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458159 wikitext text/x-wiki cx9s5ge3v3liaq06sxm9hcpmf1s82ob 2458160 2458159 2026-09-16T17:10:32Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458160 wikitext text/x-wiki adumsati2yebshimse0xkifse3ykiab 2458161 2458160 2026-09-16T17:10:33Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458161 wikitext text/x-wiki d2k9ky98uasmwklmuwavpzr62pvzzb0 2458162 2458161 2026-09-16T17:13:22Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage 2458162 wikitext text/x-wiki amfmf04ews9flx5i7ga8f2wwx3qf8iv 2458163 2458162 2026-09-16T17:14:19Z Stashbot 7414 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458163 wikitext text/x-wiki oz66w5tc7zzuebphirqpem54o1kzs0x 2458164 2458163 2026-09-16T17:20:45Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2458164 wikitext text/x-wiki 7ccav4h3tfit62lzj3d34ud4545awzc 2458165 2458164 2026-09-16T17:20:55Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2458165 wikitext text/x-wiki dhtfsrxmyerkxe5r4t6egh9ro1ni9ls 2458166 2458165 2026-09-16T17:23:08Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458166 wikitext text/x-wiki 1tjbjf7a6p7ozyu1sz87cb5dhz2vq6z 2458167 2458166 2026-09-16T17:23:09Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458167 wikitext text/x-wiki poqre0w6dw6wqvp8td0ay5klb65rgld 2458170 2458167 2026-09-16T17:27:32Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458170 wikitext text/x-wiki hc46dq3b0gdooyu59rndxfj73beqtdn 2458171 2458170 2026-09-16T17:27:34Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458171 wikitext text/x-wiki fe1gqurijedwrbsak8nsbh90jd1g4tj 2458172 2458171 2026-09-16T17:29:50Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2458172 wikitext text/x-wiki 3vd00p0sz4r7i754ydr3mjolyucjvm5 2458173 2458172 2026-09-16T17:30:36Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458173 wikitext text/x-wiki dbqfmfr4hd0q0xnztcy7oe91i2s5ubr 2458174 2458173 2026-09-16T17:30:37Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458174 wikitext text/x-wiki 86t1o139swkob3admzkfsgcfi6s9fgx 2458175 2458174 2026-09-16T17:35:38Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2458175 wikitext text/x-wiki qe2a00cz3m5nfwuzrk6nxbzxdfbfr60 2458176 2458175 2026-09-16T17:35:40Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie 2458176 wikitext text/x-wiki huub9k4z5nsx4fvwa6g6775khzs0k0n 2458177 2458176 2026-09-16T17:44:34Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458177 wikitext text/x-wiki dgglt5v1mku4u3ye9fyi4hlhvjbwj8p 2458178 2458177 2026-09-16T17:44:36Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458178 wikitext text/x-wiki g1flsnli65zewkbn9xeha0a1mo2h5uc 2458179 2458178 2026-09-16T17:45:20Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458179 wikitext text/x-wiki mghrmydpj5wiuit0ikqo7k9i7gaqypg 2458180 2458179 2026-09-16T17:45:22Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458180 wikitext text/x-wiki seufsygpp2muy0f3n3qhujjsj3f3hkd 2458181 2458180 2026-09-16T17:46:14Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2458181 wikitext text/x-wiki 4dl1paxwebuqnc9ksdhf3tcuwiduul2 2458182 2458181 2026-09-16T17:46:44Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2458182 wikitext text/x-wiki 91xlk9fm6oa6rwfscgpz14y36eh0fds 2458183 2458182 2026-09-16T17:51:17Z Stashbot 7414 vriley@cumin2003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458183 wikitext text/x-wiki 7ftntpjos0vm0qp2chz2n6zho1ao37l 2458184 2458183 2026-09-16T18:04:03Z Stashbot 7414 vriley@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458184 wikitext text/x-wiki 25xm7jkimt1u7ijgu8cs1v5adb0y4u4 2458185 2458184 2026-09-16T18:12:16Z Stashbot 7414 cdanis@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "fix known-client creation - cdanis@cumin1003" 2458185 wikitext text/x-wiki dns1w5ua4db4hqr5xgl37wztre7gfci 2458186 2458185 2026-09-16T18:12:18Z Stashbot 7414 cdanis@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: fix known-client creation - cdanis@cumin1003 2458186 wikitext text/x-wiki 8wm1s5m1s1jxwfnsk08eqqnhgtg9f31 2458187 2458186 2026-09-16T18:13:12Z Stashbot 7414 cdanis@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: fix known-client creation - cdanis@cumin1003 2458187 wikitext text/x-wiki dlppzjg4ehtoddgfurcrao1wn25qk7j 2458188 2458187 2026-09-16T18:13:13Z Stashbot 7414 cdanis@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "fix known-client creation - cdanis@cumin1003" 2458188 wikitext text/x-wiki 61tzeoi4wvgu0p6h1b0p5mmenqkgbji 2458189 2458188 2026-09-16T18:16:29Z Stashbot 7414 dzahn@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2458189 wikitext text/x-wiki 0wbpl00lstca0asdvti0gerao1y48cv 2458190 2458189 2026-09-16T18:17:07Z Stashbot 7414 dzahn@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2458190 wikitext text/x-wiki 8gzaevt5aoqeap5z50t5ylkwjp9is8f 2458191 2458190 2026-09-16T18:17:25Z Stashbot 7414 dzahn@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2458191 wikitext text/x-wiki 1x8d3ecym69zab2idbij1cgxhf6ry0w 2458192 2458191 2026-09-16T18:17:47Z Stashbot 7414 dzahn@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2458192 wikitext text/x-wiki ro7mh90g0runa8wofy2jzf23g4zeipp 2458193 2458192 2026-09-16T18:18:23Z Stashbot 7414 mutante: k8s/miscweb: admin_ng deploy: creating namespace for attribution.wikimedia.org T437635 2458193 wikitext text/x-wiki brzvz24e0ti18fcml1447q8ibjlc94a 2458194 2458193 2026-09-16T18:18:38Z Stashbot 7414 dzahn@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2458194 wikitext text/x-wiki mutcpmbnileothmjb22h7f5ggfazefu 2458195 2458194 2026-09-16T18:20:36Z Stashbot 7414 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2458195 wikitext text/x-wiki rji1xxjf6ivkwa3wduje4xysm2mw3kq 2458196 2458195 2026-09-16T18:20:49Z Stashbot 7414 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2458196 wikitext text/x-wiki 99rjp44yvf2wnbq6jei7frfjs4donxi 2458197 2458196 2026-09-16T18:21:26Z Stashbot 7414 robh@cumin1003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458197 wikitext text/x-wiki 1a69gt0jllvfqanuq9jfbyqs3n833qp 2458198 2458197 2026-09-16T18:21:27Z Stashbot 7414 robh@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458198 wikitext text/x-wiki jxwge9lqo6tku8s0shrqnu2qrmor0v4 2458199 2458198 2026-09-16T18:21:46Z Stashbot 7414 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2458199 wikitext text/x-wiki h70grlqraew8djinzxmfnmcv7bf2xku 2458200 2458199 2026-09-16T18:22:12Z Stashbot 7414 robh@cumin2003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458200 wikitext text/x-wiki 4joa442iaiki1ieqjn4onuj46iubofv 2458201 2458200 2026-09-16T18:26:52Z Stashbot 7414 robh@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458201 wikitext text/x-wiki k9thfosjwqhk803avlf3eoz2mriw148 2458202 2458201 2026-09-16T18:35:40Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2458202 wikitext text/x-wiki ruh8yase6a2uz5siztj92q41gjaryo0 2458203 2458202 2026-09-16T18:37:44Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2458203 wikitext text/x-wiki 7ybimo8ylkb0pl7i6w237ipi8cc4db2 2458204 2458203 2026-09-16T18:42:53Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2458204 wikitext text/x-wiki 2hyeic64np0pgesrox0lsntyrwe53bg 2458205 2458204 2026-09-16T18:45:00Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2458205 wikitext text/x-wiki j4v7zpbln8uzwt93m4ulpz39wyyfny2 2458206 2458205 2026-09-16T18:55:22Z Stashbot 7414 dwisehaupt@dns1005: START - running authdns-update 2458206 wikitext text/x-wiki b2qdl61ul2f8p3wnxxg51fwu3m92e35 2458207 2458206 2026-09-16T18:55:33Z Stashbot 7414 vriley@cumin2003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458207 wikitext text/x-wiki f72z2kihcheaize4y00krbgj3eemx37 2458208 2458207 2026-09-16T18:57:40Z Stashbot 7414 dwisehaupt@dns1005: END - running authdns-update 2458208 wikitext text/x-wiki gvj4671ro1s1ewgtu4ku4jmqavmf0s1 2458210 2458208 2026-09-16T19:06:24Z Stashbot 7414 vriley@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2458210 wikitext text/x-wiki 8b7lwezfvwpjsj8mtlsuws27cx2aqqn 2458211 2458210 2026-09-16T19:18:02Z Stashbot 7414 dwisehaupt@dns1005: START - running authdns-update 2458211 wikitext text/x-wiki b152awkyen7jwqy8m9qygoynix3ndhb 2458212 2458211 2026-09-16T19:20:19Z Stashbot 7414 dwisehaupt@dns1005: END - running authdns-update 2458212 wikitext text/x-wiki jsa5jfe3xypoie69yhkjufxjhhc3bwn 2458228 2458212 2026-09-16T20:13:29Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1342339|Reapply "Tell VisualEditor about the app web edit tags", modified]] 2458228 wikitext text/x-wiki 1b3x0szcu51iiri3zfnr9sjq2gi8omg 2458229 2458228 2026-09-16T20:33:10Z Stashbot 7414 kemayo@deploy1003: cklimas, kemayo: Backport for [[gerrit:1342339|Reapply "Tell VisualEditor about the app web edit tags", modified]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2458229 wikitext text/x-wiki p72dxrd185bh5xtfzqgeutxph8gcb5a 2458230 2458229 2026-09-16T20:37:09Z Stashbot 7414 kemayo@deploy1003: cklimas, kemayo: Continuing with deployment 2458230 wikitext text/x-wiki er2b2d19piiwtzu9gb044keiovu3dfy 2458231 2458230 2026-09-16T20:49:18Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342339|Reapply "Tell VisualEditor about the app web edit tags", modified]] (duration: 35m 55s) 2458231 wikitext text/x-wiki j586gkr9siy6v8glb1eqe7338631gw4 2458233 2458231 2026-09-16T21:02:23Z Stashbot 7414 vriley@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be1098.eqiad.wmnet with OS trixie 2458233 wikitext text/x-wiki 3z0tgy1x97z2or6mupun7cufgbsqd9m 2458234 2458233 2026-09-16T21:08:14Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2458234 wikitext text/x-wiki 8d8vjpo1dv376jot7rqdnd2q83hpi7w 2458235 2458234 2026-09-16T21:08:39Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2458235 wikitext text/x-wiki 4nu1rc31ys5p8wiwew5t78i997u2g1d 2458236 2458235 2026-09-16T21:09:14Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2458236 wikitext text/x-wiki inrbag7khmc8ysbu1s325z0a8acnzry 2458237 2458236 2026-09-16T21:09:50Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2458237 wikitext text/x-wiki 90yhx6lq9g3vobp331e1ej3wyain6oj 2458238 2458237 2026-09-16T21:09:57Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2458238 wikitext text/x-wiki l0euk5pdwcm1fwviqjpm1599fts75ue 2458239 2458238 2026-09-16T21:10:33Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2458239 wikitext text/x-wiki dys9givu18rx9bt4rl4febo884o8uee 2458240 2458239 2026-09-16T21:17:41Z Stashbot 7414 vriley@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1098.eqiad.wmnet with reason: host reimage 2458240 wikitext text/x-wiki rpktai5e3ge6zcm083ooi7ce42e8a4i 2458241 2458240 2026-09-16T21:18:57Z Stashbot 7414 rzl@deploy1003: Started scap sync-world: Backport for [[gerrit:1342089|Depool poolcounter[1006,2005] for reboot (T435163)]] 2458241 wikitext text/x-wiki oxp0f4owu23g8019akgj8znz8p74qje 2458242 2458241 2026-09-16T21:23:46Z Stashbot 7414 vriley@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1098.eqiad.wmnet with reason: host reimage 2458242 wikitext text/x-wiki ioh87qk99no2bkdttwxkuulki2jk3gh 2458243 2458242 2026-09-16T21:25:13Z Stashbot 7414 rzl@deploy1003: rzl: Backport for [[gerrit:1342089|Depool poolcounter[1006,2005] for reboot (T435163)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2458243 wikitext text/x-wiki do7hp8swnb5ith0rimqyhtsj3c3mb2h 2458244 2458243 2026-09-16T21:26:11Z Stashbot 7414 rzl@deploy1003: rzl: Continuing with deployment 2458244 wikitext text/x-wiki nd91175fxj0zy53xvkbu819sepl750u 2458245 2458244 2026-09-16T21:32:51Z Stashbot 7414 rzl@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342089|Depool poolcounter[1006,2005] for reboot (T435163)]] (duration: 13m 53s) 2458245 wikitext text/x-wiki m3zovnos86dgj7h6446ow28lgwm0szx 2458246 2458245 2026-09-16T21:35:30Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet 2458246 wikitext text/x-wiki 7qqo42xtc4z86mk20y0ao54iev3l5k6 2458247 2458246 2026-09-16T21:37:48Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2458247 wikitext text/x-wiki oy4brhkmr44mtsex9rhv18ogmwdsvuo 2458248 2458247 2026-09-16T21:38:33Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2458248 wikitext text/x-wiki bmso1tgcc84g95lwimh490vyj0kigkt 2458249 2458248 2026-09-16T21:38:43Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2458249 wikitext text/x-wiki b9xt5d3030kcvuam3fccj3uho7oucpy 2458250 2458249 2026-09-16T21:39:20Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet 2458250 wikitext text/x-wiki eronpszipzp2v43vnw5ibk9loxqa1b3 2458252 2458250 2026-09-16T21:39:39Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet 2458252 wikitext text/x-wiki l6cnmkbyflmr4z9wkzc24qzhi7ahaoy 2458253 2458252 2026-09-16T21:40:18Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2458253 wikitext text/x-wiki rrqanw8mjwbmwkr55h7z3xapv8ykgqf 2458254 2458253 2026-09-16T21:40:29Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2458254 wikitext text/x-wiki m33evyoq8hb1az9cxvqa7m65vogwfe3 2458255 2458254 2026-09-16T21:40:55Z Stashbot 7414 vriley@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin2003" 2458255 wikitext text/x-wiki 8go6u3uw5rgci9vd6h0gdwdox4s9hnd 2458256 2458255 2026-09-16T21:41:15Z Stashbot 7414 vriley@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin2003" 2458256 wikitext text/x-wiki m3r0reyehn86j2a62b1gyul7h0ysbzx 2458257 2458256 2026-09-16T21:41:16Z Stashbot 7414 vriley@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1098.eqiad.wmnet with OS trixie 2458257 wikitext text/x-wiki 7hsx69rc2wj74wtogjoryh08dxfsc7l 2458258 2458257 2026-09-16T21:42:05Z Stashbot 7414 vriley@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be1099.eqiad.wmnet with OS trixie 2458258 wikitext text/x-wiki sj7k1tvzg9hmn2i1bn4f1g82ie1k5vp 2458259 2458258 2026-09-16T21:43:28Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet 2458259 wikitext text/x-wiki d6cym3ejxu55ydyckhmb3kcp0xta3ar 2458260 2458259 2026-09-16T21:46:56Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2458260 wikitext text/x-wiki 3o1h7ju36nydt39e0rgperhryysrpcp 2458261 2458260 2026-09-16T21:47:20Z Stashbot 7414 rzl@deploy1003: Started scap sync-world: Backport for [[gerrit:1342090|Repool poolcounter[1006,2005]; depool poolcounter[1007,2006] for reboot (T435163)]] 2458261 wikitext text/x-wiki magaa3nt1sngy8chwmf3nr7gua7fxz5 2458265 2458261 2026-09-16T21:51:50Z Stashbot 7414 rzl@deploy1003: rzl: Backport for [[gerrit:1342090|Repool poolcounter[1006,2005]; depool poolcounter[1007,2006] for reboot (T435163)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2458265 wikitext text/x-wiki 19itqpbvpswjq32d6ir6e4ksvtvjc5b 2458266 2458265 2026-09-16T21:52:33Z Stashbot 7414 rzl@deploy1003: rzl: Continuing with deployment 2458266 wikitext text/x-wiki g87occ2ys6x3eozvhx4uq7ycsp7nyql 2458267 2458266 2026-09-16T21:56:59Z Stashbot 7414 rzl@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342090|Repool poolcounter[1006,2005]; depool poolcounter[1007,2006] for reboot (T435163)]] (duration: 09m 39s) 2458267 wikitext text/x-wiki bv2v7kajh626vsvsrugtmklzvo5x2w4 2458268 2458267 2026-09-16T22:02:03Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet 2458268 wikitext text/x-wiki j8wzwzvosieh65vb75zr3vj1zyvcnpe 2458269 2458268 2026-09-16T22:06:03Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet 2458269 wikitext text/x-wiki b3wxeggi9ctgm584w7z2lpsuhkfbsyu 2458270 2458269 2026-09-16T22:06:38Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet 2458270 wikitext text/x-wiki m491ejx4yggxg8s13k3ca5qgo3ao71u 2458271 2458270 2026-09-16T22:10:16Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet 2458271 wikitext text/x-wiki fk7qk1ymlihi7f3is1nwddds1qdreez 2458272 2458271 2026-09-16T22:12:30Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342383|DonorIdentification: Confirm before unlinking donor status in preferences (T436698)]], [[gerrit:1342385|Make learn more link to new window (T438252)]] 2458272 wikitext text/x-wiki op70n8uv6xnaw2xs4z2urln0ql5if5a 2458274 2458272 2026-09-16T22:33:10Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1342383|DonorIdentification: Confirm before unlinking donor status in preferences (T436698)]], [[gerrit:1342385|Make learn more link to new window (T438252)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2458274 wikitext text/x-wiki lgl3ypf85snne8q9pl37mmafvae3syi 2458275 2458274 2026-09-16T22:35:30Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2458275 wikitext text/x-wiki f215nys10chnjwadry2a56y7iwcpoin 2458276 2458275 2026-09-16T22:47:51Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342383|DonorIdentification: Confirm before unlinking donor status in preferences (T436698)]], [[gerrit:1342385|Make learn more link to new window (T438252)]] (duration: 35m 21s) 2458276 wikitext text/x-wiki 1vk2ze1u46es9m0ecksdyn0u0fvxfas 2458277 2458276 2026-09-16T22:50:44Z Stashbot 7414 rzl@deploy1003: Started scap sync-world: Backport for [[gerrit:1342091|Repool poolcounter[1007,2006] (T435163)]] 2458277 wikitext text/x-wiki ivuh710fe98nftn2wqj829v1chhd1fq 2458278 2458277 2026-09-16T23:10:06Z Stashbot 7414 rzl: rzl@deploy1003 Finished scap sync-world: Backport for [[gerrit:1342091|Repool poolcounter[1007,2006] (T435163)]] (duration: 11m 09s) 2458278 wikitext text/x-wiki dks979kppm0859e9kkmvq870s5phihd 2458282 2458278 2026-09-17T04:24:51Z Stashbot 7414 aokoth@cumin1004: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet 2458282 wikitext text/x-wiki h178do1iqf4d3igyq6xphg9sff4bzzg 2458283 2458282 2026-09-17T04:26:29Z Stashbot 7414 aokoth@cumin1004: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet 2458283 wikitext text/x-wiki 8t7an1vkrade3ap4kpu0zq5bh078ofx 2458286 2458283 2026-09-17T05:43:59Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.decommission 2458286 wikitext text/x-wiki 02d8t5lrkafxx0r2dsfsijy8xkg0wtn 2458287 2458286 2026-09-17T05:44:27Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts db1180.eqiad.wmnet 2458287 wikitext text/x-wiki m5jyb52r6ivng8w1uecb11p224kxmgt 2458288 2458287 2026-09-17T05:49:16Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.dns.netbox 2458288 wikitext text/x-wiki rp0z4z3hqjeummsf5xovrb09mlk84lo 2458289 2458288 2026-09-17T05:53:06Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1180.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" 2458289 wikitext text/x-wiki afr7bj1au2kyd9h1d1ix1dqfng9w97k 2458290 2458289 2026-09-17T05:53:21Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1180.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" 2458290 wikitext text/x-wiki 728leispyyhqsa9imhw1za7mj2o8lgg 2458291 2458290 2026-09-17T05:53:23Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2458291 wikitext text/x-wiki 859a7oxlzrboeymhsrg1qylrqkqsjl3 2458292 2458291 2026-09-17T05:53:24Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1180.eqiad.wmnet 2458292 wikitext text/x-wiki 7kvx1g8uw12p89wy7ah3lcyvnkcp6n6 2458293 2458292 2026-09-17T05:53:29Z Stashbot 7414 marostegui@cumin1004: Removing db1180 from zarcillo T437222 2458293 wikitext text/x-wiki q85t0w8f0ylp8pizdxs6cver9pdz6kh 2458294 2458293 2026-09-17T05:53:30Z Stashbot 7414 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) 2458294 wikitext text/x-wiki nag1be5ex2taecnq226tnr3pi6f1fe0 2458296 2458294 2026-09-17T06:07:05Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2458296 wikitext text/x-wiki 1myz56q1r3kogxe5ddw7smv196p1xb9 2458304 2458296 2026-09-17T07:10:12Z Stashbot 7414 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Jmoore111 out of all services on: 2444 hosts 2458304 wikitext text/x-wiki 25hpms5s7z5nxpjnihyc0nxjvlxexpg 2458305 2458304 2026-09-17T07:11:11Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1342398|Localisation updates from https://translatewiki.net.]], [[gerrit:1342396|Localisation updates from https://translatewiki.net.]], [[gerrit:1342395|Localisation updates from https://translatewiki.net.]], [[gerrit:1342545|Localisation updates from https://translatewiki.net.]], [[gerrit:1342546|Localisation updates from https://translatewiki.net.]], 2458305 wikitext text/x-wiki bm4toc2rx79a7k4qvjqtl7zlrr3i5fc 2458306 2458305 2026-09-17T07:18:24Z Stashbot 7414 mlitn@deploy1003: mlitn, jdlrobson: Backport for [[gerrit:1342398|Localisation updates from https://translatewiki.net.]], [[gerrit:1342396|Localisation updates from https://translatewiki.net.]], [[gerrit:1342395|Localisation updates from https://translatewiki.net.]], [[gerrit:1342545|Localisation updates from https://translatewiki.net.]], [[gerrit:1342546|Localisation updates from https://translatewiki.net.]], [[gerri 2458306 wikitext text/x-wiki eibsrdrj5k02vsjf6rywnev1q9lg2fp 2458307 2458306 2026-09-17T07:19:05Z Stashbot 7414 mlitn@deploy1003: mlitn, jdlrobson: Continuing with deployment 2458307 wikitext text/x-wiki hrt75pqb89m4dxb05bimp7d7z95j2tc 2458308 2458307 2026-09-17T07:24:22Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342398|Localisation updates from https://translatewiki.net.]], [[gerrit:1342396|Localisation updates from https://translatewiki.net.]], [[gerrit:1342395|Localisation updates from https://translatewiki.net.]], [[gerrit:1342545|Localisation updates from https://translatewiki.net.]], [[gerrit:1342546|Localisation updates from https://translatewiki.net.]], 2458308 wikitext text/x-wiki 0jtzu6h1mcy1574u9p7mnbbvgfhv2as 2458309 2458308 2026-09-17T07:25:14Z Stashbot 7414 XioNoX: Manually install gnmic 0.49.0 on netflow1004 - T438291 2458309 wikitext text/x-wiki 8rxf3xrkwjcqn2lawho97hft2bkyvso 2458311 2458309 2026-09-17T07:48:58Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@8eaca67] (releasing): T438205 to backup host 2458311 wikitext text/x-wiki dr0tavhytqrfnso161m1juijjgva4gw 2458312 2458311 2026-09-17T07:49:26Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@8eaca67] (releasing): T438205 to backup host (duration: 00m 47s) 2458312 wikitext text/x-wiki smhh39mfu7gb63nmf3pqrasinr8dmv9 2458313 2458312 2026-09-17T07:52:21Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@8eaca67] (releasing): T438205 to prod host 2458313 wikitext text/x-wiki 9c32kaq8rd5p381959dp58h2rsvhwyr 2458314 2458313 2026-09-17T07:52:59Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@8eaca67] (releasing): T438205 to prod host (duration: 00m 44s) 2458314 wikitext text/x-wiki epuwx2vmd0jga46caah6q3e1bk2h200 2458315 2458314 2026-09-17T07:57:37Z Stashbot 7414 Amir1: dropped non-links tables from db2245 (T437278) 2458315 wikitext text/x-wiki ji95pg3inipqwv7jwt3udgrph42hbe6 2458316 2458315 2026-09-17T07:59:26Z Stashbot 7414 Amir1: dropped non-links tables from db1262 (T437278) 2458316 wikitext text/x-wiki myn7yotax9bwyu4n9fhz2q0ns61yvfx 2458317 2458316 2026-09-17T08:01:29Z Stashbot 7414 Amir1: dropped links tables from db2236 (T437278) 2458317 wikitext text/x-wiki 3579pxt5chwy83ll9csi29s9presaky 2458318 2458317 2026-09-17T08:03:57Z Stashbot 7414 Amir1: dropped links tables from db2219 (T437278) 2458318 wikitext text/x-wiki nwxb4qsjuxmuigo95zkdmvqvef2d255 2458319 2458318 2026-09-17T08:07:44Z Stashbot 7414 Amir1: dropped links tables from db2206 (T437278) 2458319 wikitext text/x-wiki 355dfwcimmtnf4691u66uraovdb1j7j 2458320 2458319 2026-09-17T08:10:37Z Stashbot 7414 moritzm: imported nodejs_26.8.2-1nodesource1 to thirdparty/node26 for trixie-wikimedia T437510 2458320 wikitext text/x-wiki mgni5mvlt0rb2m6bkl6n6fjne2u2kj4 2458321 2458320 2026-09-17T08:13:59Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.20 refs T430839 2458321 wikitext text/x-wiki 85j6dfrqwtdrrvn1zfraq0xsioj2h9o 2458322 2458321 2026-09-17T08:26:32Z Stashbot 7414 brouberol@dns1004: START - running authdns-update 2458322 wikitext text/x-wiki m1iiwxik643id09n22xr58d4n5acvm3 2458323 2458322 2026-09-17T08:28:52Z Stashbot 7414 brouberol@dns1004: END - running authdns-update 2458323 wikitext text/x-wiki 20tkee69l2awxc2ladvgfof61tmf1op 2458324 2458323 2026-09-17T08:34:28Z Stashbot 7414 XioNoX: Manually install gnmic 0.49.0 on netflow2005 - T438291 2458324 wikitext text/x-wiki 4clfoh68zsqg32aj76o3w4v0wgsry1u 2458325 2458324 2026-09-17T08:37:58Z Stashbot 7414 moritzm: pruned obsolete Bullseye image prometheus-nutcracker-exporter from the docker registry T416452 2458325 wikitext text/x-wiki g6yowrybhtvlasv0i2ynbg9eoi9ibar 2458340 2458325 2026-09-17T09:35:28Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2458340 wikitext text/x-wiki snibjrcf7mh9csh2cqqvry0qwumzyhs 2458341 2458340 2026-09-17T09:35:58Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2458341 wikitext text/x-wiki nhffc4v4b9261c3m7gfmlfnf0q0kldd 2458342 2458341 2026-09-17T09:40:19Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2458342 wikitext text/x-wiki tgbrustkap8vn6a49jf1uk93irkqp2r 2458343 2458342 2026-09-17T09:40:39Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2458343 wikitext text/x-wiki fegpuhc98pn44oq3061ltr3gn16eta9 2458344 2458343 2026-09-17T09:40:55Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2458344 wikitext text/x-wiki r494ws1p1e595xi5z5x6ki6uh9r10qb 2458345 2458344 2026-09-17T09:41:13Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2458345 wikitext text/x-wiki 305kem9enk65pjnqoeldznuwfnm9mh7 2458346 2458345 2026-09-17T09:41:21Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2458346 wikitext text/x-wiki 4kp3mhqgc1aidu82luwli6vd6flumb2 2458347 2458346 2026-09-17T09:41:42Z Stashbot 7414 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2458347 wikitext text/x-wiki j7lmslajog2s6oe4aanqlhyk6ypngwv 2458348 2458347 2026-09-17T09:51:35Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Fix weights T436496', diff saved to https://phabricator.wikimedia.org/P96465 and previous config saved to /var/cache/conftool/dbconfig/20260917-095131-marostegui.json 2458348 wikitext text/x-wiki jaxna6w9e2zatil7udu04mdyawbqv2o 2458349 2458348 2026-09-17T09:52:38Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Fix weights T436496', diff saved to https://phabricator.wikimedia.org/P96466 and previous config saved to /var/cache/conftool/dbconfig/20260917-095235-marostegui.json 2458349 wikitext text/x-wiki 4in9ojsxel7cd1mexw32u7q0630rcbj 2458361 2458349 2026-09-17T10:07:28Z Stashbot 7414 blake@deploy1003: Started scap sync-world: cleanup for T417800 2458361 wikitext text/x-wiki ggcwi43yof4xbgxgf1lxzcbqvb4uxt5 2458362 2458361 2026-09-17T10:10:58Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: cleanup for T417800 (duration: 03m 57s) 2458362 wikitext text/x-wiki k1fkp7lxviirrbfe0rkyygocv5ppr57 2458364 2458362 2026-09-17T10:28:57Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2458364 wikitext text/x-wiki fch2at5ojo827jcuzeeobt3drwwmdmm 2458368 2458364 2026-09-17T10:37:47Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2458368 wikitext text/x-wiki h1wrym98i2j5b2f43b1qs5y6bn4jkii 0 17290 2458310 2457875 2026-09-17T07:29:32Z Stashbot 7414 codders: rebuilt and restarted phpunit-results-cache server 2458310 wikitext text/x-wiki 4y1f3dofed1mtq9lsrwa41szjf98snv 498 30942 2458127 2457625 2026-09-16T15:41:19Z Stashbot 7414 andrewbogott: systemctl restart mariadb@x3.service mariadb@s3.service on clouddb1023, trying to resolve memory pressure T438200 2458127 wikitext text/x-wiki 9eeeackj7nfkz7zbot69tkk9l7heove 2458326 2458127 2026-09-17T09:12:39Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458326 wikitext text/x-wiki 2zei1i2unv9tkogl933ujmi02c3vlk9 2458327 2458326 2026-09-17T09:12:58Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458327 wikitext text/x-wiki gxqlwy1gzk1amo3ojwv8yt7u345bzk0 2458328 2458327 2026-09-17T09:14:39Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 2458328 wikitext text/x-wiki dar4qoy23kllnx8uq6nt195momfox7x 2458329 2458328 2026-09-17T09:15:03Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 2458329 wikitext text/x-wiki 6vs6uzkibmaxabqjz1vazvo82ro8dvr 2458330 2458329 2026-09-17T09:15:07Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458330 wikitext text/x-wiki lczh6uu3jjvluui7nzhxslb1frnosea 2458331 2458330 2026-09-17T09:15:27Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458331 wikitext text/x-wiki lbmyolni44hyykxcpdwm6g8g2r7jjae 2458332 2458331 2026-09-17T09:16:14Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458332 wikitext text/x-wiki ongs418cj28u6e1javdq4qbkh3xrw1m 2458333 2458332 2026-09-17T09:16:34Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458333 wikitext text/x-wiki fznc230vzf9l437ocwalny3qeeon8ha 2458334 2458333 2026-09-17T09:17:43Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458334 wikitext text/x-wiki oobdbbehr5lm8rxum4mx3372fyvzcrk 2458335 2458334 2026-09-17T09:18:06Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458335 wikitext text/x-wiki 1j0zjowgrg3v1crt5qnj17j79nrpdgx 2458336 2458335 2026-09-17T09:18:59Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 2458336 wikitext text/x-wiki 6ofajz2xm523wqmnopmraxxxzkorhhf 2458337 2458336 2026-09-17T09:19:23Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 2458337 wikitext text/x-wiki 2bgmv0tjmln7s8tygp3dqh725p80ek9 2458338 2458337 2026-09-17T09:20:22Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 2458338 wikitext text/x-wiki 5dpe7ub4f925j4k39zlioky7yp5b56b 2458339 2458338 2026-09-17T09:20:46Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 2458339 wikitext text/x-wiki gyh58p913ysu1gaxzy7wp1ykfwzsimy 2458350 2458339 2026-09-17T09:53:21Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2458350 wikitext text/x-wiki 3lfhkjj1zsdl0w4s4mzd17ugju6vp2z 2458352 2458350 2026-09-17T09:55:04Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2458352 wikitext text/x-wiki d6ztdtx6oia11ndd88kvonr1z8ztgmf 2458353 2458352 2026-09-17T09:55:10Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458353 wikitext text/x-wiki jt83l2haw098a13b0zjchh0qhr42ab4 2458354 2458353 2026-09-17T09:55:31Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 2458354 wikitext text/x-wiki 6a7kseu5wf3oiogbp3yl2z6jx1awapf 2458355 2458354 2026-09-17T09:55:57Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2458355 wikitext text/x-wiki p4xyysy4rc1zp9d3tyh0r323d3co1u0 2458356 2458355 2026-09-17T09:56:54Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2458356 wikitext text/x-wiki 10635d71zy1lhxm27be7curz562zwhe 2458357 2458356 2026-09-17T09:58:00Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2458357 wikitext text/x-wiki a9b13tnnaxdvmq1ulwarvvmgq5aydti 2458358 2458357 2026-09-17T09:59:05Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2458358 wikitext text/x-wiki r9p8sosjj3gst8tkh74x6amxyush3pl 0 221634 2458168 2447691 2026-09-16T17:25:19Z BCornwall-WMF 31521 /* Deployed hardware */ Update to reflect moving two uploads to texts 2458168 wikitext text/x-wiki lndvftnoew96dz1kticuymbxje6riby 2458169 2458168 2026-09-16T17:26:29Z BCornwall-WMF 31521 /* Deployed hardware */ update caching cluster link 2458169 wikitext text/x-wiki 53h8v1e1jd53nbni1vcrzzmm7kcen6q 498 443261 2458376 2457616 2026-09-17T11:50:29Z Stashbot 7414 taavi: rebooting all etcd nodes to migrate to nftables 2458376 wikitext text/x-wiki joju9kp4gemdazg04nm7wv9ilvyp8vr 0 445919 2458096 2435654 2026-09-16T14:46:45Z Filippo Giunchedi 2939 /* Howto */ add reinitialize a stack 2458096 wikitext text/x-wiki elkuzeaz1gqlo7zrx9tyzoz32rxsr1s 2458097 2458096 2026-09-16T14:46:57Z Filippo Giunchedi 2939 /* Reinitialize a stack */ 2458097 wikitext text/x-wiki q1xnyly7spfbqpotsykjlc9jpk0bn6m 498 446692 2458114 2451116 2026-09-16T15:13:16Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1067'] 2458114 wikitext text/x-wiki jl5kzapld2we0szzyqrtclpu30kulxp 2458115 2458114 2026-09-16T15:13:40Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1067'] 2458115 wikitext text/x-wiki t3472gs76lo2ts90gnx7vqtaexspels 0 449160 2458279 2457845 2026-09-17T00:02:42Z Dexbot 30554 Bot: Updating the report 2458279 wikitext text/x-wiki hgg6l0rhvod3x9wyd95omfuxnm819ev 2458351 2458279 2026-09-17T09:54:56Z Dexbot 30554 Bot: Updating the report 2458351 wikitext text/x-wiki aumne2s3rrgellcaayayau006t9auef 0 453813 2458048 2457709 2026-09-16T13:46:04Z PFischer-WMF 38527 Revise SLIs: promote each to its own heading, add availability, update lag and weighted tags query latency with measured values, move measurement caveats into their own section, mark the four inherited render-time latency SLIs as not currently measured, rename Elasticsearch to OpenSearch, refresh Monitoring and the dashboard links 2458048 wikitext text/x-wiki 6823m50ryozknk4tkalenwul2z6me6q 2458087 2458048 2026-09-16T14:32:29Z PFischer-WMF 38527 Revise SLIs: add availability, weighted tags query latency and update lag using the standard SLI wordings, move coverage and caveats into their own subsection, drop the unmeasured render-time latency SLIs and note under Request Classes which classes have no latency objective, rename Elasticsearch to OpenSearch, refresh Monitoring, fill in realistic and ideal targets, link the Sloth definitions 2458087 wikitext text/x-wiki 3c1pj7hsofv9hg39m2ah8h1ga7j8w6y 0 455989 2458119 2453598 2026-09-16T15:23:22Z JVanderhoop-WMF 41069 /* Test */ Added PD to override option 2458119 wikitext text/x-wiki motee8k3tcv3w3ym4w3b36dn2hb4trm 1 456479 2458365 2265053 2026-09-17T10:33:00Z Xaosflux 38028 /* wiki.gives */ new section 2458365 wikitext text/x-wiki ij2h35yqeovrpsxon6ln8xvy46x9qet 2458366 2458365 2026-09-17T10:33:15Z Xaosflux 38028 yeswiki 2458366 wikitext text/x-wiki lyfyhc3vmfsqldlxd8a3a61o9d41duq 2458367 2458366 2026-09-17T10:34:32Z Xaosflux 38028 /* Proposal to delete */ Reply 2458367 wikitext text/x-wiki r3totjljobakyrejf4pjjjmonegmjvn 0 459290 2458363 2457382 2026-09-17T10:14:32Z Milimetric 612 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2458363 wikitext text/x-wiki i2fhf8aoz0aasl5w8ur035d1yna3gkm 0 459699 2458064 2457883 2026-09-16T14:02:56Z Stashbot 7414 ejegg: fundraising civicrm upgraded from fc41fd51 to af3d6e52 2458064 wikitext text/x-wiki qtto17ehh7qf4l74izpb4k53axivcdz 2458217 2458064 2026-09-16T19:38:54Z Stashbot 7414 larssandergreen: donorwiki upgraded from c56b3ad7 to 60a7b7af 2458217 wikitext text/x-wiki el8ghb830xi55ytowg6k70mehic4cxk 2458223 2458217 2026-09-16T19:54:33Z Stashbot 7414 ejegg: payments-wiki upgraded from ce2479d6 to 60a7b7af 2458223 wikitext text/x-wiki nnbv5isvbksw6c611h9ty8eofl5lhly 2458251 2458223 2026-09-16T21:39:26Z Stashbot 7414 eileen: civicrm upgraded from af3d6e52 to 22cb59e5 2458251 wikitext text/x-wiki 7t1u0yg699rlt75rou2zdmi2j496vm8 2458273 2458251 2026-09-16T22:27:05Z Stashbot 7414 eileen: config revision changed from d2c688c7 to a53bcf1d 2458273 wikitext text/x-wiki pjqt6onsl1a5z0ccsyoxergxbrcoysb 2458280 2458273 2026-09-17T01:26:53Z Stashbot 7414 eileen: config revision changed from 95bc4657 to a53bcf1d 2458280 wikitext text/x-wiki kwa0zdi3z9rppt7qoyqb95drrhbcixa 2458281 2458280 2026-09-17T02:07:39Z Stashbot 7414 eileen: civicrm upgraded from 22cb59e5 to ef52f0c1 2458281 wikitext text/x-wiki d23n05bx3g4xlab9chw8ot4i6dhjf75 0 459966 2458216 2456457 2026-09-16T19:36:17Z KReid-WMF 49588 /* Case study: Email confirmation banner sample ratio mismatch */ Added summary recommendation 2458216 wikitext text/x-wiki bpkd3jh9oqnfe7cusy9qdcuczl27r19 0 460089 2458227 2457833 2026-09-16T20:08:41Z EEvans (WMF) 5035 add some example GRANT statements 2458227 wikitext text/x-wiki 3co5pldv6fur6y2s1klqwx4f6b3o7kd 0 460232 2458232 2452568 2026-09-16T20:52:32Z JVanderhoop-WMF 41069 /* How do I…? */ Added create/run link 2458232 wikitext text/x-wiki j8qwbx84s96bzo7nbgphirin6ovo898 0 460234 2458226 2454099 2026-09-16T20:04:14Z KReid-WMF 49588 /* How do I…? */ Switch the order of the sections on importing and creating experiments, since creating is now more common than importing. 2458226 wikitext text/x-wiki 3jn27bbro0q6sqk1lxjtyfcdx7zviql 0 460366 2458156 2457118 2026-09-16T17:09:28Z TBurmeister (WMF) 27417 use page instead of document 2458156 wikitext text/x-wiki 4vemhxmaoop6tsom8pi6v94gncg9g1l 0 460375 2458155 2456619 2026-09-16T17:09:03Z TBurmeister (WMF) 27417 Move toc, use "page" instead of "document" 2458155 wikitext text/x-wiki nnu8anolkbv5q3uoat8ivf8h32sbdws 0 460376 2458213 2452950 2026-09-16T19:20:49Z TBurmeister (WMF) 27417 Revise for structure, style, formatting, removing unnecessary content 2458213 wikitext text/x-wiki omtvkta0ic8h883zzoeqqdxmqs58uig 0 460424 2458122 2457571 2026-09-16T15:34:38Z RCoccioli (WMF) 7416 Signed off remaining items but the security review 2458122 wikitext text/x-wiki 95b7hkhvh7em7fnqsj8b8pyyihz38g1 2458126 2458122 2026-09-16T15:37:23Z RCoccioli (WMF) 7416 add missing assignee 2458126 wikitext text/x-wiki aa5yjgy1jsi1wsx6tab534bc6g3zu6j 0 460621 2458094 2455151 2026-09-16T14:44:41Z JGiannelos (WMF) 17083 /* NOTES */ 2458094 wikitext text/x-wiki 23in2z1w8gkns0fa662stig4wz0sno2 2458100 2458094 2026-09-16T14:49:55Z JGiannelos (WMF) 17083 2458100 wikitext text/x-wiki dno7iho261fdatdrju1xuwea1wx3nwa 2458101 2458100 2026-09-16T14:50:48Z JGiannelos (WMF) 17083 2458101 wikitext text/x-wiki athqzy3xgr3nf3zbfay1wq20g0ipnk4 2458103 2458101 2026-09-16T14:51:39Z JGiannelos (WMF) 17083 /* NOTES */ 2458103 wikitext text/x-wiki kfai1va04ex6synkskhdbjck76bmlbs 2458104 2458103 2026-09-16T14:52:40Z JGiannelos (WMF) 17083 /* NOTES */ 2458104 wikitext text/x-wiki 1tibphawjnzys1gtpmmnihozum9glb8 2458105 2458104 2026-09-16T14:56:05Z JGiannelos (WMF) 17083 /* NOTES */ 2458105 wikitext text/x-wiki me2vre6rzsepwlk2d94pejszccq8g4f 2458106 2458105 2026-09-16T14:58:46Z JGiannelos (WMF) 17083 2458106 wikitext text/x-wiki 1bm7p846vmpne9gye6kbtc8pcy8ds1e 2458107 2458106 2026-09-16T15:02:07Z JGiannelos (WMF) 17083 2458107 wikitext text/x-wiki olmdvwirczvhpkpfpfafn6nly550t26 2458108 2458107 2026-09-16T15:03:23Z JGiannelos (WMF) 17083 2458108 wikitext text/x-wiki gulifse87pq1ph26uuyd6i6ygqyh4gc 2458110 2458108 2026-09-16T15:04:33Z JGiannelos (WMF) 17083 2458110 wikitext text/x-wiki nf08qto3xufkv3jetlunxx8usexb3bk 2458111 2458110 2026-09-16T15:06:38Z JGiannelos (WMF) 17083 2458111 wikitext text/x-wiki saem8kjs83uhrbgmvkrzp4k6cfdoc19 2458112 2458111 2026-09-16T15:07:53Z JGiannelos (WMF) 17083 2458112 wikitext text/x-wiki b4nk1xmmqy8is8a9501zdw9lccfjk5p 2458113 2458112 2026-09-16T15:09:41Z JGiannelos (WMF) 17083 /* NOTES */ 2458113 wikitext text/x-wiki anbo0dvsivlka62c8e2nbbvuy11vbcb 2458133 2458113 2026-09-16T16:12:49Z JGiannelos (WMF) 17083 /* Service dependencies */ 2458133 wikitext text/x-wiki t22ak6etrcdtbpkc80bybz6gxxuvn8m 2458134 2458133 2026-09-16T16:17:19Z JGiannelos (WMF) 17083 /* Architectural */ 2458134 wikitext text/x-wiki 5looi8tiydy6vda3fwftjtacbcrjucv 2458136 2458134 2026-09-16T16:21:06Z JGiannelos (WMF) 17083 2458136 wikitext text/x-wiki 7tac1oqneghsc25pbuk8p544b96f0nx 2458137 2458136 2026-09-16T16:24:42Z JGiannelos (WMF) 17083 2458137 wikitext text/x-wiki r9lus2iw3ishayq1fuzvcybu0y40wk6 2458138 2458137 2026-09-16T16:33:30Z JGiannelos (WMF) 17083 /* Service dependencies */ 2458138 wikitext text/x-wiki m89ue0dbe9yi0qdexums951mqvx64bg 2458139 2458138 2026-09-16T16:33:44Z JGiannelos (WMF) 17083 2458139 wikitext text/x-wiki 1r6luw6acwvtf3tqcvlcder6wrx2fo2 2458140 2458139 2026-09-16T16:39:41Z JGiannelos (WMF) 17083 2458140 wikitext text/x-wiki k9ctzei1igm3xilgjhacpl2vp4bcf5w 2458144 2458140 2026-09-16T16:54:22Z JGiannelos (WMF) 17083 2458144 wikitext text/x-wiki sftz54zjrx2fmdc2j6y4e0rcbpy2t0l 2458145 2458144 2026-09-16T16:56:23Z JGiannelos (WMF) 17083 /* Realistic targets */ 2458145 wikitext text/x-wiki qahdg7ygnnoko6gnm2a21qgchwv2azm 2458146 2458145 2026-09-16T16:56:37Z JGiannelos (WMF) 17083 2458146 wikitext text/x-wiki cuahxjoq84hf1dvef4mioydkpp8lyle 2458147 2458146 2026-09-16T16:57:40Z JGiannelos (WMF) 17083 2458147 wikitext text/x-wiki o772pj4gdjkvxgsw7m6a37w6bdr0z0p 2458148 2458147 2026-09-16T17:00:18Z JGiannelos (WMF) 17083 2458148 wikitext text/x-wiki mrhjowo9j4mneyhg27emzd5bcb9s35s 2458359 2458148 2026-09-17T09:59:56Z JGiannelos (WMF) 17083 2458359 wikitext text/x-wiki 9m5dxlgb45j7yfj7okj9ju7wqubep80 2458360 2458359 2026-09-17T10:00:28Z JGiannelos (WMF) 17083 2458360 wikitext text/x-wiki bz9u1q5fkuhxnnmk9k9iqfs7vlyhu3f 2458369 2458360 2026-09-17T10:50:49Z Krinkle 44 2458369 wikitext text/x-wiki 9p6knldnlsdefm15i35z0de533jawnz 2458371 2458369 2026-09-17T10:56:31Z Krinkle 44 /* Metric */ the hook does not scope the read view functionality, the hook represents a moment in time form which the timer is retroactively started for the request and the timer is stopped from a deferred update that runs no matter what. So "outside" means only "before". 2458371 wikitext text/x-wiki lu7ym3khtfjgeoq0dbxbztrufn82zd8 2458372 2458371 2026-09-17T11:08:15Z Krinkle 44 /* Metric */ match indentation between the two promql blocks, clarify what they are about (i.e. why there are two, I missed the number difference at first) 2458372 wikitext text/x-wiki fxk62t1x762yohyudjzwuicsdwsse22 2458373 2458372 2026-09-17T11:08:48Z Krinkle 44 +{{Tracked|T435644}} 2458373 wikitext text/x-wiki 7gw3k46oecxqgv5c6p331f4ca07iodb 0 460624 2457985 2457979 2026-09-16T12:09:22Z MMuhlenhoff (WMF) 5268 /* Service Summary */ 2457985 wikitext text/x-wiki 4gybd6kyxw9fxez02u4xx049xc7ptqv 2457991 2457985 2026-09-16T12:26:31Z MMuhlenhoff (WMF) 5268 /* Service Summary */ Phab all fine 2457991 wikitext text/x-wiki ofc1dm3acbamuvi2y0xzrg9oo7aei4l 2458001 2457991 2026-09-16T12:43:40Z MMuhlenhoff (WMF) 5268 /* Service Summary */ design looks good 2458001 wikitext text/x-wiki q5655ipe1gpi87zhoxsrqwof0n1frmc 2458009 2458001 2026-09-16T12:55:05Z MMuhlenhoff (WMF) 5268 /* Reliability and performance */ sign off some reliability bits 2458009 wikitext text/x-wiki 7w519owhtsxqjx4fki7btjhn7s4ry15 2458021 2458009 2026-09-16T13:15:20Z MMuhlenhoff (WMF) 5268 /* Reliability and performance */ more reliability sign offs 2458021 wikitext text/x-wiki 18nkrgtkbyiaoimfndt4j5t282lx4km 2458032 2458021 2026-09-16T13:26:06Z MMuhlenhoff (WMF) 5268 /* Reliability and performance */ More signoffs 2458032 wikitext text/x-wiki b35oej008ebuzbpn5p2eqtuhqijgd2p 2458038 2458032 2026-09-16T13:36:16Z MMuhlenhoff (WMF) 5268 /* Reliability and performance */ sign off 2458038 wikitext text/x-wiki kwhyi3no0xneqn04krpct2whx3canxj 2458054 2458038 2026-09-16T13:53:20Z MMuhlenhoff (WMF) 5268 /* Operating Procedures */ docs are fine 2458054 wikitext text/x-wiki ljm8rtxs3clzebdbje6w85m7xq8cd7i 3 460680 2458149 2026-09-16T17:01:08Z StrikerBot 8475 Welcome to Toolforge! 2458149 wikitext text/x-wiki 9hzu3uojgp3jae8ynm9v3snqy84mm3e 3 460681 2458150 2026-09-16T17:01:23Z StrikerBot 8475 Welcome to Toolforge! 2458150 wikitext text/x-wiki mcti1gaz4fnl109jqbsp011mawyrhkn 0 460682 2458214 2026-09-16T19:33:00Z AndrewBogott (WMF) 51047 Created page with "{{Cloud VPS nav}} Magnum is an openstack service running on cloudcontrols which calls out to a cloud-hosted kubernetes cluster to do the actual work. The full explanation of the moving parts is [[Portal:Cloud_VPS/Admin/Magnum_setup|here]]. {{Requires admin permissions banner}} == Magnum infra reports as down == === Error/Incident === === Fixing === == Magnum cluster api worker node is not ready == === Error/Incident === === Fixing === == No capi pod running in the..." 2458214 wikitext text/x-wiki eq48ciwpor6k1t2gwnzl69fhhlrd7cz 2458215 2458214 2026-09-16T19:34:18Z AndrewBogott (WMF) 51047 /* Error/Incident */ 2458215 wikitext text/x-wiki e7i6bc52ddlgue7zwrxu8nqs3uezq1x 2458218 2458215 2026-09-16T19:40:18Z AndrewBogott (WMF) 51047 /* Fixing */ 2458218 wikitext text/x-wiki eyt2riz8att68s8ok51t4jm727un7jq 2458219 2458218 2026-09-16T19:46:01Z AndrewBogott (WMF) 51047 /* Error/Incident */ 2458219 wikitext text/x-wiki dcr566al5myby9wrofersbcmqshscsy 2458220 2458219 2026-09-16T19:46:31Z AndrewBogott (WMF) 51047 /* Error/Incident */ 2458220 wikitext text/x-wiki i1bs37jgcufad2e29i7qra2dugkqdba 2458221 2458220 2026-09-16T19:52:16Z AndrewBogott (WMF) 51047 /* Fixing */ 2458221 wikitext text/x-wiki 6p67rqim4dg6vnv50zqweawid3kh2ry 2458222 2458221 2026-09-16T19:54:32Z AndrewBogott (WMF) 51047 /* No capi pod running in the magnum cluster api worker cluster */ 2458222 wikitext text/x-wiki o3yxfoov6b5jjxngef3e7j83s7kgx5g 2458224 2458222 2026-09-16T19:54:38Z AndrewBogott (WMF) 51047 /* Fixing */ 2458224 wikitext text/x-wiki r6fv54hnrpqq4ahojvn8s189roosgjx 2458225 2458224 2026-09-16T19:55:27Z AndrewBogott (WMF) 51047 /* Error/Incident */ 2458225 wikitext text/x-wiki b6sobwqpcz9yyu90j7mfu8t9gs45ti1 0 460683 2458284 2026-09-17T05:20:22Z CCiufo-WMF 37368 Created page with "{{draft}} == Summary == {{tracked|T432860}} The Tools Platform team is developing a new set of monitoring and alerting features in Toolforge. Initially this will include an option for tool maintainers to receive an email notification when their web service stops responding, has a spike in errors, or is frequently restarting. == Timeline == Tool maintainers will be able to start opting-in to receiving alerts based on the predefined thresholds and criteria in October 202..." 2458284 wikitext text/x-wiki f32eyuwdjqxpkq7hpjqefmc97ls037l 2458297 2458284 2026-09-17T06:50:18Z APuthin-WMF 52425 Added links & refs 2458297 wikitext text/x-wiki c6mpph4c4qnby06xgg7xxslvg2fco2p 3 460684 2458285 2026-09-17T05:26:25Z Mustleib 54367 Created page 2458285 wikitext text/x-wiki phoiac9h4m842xq45sp7s6u21eteeq1 2 460685 2458295 2026-09-17T06:01:27Z Sohom Datta 15330 Created page with "{{Toolforge nav}} {{Note|If you only need to run [[wmdoc:pywikibot/stable/scripts/index.html|a script that is included with Pywikibot itself]], it is recommended that you follow the [[Help:Toolforge/Running Pywikibot scripts|running Pywikibot scripts tutorial]] instead.}} This tutorial contains instructions on how to run custom '''[[:mw:Special:MyLanguage/Manual:Pywikibot|Pywikibot]]''' scripts on Toolforge using the Help:Toolforge/Jobs framework|Toolforge jobs framewo..." 2458295 wikitext text/x-wiki rw92xzf26na6vjc5z47h05x0w418pig 0 460687 2458374 2026-09-17T11:09:41Z Krinkle 44 Redirected page to [[SLO/Mediawiki Page View]] 2458374 wikitext text/x-wiki lj3ni4rsobpbs7rohjdsp4q2zuhd4p8 0 460688 2458375 2026-09-17T11:31:24Z Novem Linguae 23682 create 2458375 wikitext text/x-wiki 77lbe8o5j13k3yqgpwoszzi0rr5v0r0