Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.48.0-wmf.1 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2474205 2473205 2026-10-27T15:15:39Z Stashbot 7414 dcaro: round of pod restarts to pick up new pre-built images 2474205 wikitext text/x-wiki 4x4gsw518c97jxcbunqakbhqqbfwqhk 0 4108 2474061 2473795 2026-10-27T13:02:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1359263]] to Tuesday, October 27 UTC afternoon backport window 2474061 wikitext text/x-wiki 92nyfbf96h09uhexa0n33koakhvax1b 2474062 2474061 2026-10-27T13:02:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1359261]] to Tuesday, October 27 UTC afternoon backport window 2474062 wikitext text/x-wiki 8gbz8a192qja581x0lrd0lni210jxgz 2474073 2474062 2026-10-27T13:07:20Z ScheduleDeploymentBot 37566 Add [[gerrit:1358893]] to Tuesday, October 27 UTC afternoon backport window 2474073 wikitext text/x-wiki isvopplv0z9o7fyt6atykeflajw3wi8 2474081 2474073 2026-10-27T13:16:00Z Thiemo Kreuz (WMDE) 2595 /* {{Deployment_day|date=2026-10-27}} */ Postponed 2474081 wikitext text/x-wiki kszwqseovsz10dc598uduimnhukq184 2474084 2474081 2026-10-27T13:18:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1355485]] to Thursday, October 29 UTC afternoon backport window 2474084 wikitext text/x-wiki hi3ifpc5q6kt5vqyyesxgxjs8gh79oz 2474270 2474084 2026-10-27T17:04:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1359334]] to Tuesday, October 27 UTC late backport window 2474270 wikitext text/x-wiki 5d27ryt45vyg5t56hp4ljknzk6no70v 2474303 2474270 2026-10-27T18:12:31Z ScheduleDeploymentBot 37566 Add [[gerrit:1358456]] to Thursday, October 29 UTC late backport window 2474303 wikitext text/x-wiki barojyymd8xue1l78rh1np3p9pi6ucq 2474529 2474303 2026-10-28T11:06:41Z ScheduleDeploymentBot 37566 Add [[gerrit:1359782]] to Monday, November 02 UTC morning backport window 2474529 wikitext text/x-wiki 0pwhm3ei8x2jflpn9zt5eapzb9zua0j 2474554 2474529 2026-10-28T11:34:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1359789]] to Wednesday, October 28 UTC afternoon backport window 2474554 wikitext text/x-wiki 8ljms78csn4bmab6awzncsal98hbzr6 2474557 2474554 2026-10-28T11:35:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1359789]] to Wednesday, October 28 UTC afternoon backport window 2474557 wikitext text/x-wiki ihy4dc9sqmkzz1q4g9f2evzke53wc4t 2474558 2474557 2026-10-28T11:35:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1359790]] to Wednesday, October 28 UTC afternoon backport window 2474558 wikitext text/x-wiki jyp5ben5mzueydkw4sqkzcbvp6n6rk9 2474559 2474558 2026-10-28T11:36:36Z Mmullie (WMF) 291 /* {{Deployment_day|date=2026-10-28}} */ 2474559 wikitext text/x-wiki d7txqrt57hvltj5xygl7ot6er2lujfi 0 7919 2473981 2473980 2026-10-27T12:00:08Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync 2473981 wikitext text/x-wiki q7btnj5n9u2y58wziup8ac5exwfh0de 2473982 2473981 2026-10-27T12:00:15Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync 2473982 wikitext text/x-wiki 70hqe3acco0mc404bzq86cj2ob3srkr 2473983 2473982 2026-10-27T12:00:19Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync 2473983 wikitext text/x-wiki bjjy6aha2fbhxazizen9rccizbsoyp5 2473984 2473983 2026-10-27T12:00:27Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync 2473984 wikitext text/x-wiki 02a62bwh90abs55vmt8o1rtm3aaxrde 2473985 2473984 2026-10-27T12:00:41Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet 2473985 wikitext text/x-wiki 1lreylnzw2uooh2ylibepq0b0ahxamz 2473986 2473985 2026-10-27T12:02:03Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet 2473986 wikitext text/x-wiki ihsfzbhr4mbudeebs6uqceszljdrbzy 2473987 2473986 2026-10-27T12:02:06Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet 2473987 wikitext text/x-wiki m1tprmrerzozkitz8pzyrnak9ew81rs 2473988 2473987 2026-10-27T12:02:11Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet 2473988 wikitext text/x-wiki puankzlttcfti6iyzadvzfye338pazg 2473989 2473988 2026-10-27T12:02:13Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet 2473989 wikitext text/x-wiki p3dy0681zlfq3pm61rwqkqe8vh2se0c 2473990 2473989 2026-10-27T12:02:33Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS trixie 2473990 wikitext text/x-wiki iukem9vas90mlgpzip3253idcmznu26 2473991 2473990 2026-10-27T12:05:40Z Stashbot 7414 blake@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2473991 wikitext text/x-wiki skq21q4qwowvhwdjh64ytkg1ur1fo99 2473992 2473991 2026-10-27T12:06:00Z Stashbot 7414 cgoubert@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2473992 wikitext text/x-wiki n6p6fd78jp3csz3rzck4iva3sp7i51c 2473993 2473992 2026-10-27T12:06:13Z Stashbot 7414 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2473993 wikitext text/x-wiki 4py05n4gcrbes4f5nmn8lbpx82c304e 2473994 2473993 2026-10-27T12:06:17Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2473994 wikitext text/x-wiki hjiyujezgkspcmb0i32m7yjy97rk0x8 2473995 2473994 2026-10-27T12:06:40Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2473995 wikitext text/x-wiki i49u09myz0eeyzxf5q13x25u06qvvq0 2473996 2473995 2026-10-27T12:06:43Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2473996 wikitext text/x-wiki 9gkb6cf8qqd2j0ey841qrgvgggmir01 2473997 2473996 2026-10-27T12:06:46Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet 2473997 wikitext text/x-wiki lptx37amtei36ky1wenwt1ur4myfa7g 2473998 2473997 2026-10-27T12:06:51Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet 2473998 wikitext text/x-wiki 544v8g0bqnxi89w60ghyx5iw20gfs0f 2473999 2473998 2026-10-27T12:06:56Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2473999 wikitext text/x-wiki pt6mrzk5kcr20basd66jzpxlbogl14a 2474000 2473999 2026-10-27T12:08:10Z Stashbot 7414 cgoubert@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2474000 wikitext text/x-wiki i91x83tjkfhgcq96eus3s96g3st7ns4 2474001 2474000 2026-10-27T12:08:30Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2474001 wikitext text/x-wiki nz8f3p9xgqkuh5j3geen7p0uxglean4 2474002 2474001 2026-10-27T12:08:34Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet 2474002 wikitext text/x-wiki 67v13i33n4ww4hgbhddlfnyb2hqlikr 2474003 2474002 2026-10-27T12:08:36Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet 2474003 wikitext text/x-wiki 9rvdn0eppsvnlrycxo9zma3jucrmhsd 2474004 2474003 2026-10-27T12:08:41Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet 2474004 wikitext text/x-wiki hzcpjcdhpm6nacxcw7xj6ckctzyr4my 2474005 2474004 2026-10-27T12:08:43Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet 2474005 wikitext text/x-wiki n7ipnqfs45owyzu3ede2t7hhjiaxpr5 2474006 2474005 2026-10-27T12:09:31Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2474006 wikitext text/x-wiki 1xfjdnb94t46if1nf36encyz4c3izsj 2474007 2474006 2026-10-27T12:10:16Z Stashbot 7414 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply 2474007 wikitext text/x-wiki ihtg8ug5q3niqw21g9a822m7q84ydpv 2474008 2474007 2026-10-27T12:10:32Z Stashbot 7414 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply 2474008 wikitext text/x-wiki hkcs8wxwb096y9kqihadwv3ze0i4tdj 2474009 2474008 2026-10-27T12:11:41Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2474009 wikitext text/x-wiki 00m6icmp6eny5a7lxbkboigw3exfl3o 2474010 2474009 2026-10-27T12:12:24Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2474010 wikitext text/x-wiki f7dndttiqm5gwtg7y6005an7bzw4rxm 2474011 2474010 2026-10-27T12:12:34Z Stashbot 7414 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply 2474011 wikitext text/x-wiki 5so0viiiij9yh980wt8xlmnzi9lnqxo 2474012 2474011 2026-10-27T12:12:59Z Stashbot 7414 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply 2474012 wikitext text/x-wiki 7w13xyx8ccma97ue6vp7tq6k1ino4qo 2474013 2474012 2026-10-27T12:13:07Z Stashbot 7414 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply 2474013 wikitext text/x-wiki bo5jo59iu4zpd60ec6jowc0g2wjwes2 2474014 2474013 2026-10-27T12:13:12Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet 2474014 wikitext text/x-wiki t4snxqh13o5dycm1huss26ejfiefdfq 2474015 2474014 2026-10-27T12:13:36Z Stashbot 7414 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply 2474015 wikitext text/x-wiki rm3tv1kby4qyaono0utmb769q5fmivs 2474016 2474015 2026-10-27T12:13:39Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2474016 wikitext text/x-wiki jf1w23vj2nzulg14f3dhz86zfecj8y5 2474017 2474016 2026-10-27T12:14:37Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2474017 wikitext text/x-wiki fx472vi0kp4y0gmxolfcvfwd6n1jg87 2474018 2474017 2026-10-27T12:14:57Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2474018 wikitext text/x-wiki lt0j15c5ie365gxmncobyg6z8utuno5 2474019 2474018 2026-10-27T12:15:07Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet 2474019 wikitext text/x-wiki 3maeeiu52alheovin9z4s365b2mmrbx 2474020 2474019 2026-10-27T12:15:09Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet 2474020 wikitext text/x-wiki seuc9x3n5jwmp4nzc8q0zax6e9r216p 2474021 2474020 2026-10-27T12:15:15Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2006.codfw.wmnet 2474021 wikitext text/x-wiki 2e3k0agujonua9fm28eu1k04hblc9e5 2474022 2474021 2026-10-27T12:15:17Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2006.codfw.wmnet 2474022 wikitext text/x-wiki mwqfm3vdz7i8pc0mao0jotxutrd96qo 2474023 2474022 2026-10-27T12:15:19Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org 2474023 wikitext text/x-wiki jlfatcc7piklfyy0k1j9lm5tc6xegzy 2474024 2474023 2026-10-27T12:17:14Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2474024 wikitext text/x-wiki s4zlvqqh9eo5fhc0lqef50pwoq7s2e4 2474025 2474024 2026-10-27T12:18:00Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet 2474025 wikitext text/x-wiki 35092csubxufqcev82rp2ew5jvxm9dn 2474026 2474025 2026-10-27T12:19:03Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3005.esams.wmnet 2474026 wikitext text/x-wiki 1p8fjeu69scjp981iwg7ac8vgpq1czx 2474027 2474026 2026-10-27T12:20:08Z Stashbot 7414 herron@cumin1004: END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling reboot on A:logstash-eqiad 2474027 wikitext text/x-wiki b29d1i2y9sesdzpkfqj3fvzjro4smnd 2474028 2474027 2026-10-27T12:20:24Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3005.esams.wmnet 2474028 wikitext text/x-wiki eb0d8ubxrlr4jvvjikpjuavu14ypxfx 2474029 2474028 2026-10-27T12:21:13Z Stashbot 7414 cgoubert@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2474029 wikitext text/x-wiki 8rxm72begjfj7fj1vln4r1hhwvdl5t5 2474030 2474029 2026-10-27T12:21:27Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org 2474030 wikitext text/x-wiki na2ubrefn6fqnscufc04ftdihb7kq4m 2474031 2474030 2026-10-27T12:21:37Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2006.codfw.wmnet 2474031 wikitext text/x-wiki pip00juk89zh3j40lshljjmrxk6hu2d 2474032 2474031 2026-10-27T12:21:40Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2006.codfw.wmnet 2474032 wikitext text/x-wiki d6peu6d03b1z3x3xigxnmoaz6xb968x 2474033 2474032 2026-10-27T12:21:43Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw 2474033 wikitext text/x-wiki 7hn5gp1wpo6qtshp5zq893mxmiuya2o 2474034 2474033 2026-10-27T12:22:21Z Stashbot 7414 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet 2474034 wikitext text/x-wiki g1egs42tketqn053pliixlcp1yd1rzj 2474035 2474034 2026-10-27T12:22:27Z Stashbot 7414 cgoubert@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2474035 wikitext text/x-wiki py9wgwk8ai86jjvqp6xt0gthgx1n5h0 2474036 2474035 2026-10-27T12:24:03Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet 2474036 wikitext text/x-wiki oznx3je4oqnf5vuuw8qo5s1kkihanxg 2474037 2474036 2026-10-27T12:24:12Z Stashbot 7414 cgoubert@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2474037 wikitext text/x-wiki hz3p32fgiz12aqbk2bt9j321u0pn8zy 2474038 2474037 2026-10-27T12:24:34Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet 2474038 wikitext text/x-wiki 3e064ot87pcnbgpydvmtx95qyfcrsky 2474039 2474038 2026-10-27T12:24:48Z Stashbot 7414 cgoubert@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2474039 wikitext text/x-wiki eqy4j5byyilwo9czrqlgju7ompf13kf 2474040 2474039 2026-10-27T12:25:12Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2474040 wikitext text/x-wiki 4ui5oeg44u7c1sju6jtx7eoczwz1176 2474041 2474040 2026-10-27T12:25:33Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet 2474041 wikitext text/x-wiki bhm0cccnxahx6042e12q147vrlezuw9 2474042 2474041 2026-10-27T12:26:51Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2474042 wikitext text/x-wiki 30t8cxegz3wigjeyg4lj5g66ybqampo 2474043 2474042 2026-10-27T12:27:08Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2474043 wikitext text/x-wiki lou9rmx0zgcvvl9fhz05qrusy3ctomf 2474044 2474043 2026-10-27T12:27:40Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2474044 wikitext text/x-wiki h9mofmkd5f4yi8efjl83c7rf55srgmx 2474045 2474044 2026-10-27T12:28:23Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3005.esams.wmnet 2474045 wikitext text/x-wiki 04t314xdzfb6bvj8gggdwcvfryrrmin 2474046 2474045 2026-10-27T12:28:29Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3005.esams.wmnet 2474046 wikitext text/x-wiki eowwdbuu1gvky3ignjml4k1p8hybd44 2474047 2474046 2026-10-27T12:29:34Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet 2474047 wikitext text/x-wiki 3mqsrgmknyrz7hdf5r9tcnxws38hgk2 2474048 2474047 2026-10-27T12:30:33Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica 2474048 wikitext text/x-wiki jhs0mel2632vrdkeocius8iamfve93h 2474049 2474048 2026-10-27T12:30:51Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica 2474049 wikitext text/x-wiki jivt881amzz1w46ek8oneymeegpkheo 2474050 2474049 2026-10-27T12:38:48Z Stashbot 7414 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet 2474050 wikitext text/x-wiki 0pbfaddbg9gwianc5qdrt1wx8fx96si 2474051 2474050 2026-10-27T12:38:51Z Stashbot 7414 fabfur@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_esams 2474051 wikitext text/x-wiki 29he8s5dtuwxbz97aoc2q3q2lua3u62 2474057 2474051 2026-10-27T12:54:01Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet 2474057 wikitext text/x-wiki evv6aeiexozjt57wsdbq2ohnsa9q5i6 2474058 2474057 2026-10-27T12:54:47Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet 2474058 wikitext text/x-wiki 68x6zfn2k1pwmstqbp2ycsmdjp5e36r 2474063 2474058 2026-10-27T13:04:20Z Stashbot 7414 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet 2474063 wikitext text/x-wiki b1wonzvv243a7kpwmu18wcilppvityo 2474074 2474063 2026-10-27T13:07:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet 2474074 wikitext text/x-wiki t0dw177gu63ck6061db2kwihcxfdbfw 2474075 2474074 2026-10-27T13:07:42Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3006.esams.wmnet 2474075 wikitext text/x-wiki tiy2q6qt9kw8mqydlxfzbbb491opxzz 2474076 2474075 2026-10-27T13:09:29Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2474076 wikitext text/x-wiki ouj5tcg2bdebw939x3cmti3js5j44w2 2474077 2474076 2026-10-27T13:10:20Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2474077 wikitext text/x-wiki rbqdosqa8v6nv6201pnhtk6adpqveu1 2474078 2474077 2026-10-27T13:15:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet 2474078 wikitext text/x-wiki anq80f0x3arzxmd32zs6zyeledvrlxq 2474079 2474078 2026-10-27T13:15:49Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet 2474079 wikitext text/x-wiki 4n7beiybkt3pxqd2zq5k7s6xx8er7j1 2474080 2474079 2026-10-27T13:15:54Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3006.esams.wmnet 2474080 wikitext text/x-wiki 1pmtjip3nknthzg6ikngovrt1ss6p62 2474082 2474080 2026-10-27T13:16:00Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet 2474082 wikitext text/x-wiki cjwl6x5x3to7mvz1elp3ilwgnr62lqe 2474083 2474082 2026-10-27T13:16:37Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) ([[gerrit:1359263|1.47.0-wmf.24]], [[gerrit:1359261|1.48.0-wmf.1]]), [[gerrit:1358893|WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379)]] 2474083 wikitext text/x-wiki b8xrfj26c8aim4gvazn5hgyeon6t7ln 2474085 2474083 2026-10-27T13:21:03Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet 2474085 wikitext text/x-wiki rj040crqui3wzf6gcnkcfzif9oe9a4a 2474086 2474085 2026-10-27T13:21:08Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3007.esams.wmnet 2474086 wikitext text/x-wiki 2aa0gxcyc8beaqyywhh9m5uis0ayi63 2474088 2474086 2026-10-27T13:27:11Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3007.esams.wmnet 2474088 wikitext text/x-wiki 88362p7q0pdvcn1ag8ctt7xcl1t0fa5 2474089 2474088 2026-10-27T13:27:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet 2474089 wikitext text/x-wiki 405vwp1yp5ru3rgdmk0itfccvdi9d81 2474090 2474089 2026-10-27T13:32:34Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet 2474090 wikitext text/x-wiki h3uj643vg0dc6liw1jfap69da2sevmk 2474091 2474090 2026-10-27T13:32:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet 2474091 wikitext text/x-wiki 34mto8zg7qe8uygxjpvpc07fi8gtg89 2474092 2474091 2026-10-27T13:34:00Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet 2474092 wikitext text/x-wiki ij42mkdt3v1g3lid2rfz4bxghspa8q8 2474093 2474092 2026-10-27T13:35:21Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3007.esams.wmnet 2474093 wikitext text/x-wiki nz1dh3rrgywd5dbyr6i1sr7tyfax517 2474094 2474093 2026-10-27T13:35:28Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3007.esams.wmnet 2474094 wikitext text/x-wiki 1pbt7js5pa0z5m1t5fysme7ah76a3id 2474095 2474094 2026-10-27T13:38:28Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) ([[gerrit:1359263|1.47.0-wmf.24]], [[gerrit:1359261|1.48.0-wmf.1]]), [[gerrit:1358893|WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474095 wikitext text/x-wiki 25dexwhmse118sb04f5kp3xuuhhuqb3 2474096 2474095 2026-10-27T13:39:31Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2474096 wikitext text/x-wiki tlkzffrjru9cs0lfiolwc00750h7v0h 2474097 2474096 2026-10-27T13:45:47Z Stashbot 7414 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply 2474097 wikitext text/x-wiki d7065dqk5j54gkmyu41x3x4r1m25ief 2474098 2474097 2026-10-27T13:45:56Z Stashbot 7414 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet 2474098 wikitext text/x-wiki ok1pdnub9n5v6ugrxew27xoiuelzwj7 2474099 2474098 2026-10-27T13:47:06Z Stashbot 7414 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply 2474099 wikitext text/x-wiki rkarb1sqf061er9ma5ipz4ucxg5a83h 2474100 2474099 2026-10-27T13:49:47Z Stashbot 7414 akhatun@deploy1003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply 2474100 wikitext text/x-wiki k81x0koeryba42gf6fur5euj1wtavcc 2474102 2474100 2026-10-27T13:51:10Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) ([[gerrit:1359263|1.47.0-wmf.24]], [[gerrit:1359261|1.48.0-wmf.1]]), [[gerrit:1358893|WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379)]] (duration: 34m 33s) 2474102 wikitext text/x-wiki q1g6527rcjelh055i9o16vr68ctzui1 2474103 2474102 2026-10-27T13:51:15Z Stashbot 7414 akhatun@deploy1003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply 2474103 wikitext text/x-wiki bk94el3c56zecxlhn8gx9j5v29496tv 2474104 2474103 2026-10-27T13:51:27Z Stashbot 7414 akhatun@deploy1003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply 2474104 wikitext text/x-wiki 0fwl60x17z3e8rwxml9cgmgv0gk5rn5 2474106 2474104 2026-10-27T13:52:26Z Stashbot 7414 akhatun@deploy1003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply 2474106 wikitext text/x-wiki tctjj0063du4ggim1n4axby5pms0txg 2474107 2474106 2026-10-27T13:52:30Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474107 wikitext text/x-wiki huwsdzslp50v6gvgaur678pcfxs69du 2474108 2474107 2026-10-27T13:52:58Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474108 wikitext text/x-wiki njay63keeh1e3rbhi6yvg4lrs37t1a7 2474109 2474108 2026-10-27T13:53:26Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474109 wikitext text/x-wiki gur1mkkldt7e3ozg8h1ttskd1n96zfi 2474111 2474109 2026-10-27T13:56:23Z Stashbot 7414 moritzm: failover Ganeti master in esams to ganeti3005 2474111 wikitext text/x-wiki dauv4higj9f3j4tz1ec2sxgf5aas5yl 2474114 2474111 2026-10-27T13:59:07Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474114 wikitext text/x-wiki rnoey1ve6ejjmtg8ogsf2yrnwcpsxmp 2474115 2474114 2026-10-27T13:59:19Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474115 wikitext text/x-wiki eeslwerbt1f3nsq3tj0xi808t2nkezk 2474117 2474115 2026-10-27T14:00:29Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474117 wikitext text/x-wiki psm0ju95xzmvcbjld7j60fsuyum1xnz 2474118 2474117 2026-10-27T14:01:22Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie 2474118 wikitext text/x-wiki 190jlqw709wsjwwfrt3xavjffotecwk 2474119 2474118 2026-10-27T14:02:01Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2474119 wikitext text/x-wiki 52xsx7bdv8ykovo9v5hkq7p3yxarwm3 2474120 2474119 2026-10-27T14:02:23Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) 2474120 wikitext text/x-wiki 3mt91aup4xhbge1di9347w88v4qx2w6 2474121 2474120 2026-10-27T14:02:37Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) 2474121 wikitext text/x-wiki 60lxhckr8e607pp7ocl01ctut62ibnx 2474122 2474121 2026-10-27T14:03:36Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) 2474122 wikitext text/x-wiki s60d1l337pgou7t4n1kyyblmucc2oj9 2474123 2474122 2026-10-27T14:03:53Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) 2474123 wikitext text/x-wiki sgtixj2s57igqbip3gk6g3w0rbbvahy 2474124 2474123 2026-10-27T14:03:56Z Stashbot 7414 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie 2474124 wikitext text/x-wiki 8esx33p83mm536qvmg61v4zwujqmluy 2474125 2474124 2026-10-27T14:05:01Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2474125 wikitext text/x-wiki s1vo6l1fxk437qy3bhh1rjq5zmjccbj 2474130 2474125 2026-10-27T14:09:27Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie 2474130 wikitext text/x-wiki 4im5e0255kc1o4j5wqunfew0i6kaihd 2474131 2474130 2026-10-27T14:09:37Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) 2474131 wikitext text/x-wiki j21nx4o1hzquro57ooiehs8opwh7nl8 2474132 2474131 2026-10-27T14:09:50Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) 2474132 wikitext text/x-wiki cunfr78cihvax9xt6o68hw615b54o5r 2474134 2474132 2026-10-27T14:10:11Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1359277|AbuseReview: Enable queues on several wikis (T442476 T442204)]] 2474134 wikitext text/x-wiki 0geb0emp0ipq3am0ed9v7r48og19crs 2474136 2474134 2026-10-27T14:10:50Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) 2474136 wikitext text/x-wiki bxfnuxhzfn4ahgm4z1hslfmbgpgeml1 2474137 2474136 2026-10-27T14:10:52Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2474137 wikitext text/x-wiki 5o5exyjzxexyuza6kuguvltwm707ywa 2474138 2474137 2026-10-27T14:10:56Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2474138 wikitext text/x-wiki eptfh4aqwuo6yxj4ldawkikc9ny4jhp 2474139 2474138 2026-10-27T14:11:04Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) 2474139 wikitext text/x-wiki g99p0t5dzqrz15grik3lcp3lw4854yy 2474140 2474139 2026-10-27T14:11:07Z Stashbot 7414 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie 2474140 wikitext text/x-wiki a5l1dfx0msmttrnfq1jh5swzjak200m 2474143 2474140 2026-10-27T14:13:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet 2474143 wikitext text/x-wiki slsulagikbqou4zckjqkzdto8ppocah 2474144 2474143 2026-10-27T14:14:25Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1359277|AbuseReview: Enable queues on several wikis (T442476 T442204)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474144 wikitext text/x-wiki jvhquixx6n7pqki7dxbcqe89ptarkai 2474145 2474144 2026-10-27T14:14:51Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.powercycle for host lvs5006 2474145 wikitext text/x-wiki saugw3dqrbhulnennhigz30wyr5pl38 2474150 2474145 2026-10-27T14:19:04Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet 2474150 wikitext text/x-wiki tpbha9xa21r35y8coc4b542522ucshw 2474151 2474150 2026-10-27T14:19:28Z Stashbot 7414 vgutierrez@cumin1004: END (ERROR) - Cookbook sre.hosts.powercycle (exit_code=97) for host lvs5006 2474151 wikitext text/x-wiki cmr11xj0fhqew8h7f7jm0r86nat2eob 2474152 2474151 2026-10-27T14:19:30Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet 2474152 wikitext text/x-wiki 5dqmkpy5load3pag3r3os1w2h1dumw7 2474153 2474152 2026-10-27T14:19:45Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2474153 wikitext text/x-wiki 355ghoxybiezhz1s34cy7sjzgph8ehm 2474154 2474153 2026-10-27T14:19:54Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie 2474154 wikitext text/x-wiki 0d5sa8vik5xt04s1lrspm76ole0d62i 2474155 2474154 2026-10-27T14:20:04Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) 2474155 wikitext text/x-wiki g3yukh0dmmqug0o2fvvcqd5yzfucm08 2474156 2474155 2026-10-27T14:20:16Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) 2474156 wikitext text/x-wiki 674sxa2kevgwgw3kqytwpuj4atlyp2n 2474157 2474156 2026-10-27T14:27:24Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1359277|AbuseReview: Enable queues on several wikis (T442476 T442204)]] (duration: 17m 13s) 2474157 wikitext text/x-wiki q5w264lic7z7andbuhm24ht9509in24 2474158 2474157 2026-10-27T14:28:05Z Stashbot 7414 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet 2474158 wikitext text/x-wiki 47f7tqd5vadjla5jbd8vtaeqwoh8jpx 2474159 2474158 2026-10-27T14:31:41Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) 2474159 wikitext text/x-wiki a4w6toak4u948zrxw5qfcfb2hgo7ow6 2474160 2474159 2026-10-27T14:31:58Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) 2474160 wikitext text/x-wiki 2dgb3qi5hvubig6fym2w8mgvk18jmb6 2474161 2474160 2026-10-27T14:32:00Z Stashbot 7414 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie 2474161 wikitext text/x-wiki 12f3wgzt67whpa9ghzvmzv6paticak3 2474162 2474161 2026-10-27T14:33:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet 2474162 wikitext text/x-wiki t34vhtfi7wxoaegxkwtbdkw8j6wkva2 2474163 2474162 2026-10-27T14:34:59Z Stashbot 7414 moritzm: installing Django security updates 2474163 wikitext text/x-wiki aech8024acg06tgddlkm6ifwbk9tk9i 2474165 2474163 2026-10-27T14:38:08Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1358370|xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223)]] 2474165 wikitext text/x-wiki qbyobxd2ozq1ndhhnf6n85vcpe6kmhr 2474167 2474165 2026-10-27T14:41:30Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2474167 wikitext text/x-wiki kihu74dpe43exbjfx4hl4ayu2jst33d 2474169 2474167 2026-10-27T14:41:50Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary 2474169 wikitext text/x-wiki f13ux7kz8wdfbc75lh96entgrfu1v0b 2474170 2474169 2026-10-27T14:42:20Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1358370|xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474170 wikitext text/x-wiki 6f07v0dq4hspjbqsvceppoj3w3ryjb1 2474171 2474170 2026-10-27T14:42:57Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie 2474171 wikitext text/x-wiki 8lvqpnp6w52fv6308bfp399mm9g5j9y 2474172 2474171 2026-10-27T14:43:04Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ping1004.eqiad.wmnet 2474172 wikitext text/x-wiki kk69xe8tkurh06aa8x9ipc9wvqrj2cw 2474173 2474172 2026-10-27T14:43:08Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) 2474173 wikitext text/x-wiki 6zn3bggsj04rpv71lxtfio1pt3ziids 2474174 2474173 2026-10-27T14:43:17Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) 2474174 wikitext text/x-wiki t33tvnp7li7srm9ygi1pzkc681udvgo 2474175 2474174 2026-10-27T14:43:43Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet 2474175 wikitext text/x-wiki 46uoggsnfjxjrsnwd23m1ocz6pcs8r8 2474177 2474175 2026-10-27T14:44:12Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox 2474177 wikitext text/x-wiki sglxqnhurjplxre3y9ljqew878cdd15 2474178 2474177 2026-10-27T14:44:39Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox 2474178 wikitext text/x-wiki lymhcgn1n5btsc282scm58teu9l3s5c 2474179 2474178 2026-10-27T14:45:51Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2474179 wikitext text/x-wiki d5lws5yo9fnwqa7nlwuzhi8dqsl3gzw 2474180 2474179 2026-10-27T14:46:54Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1004.eqiad.wmnet 2474180 wikitext text/x-wiki 9uqikz8kycqllt40kf58bhlq6oc61eq 2474181 2474180 2026-10-27T14:47:06Z Stashbot 7414 hashar: Restarting Gerrit 2474181 wikitext text/x-wiki 3782jwq4catoqb1zr03v7skbn2lwcks 2474182 2474181 2026-10-27T14:47:17Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2474182 wikitext text/x-wiki 49ry94t2uvlgzb61zqqq1csvn6jkof0 2474183 2474182 2026-10-27T14:47:29Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2474183 wikitext text/x-wiki jhwiwzxg8lehfdgzkkmoy2fp3yfnper 2474184 2474183 2026-10-27T14:49:05Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet 2474184 wikitext text/x-wiki 0w9ml3043m6mzk5o2oq2knedokdqfmn 2474185 2474184 2026-10-27T14:49:37Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet 2474185 wikitext text/x-wiki o1emijobcqbs6id40go8qa17c08q47h 2474186 2474185 2026-10-27T14:51:22Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2474186 wikitext text/x-wiki q628oyyn7n2s2nrbuwmix1o24univb5 2474187 2474186 2026-10-27T14:51:31Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1358370|xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223)]] (duration: 13m 23s) 2474187 wikitext text/x-wiki tii6xy1z2631kn6feto38g13c4wqa6k 2474188 2474187 2026-10-27T14:55:15Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2474188 wikitext text/x-wiki hu97gvkaxhnds9m27st5mffxsafi13g 2474189 2474188 2026-10-27T14:57:24Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet 2474189 wikitext text/x-wiki 5evhtfu23wbhq4vqgh9lw1j1dsslb4i 2474190 2474189 2026-10-27T14:57:43Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3008.esams.wmnet 2474190 wikitext text/x-wiki 34bmvqk2cp1cidzhx4waadu1r6q2fzp 2474192 2474190 2026-10-27T15:04:55Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2474192 wikitext text/x-wiki 056ebi4wlj38otrvowgofdzuoa3v25u 2474193 2474192 2026-10-27T15:06:07Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2474193 wikitext text/x-wiki fleeiawnu60xizndn0oipljuh3irbg1 2474194 2474193 2026-10-27T15:07:48Z Stashbot 7414 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet 2474194 wikitext text/x-wiki ls2sdh6fv5r0m22cmezhx1ujze83qoa 2474196 2474194 2026-10-27T15:08:37Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) 2474196 wikitext text/x-wiki 5ffowapiarr9gcpwfcqg5wcl4svahz8 2474197 2474196 2026-10-27T15:08:48Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) 2474197 wikitext text/x-wiki 8hdvk0h40nu192lnwjdf8fw7qdquio7 2474198 2474197 2026-10-27T15:08:50Z Stashbot 7414 vgutierrez@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie 2474198 wikitext text/x-wiki mkg85ldk27we1msadzmchpehimh3g1q 2474199 2474198 2026-10-27T15:08:58Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified] 2474199 wikitext text/x-wiki kz017n0es33ofvg02c5y80kf43qqgqg 2474200 2474199 2026-10-27T15:09:05Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified] 2474200 wikitext text/x-wiki q2it91x42mupqwsas4ytc2vxsryoqfd 2474201 2474200 2026-10-27T15:11:36Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: T439675 - DC Switchover 2474201 wikitext text/x-wiki j1yvu2s948verhvx36ux10p9hmjqpp1 2474202 2474201 2026-10-27T15:13:11Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie 2474202 wikitext text/x-wiki 0gtalm4rnqg3h8jc9ozoxjpaxv9e7pu 2474203 2474202 2026-10-27T15:13:23Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) 2474203 wikitext text/x-wiki o71twgoad7u9i4s3f3mq86lgb3pnx6s 2474204 2474203 2026-10-27T15:13:31Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) 2474204 wikitext text/x-wiki 83tej8k098l2v7x97qm6jpfym39cjsc 2474206 2474204 2026-10-27T15:16:41Z Stashbot 7414 slyngshede@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-api-ext-ro in eqiad: T439675 - DC Switchover 2474206 wikitext text/x-wiki 88hdp47svdg1sh3mkcuepmri9pi2031 2474207 2474206 2026-10-27T15:21:12Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.discovery.service-route check mw-api-ext-ro: maintenance 2474207 wikitext text/x-wiki 8e3gzcaa3cleq2fic7ftyx7xzhfhbvo 2474208 2474207 2026-10-27T15:21:17Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-ext-ro: maintenance 2474208 wikitext text/x-wiki c9a53v0cvz48t1abfw4yphntw0iua48 2474209 2474208 2026-10-27T15:32:39Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: T439675 - DC Switchover 2474209 wikitext text/x-wiki imj3iidv2t7nrw6ez4z3h9995e3gm7s 2474210 2474209 2026-10-27T15:33:20Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) 2474210 wikitext text/x-wiki p05bzofmfntdbklovrmx45cay9em2ui 2474211 2474210 2026-10-27T15:33:33Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) 2474211 wikitext text/x-wiki 3yw6e8g5mzauhqstgf2fq0q70xhonle 2474212 2474211 2026-10-27T15:33:38Z Stashbot 7414 vgutierrez@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie 2474212 wikitext text/x-wiki e49p6erm0qw0irlaoh3si8ro7kh0i68 2474213 2474212 2026-10-27T15:37:43Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie 2474213 wikitext text/x-wiki gzs3o4veipdwet461v8h07vhb04tats 2474214 2474213 2026-10-27T15:37:51Z Stashbot 7414 slyngshede@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: T439675 - DC Switchover 2474214 wikitext text/x-wiki dm12f4dgjwkr72m4ras7bhu8l29f0gi 2474215 2474214 2026-10-27T15:37:58Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) 2474215 wikitext text/x-wiki b6macp7sqrrk3gesu5f8im8g8447oen 2474216 2474215 2026-10-27T15:38:04Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) 2474216 wikitext text/x-wiki r7bt93noc1jzo505e3lv3bj3895xbm8 2474217 2474216 2026-10-27T15:46:27Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - T439675 2474217 wikitext text/x-wiki 5xcy3nrdwapjw67kqcshpc7o4jb3kvk 2474218 2474217 2026-10-27T15:49:32Z Stashbot 7414 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet 2474218 wikitext text/x-wiki g8ouwdjl5oj5cis65bo9ebqzoj0kucy 2474225 2474218 2026-10-27T16:05:31Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=eqsin,cluster=dnsbox 2474225 wikitext text/x-wiki no4fue079gcmt8rcbx4p19sctejunfb 2474226 2474225 2026-10-27T16:10:46Z Stashbot 7414 vgutierrez@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie 2474226 wikitext text/x-wiki e81wur9tbcp1awncg9dnvsjk62l8oe5 2474227 2474226 2026-10-27T16:12:24Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474227 wikitext text/x-wiki nsp3q2d5efc6vw5w0jzeu2oqqcru7lw 2474228 2474227 2026-10-27T16:12:43Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie 2474228 wikitext text/x-wiki rypd9hhjlc7ww0wrqxnqzkja1ceysao 2474229 2474228 2026-10-27T16:12:54Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) 2474229 wikitext text/x-wiki k3ha7qfr518zvuisvyo1sbn0w8hv5sa 2474230 2474229 2026-10-27T16:13:03Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) 2474230 wikitext text/x-wiki 9j7i59ievu5c8g0o3ppjs0xk0cmgak9 2474235 2474230 2026-10-27T16:22:33Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474235 wikitext text/x-wiki rw6q59rgsvm8sdr74p1xqw2qmd4to8w 2474239 2474235 2026-10-27T16:25:10Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - T439675 2474239 wikitext text/x-wiki 9xqpeaxh8uemrxklhyh71c7ourkg2g5 2474240 2474239 2026-10-27T16:26:56Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=eqsin,cluster=dnsbox 2474240 wikitext text/x-wiki d2yatxidcver9vxhtoawuhuatfm7pxj 2474241 2474240 2026-10-27T16:27:09Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2474241 wikitext text/x-wiki l2mx9jt032x6k7kpf6neuu3j5mbzhqw 2474242 2474241 2026-10-27T16:29:09Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.discovery.service-route pool eventstreams in codfw: T439675 - DC Switchover 2474242 wikitext text/x-wiki tp933ygfm5y7c1eax8f29p6hz95igwa 2474243 2474242 2026-10-27T16:29:34Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2474243 wikitext text/x-wiki baal3gbay1r3nxymz0995cthn4u6tnw 2474244 2474243 2026-10-27T16:30:51Z Stashbot 7414 fabfur@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet 2474244 wikitext text/x-wiki btf6zegklkxeivja6etkga5vn01li5o 2474245 2474244 2026-10-27T16:30:56Z Stashbot 7414 fabfur@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_esams 2474245 wikitext text/x-wiki 6ynwbhy0wotqpbbniyebkp0a3h850jz 2474248 2474245 2026-10-27T16:34:13Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool eventstreams in codfw: T439675 - DC Switchover 2474248 wikitext text/x-wiki 32uef5pe3aapg95trxza2a8xpmlhe6g 2474249 2474248 2026-10-27T16:36:33Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ping2004.codfw.wmnet 2474249 wikitext text/x-wiki n0jrco62augr5eccqky0n7576cvxe2y 2474252 2474249 2026-10-27T16:37:14Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.discovery.service-route depool eventstreams in eqiad: T439675 - DC Switchover 2474252 wikitext text/x-wiki gb9d3p1blmnuxzrrfb10axh4acfdj7n 2474254 2474252 2026-10-27T16:40:23Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2004.codfw.wmnet 2474254 wikitext text/x-wiki eoi7bligcsmm83wva3jpe34h75b2dyi 2474255 2474254 2026-10-27T16:41:18Z Stashbot 7414 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2474255 wikitext text/x-wiki stuot22uwvx8o7w7gv6pgv2xmhk7djo 2474256 2474255 2026-10-27T16:42:18Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool eventstreams in eqiad: T439675 - DC Switchover 2474256 wikitext text/x-wiki dixm1d0tu15a50airi0ihq90ryu78qn 2474257 2474256 2026-10-27T16:43:52Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) 2474257 wikitext text/x-wiki 2gwq8cwxqfcyoplnfuwan5lfujh3grb 2474258 2474257 2026-10-27T16:44:03Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) 2474258 wikitext text/x-wiki scfw3bqt55z8djiewjsnaphmqh7n0nj 2474259 2474258 2026-10-27T16:44:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3008.esams.wmnet 2474259 wikitext text/x-wiki 75hsb5hjmvo8rq10j2zh8p2ackek812 2474260 2474259 2026-10-27T16:46:37Z Stashbot 7414 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2474260 wikitext text/x-wiki 7ud60bgd8s28zm8thoxnsa3ebjzj0qd 2474261 2474260 2026-10-27T16:47:16Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet 2474261 wikitext text/x-wiki tnojyzi7hkm5bzjicki9s0s9zydaocc 2474262 2474261 2026-10-27T16:48:53Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5006.eqsin.wmnet with reason: host reimage 2474262 wikitext text/x-wiki c9l967g901c0v2lrnysbv8j0flzg660 2474263 2474262 2026-10-27T16:52:38Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet 2474263 wikitext text/x-wiki 0hrk7so64kq9kvgcl1bqzpqda7xqf2r 2474264 2474263 2026-10-27T16:52:48Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet 2474264 wikitext text/x-wiki 6mbcgc0xbxegmgq2crthvhqwu0exzl6 2474265 2474264 2026-10-27T16:53:02Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3008.esams.wmnet 2474265 wikitext text/x-wiki 052k3j81ytlkc90rsn4d29b0rhw0ug5 2474266 2474265 2026-10-27T16:53:08Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3008.esams.wmnet 2474266 wikitext text/x-wiki h0ni8y1kk5m9vz1qxeabb2zkxm3uqgh 2474268 2474266 2026-10-27T16:53:41Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5006.eqsin.wmnet with reason: host reimage 2474268 wikitext text/x-wiki ahw1juf9zn4l4zcck5zyfuabch7s5pi 2474272 2474268 2026-10-27T17:09:25Z Stashbot 7414 mutante: confctl: depooling releases.wikimedia.org in codfw - it is wrongly marked active/active but in reality it is not. only releases1003 is behind the releases.discovery.wmnet name - adjusting config to reality. 2474272 wikitext text/x-wiki lwu05nbmwetggvqg4jrjvofri584j1t 2474274 2474272 2026-10-27T17:11:24Z Stashbot 7414 dzahn@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=releases,name=codfw 2474274 wikitext text/x-wiki bdnr2fw77tq1ntu1wevhei7hkjc3gof 2474277 2474274 2026-10-27T17:17:23Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5006.eqsin.wmnet with OS trixie 2474277 wikitext text/x-wiki fa5o7micw1slph42jswuhsarl17ftbr 2474280 2474277 2026-10-27T17:31:28Z Stashbot 7414 mutante: codesearch: fixed the zombie process leak in hound - containers now run with --init (1358873) which reaps orphaned git processes; codesearch1001/2001 went from 160k+ zombies to 0 Now applying to codesearch10 (Cloud VPS, serves the current public codesearch), search incomplete for ~1h while it reindexes T268199 2474280 wikitext text/x-wiki q33m4hu8zeenuoezcu5ece7xcos5g1u 2474281 2474280 2026-10-27T17:31:47Z Stashbot 7414 akhatun@deploy1003: Started scap sync-world: Backport for [[gerrit:1359282|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695)]] 2474281 wikitext text/x-wiki 86w2snw8zvmis5nx29fcl2ylosyu9gm 2474282 2474281 2026-10-27T17:36:19Z Stashbot 7414 akhatun@deploy1003: akhatun, otto: Backport for [[gerrit:1359282|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474282 wikitext text/x-wiki g7co12l5zv2mtivq76l4yzidj7lj4ih 2474283 2474282 2026-10-27T17:37:23Z Stashbot 7414 akhatun@deploy1003: akhatun, otto: Rolling back deployment 2474283 wikitext text/x-wiki 9kq2s3817wkw7301mmqhoj8h8i7ra11 2474284 2474283 2026-10-27T17:39:45Z Stashbot 7414 akhatun@deploy1003: Failed scap sync-world: Backport for [[gerrit:1359282|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695)]] (duration: 07m 58s) 2474284 wikitext text/x-wiki dfze1jx5j43oi1y5gvovuyubuhulkhj 2474285 2474284 2026-10-27T17:44:18Z Stashbot 7414 rzl: rzl@apt1002:~$ sudo -i reprepro -C component/envoy-future include bookworm-wikimedia /home/rzl/envoyproxy_1.40.1-1_amd64.changes 2474285 wikitext text/x-wiki 50mony8kmvyly270z9fhrg6h7rw6zbi 2474286 2474285 2026-10-27T17:51:40Z Stashbot 7414 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1356976|EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426)]] 2474286 wikitext text/x-wiki 47j8o3urmy0bckvz6pk6t60k2gaeo6o 2474287 2474286 2026-10-27T17:53:36Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_eqiad and A:cp - 9.2.16 upgrade (T441945) 2474287 wikitext text/x-wiki cika774oatbsjoh0v3p71xsazvf4jcn 2474288 2474287 2026-10-27T17:53:42Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_eqiad and A:cp - 9.2.16 upgrade (T441945) 2474288 wikitext text/x-wiki ga0rkkrfl7ldhr5e8g76ua2231lrt0v 2474289 2474288 2026-10-27T17:56:09Z Stashbot 7414 otto@deploy1003: otto, ebysans: Backport for [[gerrit:1356976|EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474289 wikitext text/x-wiki nksm42yd6900wsr80fug5ei9w68wz1v 2474290 2474289 2026-10-27T17:57:37Z Stashbot 7414 otto@deploy1003: otto, ebysans: Continuing with deployment 2474290 wikitext text/x-wiki aisjli855v27msc225g37zlhwoovlw7 2474291 2474290 2026-10-27T18:00:12Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=dns4003.* 2474291 wikitext text/x-wiki c9uwe0522sepp2ady61epc7dgvt2rei 2474292 2474291 2026-10-27T18:03:21Z Stashbot 7414 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1356976|EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426)]] (duration: 11m 41s) 2474292 wikitext text/x-wiki j9tq44rcc8gnjkhlj300oe7k328py9g 2474293 2474292 2026-10-27T18:04:41Z Stashbot 7414 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-main: sync 2474293 wikitext text/x-wiki ehtki29h0vklc03rmy27j2e5bh68jk3 2474294 2474293 2026-10-27T18:04:52Z Stashbot 7414 ottomata: roll restarting eventgate-main to pick up stream config change 2474294 wikitext text/x-wiki 4reanfbbixhcptagkajtznhmfgim0mj 2474295 2474294 2026-10-27T18:05:02Z Stashbot 7414 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-main: sync 2474295 wikitext text/x-wiki 4ul8a50tdyigiby4bqim68e77m05zj4 2474299 2474295 2026-10-27T18:07:47Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync 2474299 wikitext text/x-wiki slr3x49w71owamdvpeg1b5epwbextkv 2474300 2474299 2026-10-27T18:08:08Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync 2474300 wikitext text/x-wiki p2l8gl6n959gdb3tf34cn3novrkbip0 2474301 2474300 2026-10-27T18:08:22Z Stashbot 7414 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync 2474301 wikitext text/x-wiki hohdbns8ztx3wtedh4hzebj99e2vfmx 2474302 2474301 2026-10-27T18:08:46Z Stashbot 7414 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync 2474302 wikitext text/x-wiki rb0hv823o0b6mbwztuvk9ejyzwqxbuy 2474304 2474302 2026-10-27T18:13:53Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns4003.* 2474304 wikitext text/x-wiki sqmvb9ru4yot0u27r3rca1098x38o81 2474307 2474304 2026-10-27T18:22:19Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_eqiad and A:cp - 9.2.16 upgrade (T441945) 2474307 wikitext text/x-wiki 7ue5t35gf0521v6lgrxf33x7095z1u7 2474311 2474307 2026-10-27T18:46:45Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_eqiad and A:cp - 9.2.16 upgrade (T441945) 2474311 wikitext text/x-wiki h8v4qruuzkke7z63cz58gzctx6t7nb9 2474312 2474311 2026-10-27T18:49:53Z Stashbot 7414 sandraebele: Deploying Refinery at 4a515e4 for change 1355145/T434136 Add per wiki filtering to edits/v3/per_editor 2474312 wikitext text/x-wiki q3vxlyeh1b7d2u13m1pvbsmwfozzk7a 2474313 2474312 2026-10-27T18:56:18Z Stashbot 7414 ebysans@deploy1003: Started deploy [analytics/refinery@4a515e4] (hadoop-test): Deploying Refinery Patch 1355145 and Ticket T434136 2474313 wikitext text/x-wiki 4rs0v8eb9ih7aw0feocv6cerqrtwvva 2474314 2474313 2026-10-27T18:57:01Z Stashbot 7414 ebysans@deploy1003: Finished deploy [analytics/refinery@4a515e4] (hadoop-test): Deploying Refinery Patch 1355145 and Ticket T434136 (duration: 00m 43s) 2474314 wikitext text/x-wiki 1yp8z45xxaxuz0dl8keruq35sw5kqng 2474315 2474314 2026-10-27T19:03:03Z Stashbot 7414 ebysans@deploy1003: Started deploy [analytics/refinery@4a515e4]: Deploying Refinery Patch 1355145 [analytics/refinery@4a515e4f] 2474315 wikitext text/x-wiki t6md1nao9xmw29bxvxn0mapbmde8i5b 2474316 2474315 2026-10-27T19:08:07Z Stashbot 7414 ebysans@deploy1003: Finished deploy [analytics/refinery@4a515e4]: Deploying Refinery Patch 1355145 [analytics/refinery@4a515e4f] (duration: 05m 03s) 2474316 wikitext text/x-wiki fffg0jmokzh3yqd6kw1jfodendmzzz7 2474318 2474316 2026-10-27T19:17:17Z Stashbot 7414 sandraebele: Deployed Refinery using Scap, and then deployed onto hdfs 2474318 wikitext text/x-wiki mhavjpjm3tjq1oojxw7sruklsly1r70 2474319 2474318 2026-10-27T19:35:13Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough 2474319 wikitext text/x-wiki 0nomt5800r1un0c9hylkbofhooa8o95 2474322 2474319 2026-10-27T19:46:41Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_codfw and A:cp - 9.2.16 upgrade (T441945) 2474322 wikitext text/x-wiki rxk9r9frawbukpr14kbb4zo0ujxoa6t 2474323 2474322 2026-10-27T19:46:46Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_codfw and A:cp - 9.2.16 upgrade (T441945) 2474323 wikitext text/x-wiki sgqdnssdn66fi5frsm5ndsjl0sqj00s 2474324 2474323 2026-10-27T19:48:17Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough 2474324 wikitext text/x-wiki euwjq47a3brcz4wjb983z7xg731j708 2474328 2474324 2026-10-27T20:07:06Z Stashbot 7414 robh@cumin1004: START - Cookbook sre.hosts.decommission for hosts sretest2005.codfw.wmnet 2474328 wikitext text/x-wiki 3o0w73rtv9ks8j3bloz5saqv0woglgr 2474329 2474328 2026-10-27T20:11:03Z Stashbot 7414 robh@cumin1004: START - Cookbook sre.dns.netbox 2474329 wikitext text/x-wiki bd3ssgb4xej4oyn4rlr5zvmadmh65ox 2474330 2474329 2026-10-27T20:11:17Z Stashbot 7414 hashar@deploy1003: Started scap sync-world: Backport for [[gerrit:1359334|[arwiki] Remove uploader user group and merge it with autoreview (T442578)]] 2474330 wikitext text/x-wiki cia51yjap266ouqu4oljckkk28a7ppn 2474331 2474330 2026-10-27T20:13:15Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_codfw and A:cp - 9.2.16 upgrade (T441945) 2474331 wikitext text/x-wiki b4oyp6i8pwudb4j68agpbovlm34saok 2474332 2474331 2026-10-27T20:14:46Z Stashbot 7414 rzl: rzl@apt1002:~$ sudo -i reprepro copy bookworm-wikimedia bullseye-wikimedia kubeconform 2474332 wikitext text/x-wiki czba850rscmwapdqqbyoi5wlodt0sph 2474333 2474332 2026-10-27T20:15:40Z Stashbot 7414 hashar@deploy1003: gergesshamon, hashar: Backport for [[gerrit:1359334|[arwiki] Remove uploader user group and merge it with autoreview (T442578)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474333 wikitext text/x-wiki ippfteqiakm4yu65qanbp686fhzs4pz 2474334 2474333 2026-10-27T20:18:00Z Stashbot 7414 robh@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - robh@cumin1004" 2474334 wikitext text/x-wiki awag7hsn5gsdcycrati81yolbyznfw0 2474335 2474334 2026-10-27T20:19:38Z Stashbot 7414 hashar@deploy1003: gergesshamon, hashar: Continuing with deployment 2474335 wikitext text/x-wiki b5m91x0anau2tz8xtwx0aiepy0rhe1g 2474336 2474335 2026-10-27T20:20:01Z Stashbot 7414 robh@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - robh@cumin1004" 2474336 wikitext text/x-wiki es8qajaq7o48fhezv8cezajlpa4c8k1 2474337 2474336 2026-10-27T20:20:06Z Stashbot 7414 robh@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2474337 wikitext text/x-wiki 1jv4t7kxt7l5vml3g3uh5urxhibxm3b 2474338 2474337 2026-10-27T20:20:11Z Stashbot 7414 robh@cumin1004: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts sretest2005.codfw.wmnet 2474338 wikitext text/x-wiki 7caizo4i6i2oysck7u4bxzpz2uq4at6 2474339 2474338 2026-10-27T20:22:51Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.dns.netbox 2474339 wikitext text/x-wiki m2bvvrix0wd55j25larfe49ev1kk8af 2474340 2474339 2026-10-27T20:25:24Z Stashbot 7414 hashar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1359334|[arwiki] Remove uploader user group and merge it with autoreview (T442578)]] (duration: 14m 07s) 2474340 wikitext text/x-wiki kryv3b5a0lvye4b96af81z07obajuec 2474341 2474340 2026-10-27T20:27:39Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: normalize - ayounsi@cumin1004" 2474341 wikitext text/x-wiki j9etk5q3wbab074jt5q1pndznoame4q 2474342 2474341 2026-10-27T20:28:52Z Stashbot 7414 hashar: arwiki: mwscript emptyUserGroup.php --wiki=arwiki uploader --create-log --log-reason 'uploader user group merged into autoreview - T442578' 2474342 wikitext text/x-wiki ce1z6bihnv8kwx4uxxz9aap48nginct 2474343 2474342 2026-10-27T20:31:10Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: normalize - ayounsi@cumin1004" 2474343 wikitext text/x-wiki ohocr6wjksz9wyfreedbn0jlgyqz381 2474344 2474343 2026-10-27T20:31:16Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2474344 wikitext text/x-wiki ebmxu5udf93ifkexrhc83eiwn9thffh 2474345 2474344 2026-10-27T20:32:08Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2474345 wikitext text/x-wiki 6grbmqrf1yt4ajssmboy5e7ea9d4tha 2474347 2474345 2026-10-27T20:39:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix IPv6 PTR - bking@cumin2003" 2474347 wikitext text/x-wiki 2mamd2qvxnff7c93rnat7wqeukfxpme 2474348 2474347 2026-10-27T20:39:28Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix IPv6 PTR - bking@cumin2003" 2474348 wikitext text/x-wiki 52uxpr9ora4r69xtnb2jdv3qbb9vibs 2474349 2474348 2026-10-27T20:39:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2474349 wikitext text/x-wiki himi3kqtuxeu2vimhl46i5t24jek9ko 2474350 2474349 2026-10-27T20:41:25Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_codfw and A:cp - 9.2.16 upgrade (T441945) 2474350 wikitext text/x-wiki 8us88u91uwsx5732i4dghh9rbpx2l3z 2474352 2474350 2026-10-27T21:03:43Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1358888|Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585)]] (1.48.0-wmf.1) 2474352 wikitext text/x-wiki 98tp8quwyd8lg69yo7vy5ulk0r7rxg1 2474353 2474352 2026-10-27T21:05:28Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1358888|Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474353 wikitext text/x-wiki f3yscdqxtgyihm2wo1sukbxofm4qxsu 2474355 2474353 2026-10-27T21:06:54Z Stashbot 7414 kamila@cumin1004: START - Cookbook sre.hosts.decommission for hosts deploy2002.codfw.wmnet 2474355 wikitext text/x-wiki gz69uao00yd5dk4qmef9hvsz9d7u7mp 2474356 2474355 2026-10-27T21:07:06Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2474356 wikitext text/x-wiki 6sodc08vjj8bz7vjleegyk6qftxd4jc 2474357 2474356 2026-10-27T21:08:00Z Stashbot 7414 kamila@cumin1004: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts deploy2002.codfw.wmnet 2474357 wikitext text/x-wiki 6kf7p7jrnlaurt3sba7pxk3nl85gl1o 2474358 2474357 2026-10-27T21:09:32Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1358888|Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585)]] (1.48.0-wmf.1) (duration: 05m 37s) 2474358 wikitext text/x-wiki enx7sqt65nyzzycl4ssmmcovvhgm28z 2474359 2474358 2026-10-27T21:14:20Z Stashbot 7414 andrewbogott: sudo cumin "gitlab-runner*" "systemctl restart ferm" on Tyler's request 2474359 wikitext text/x-wiki ing6q3n26n6c7s8bhzwv3a0wc412de4 2474360 2474359 2026-10-27T21:16:44Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1358273|Enable Vector 2022 CTA on test wikis (T441705)]] 2474360 wikitext text/x-wiki mpf76hlwbmsb6m0y6srlp74c6z1hcq8 2474361 2474360 2026-10-27T21:36:46Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1358273|Enable Vector 2022 CTA on test wikis (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474361 wikitext text/x-wiki 2sy855igc7p2u5bxh5fw2ovexasyu8a 2474362 2474361 2026-10-27T21:40:01Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2474362 wikitext text/x-wiki mczslnwd9q8xdefwpvt1cc4h4hqnmzw 2474363 2474362 2026-10-27T21:51:45Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1358273|Enable Vector 2022 CTA on test wikis (T441705)]] (duration: 35m 00s) 2474363 wikitext text/x-wiki ef2ztttntesux97vbi0fxx08ld3vyoh 2474364 2474363 2026-10-27T21:53:28Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1359401|Fix typo in config name (T441705)]] 2474364 wikitext text/x-wiki on33nmhczw0zux4f0eqxm4pxlmy9htf 2474366 2474364 2026-10-27T21:58:02Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1359401|Fix typo in config name (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474366 wikitext text/x-wiki qx2rr52ju6t8tu9gk289vr4ci7ch67d 2474367 2474366 2026-10-27T21:59:59Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2474367 wikitext text/x-wiki p41vb6tg4tgo6zjerbbyv14dke5wwqd 2474368 2474367 2026-10-27T22:07:55Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1359401|Fix typo in config name (T441705)]] (duration: 14m 28s) 2474368 wikitext text/x-wiki bvxyygff48a8x8bcqvi7htwxdmlz858 2474370 2474368 2026-10-27T22:49:08Z Stashbot 7414 mutante: planet2003,planet1003 - re-enabled puppet after gerrit:1359406 T442071 and being disabled for somet ime 2474370 wikitext text/x-wiki 3lqu3cwotkdsizrk2yjjo5e709j0ffb 2474372 2474370 2026-10-27T23:23:14Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2474372 wikitext text/x-wiki 6uh7gwrzqjs8q0r5t5yfdwmxz8mds82 2474373 2474372 2026-10-27T23:23:33Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2474373 wikitext text/x-wiki 1nucs27sfgvh1ty7r7ldkf7ypkao9rs 2474377 2474373 2026-10-27T23:38:25Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for ffmpeg: Actually apply the -speed argument ([[gerrit:1359411|1.48.0-wmf.1]], [[gerrit:1359412|1.47.0-wmf.24]]) 2474377 wikitext text/x-wiki 0hjgm5d5ysy3lbhrnxhulh0u3c4wp9g 2474378 2474377 2026-10-27T23:40:58Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2474378 wikitext text/x-wiki ipypgi8yur6z75kzn36h4z80gpcmxio 2474379 2474378 2026-10-27T23:41:22Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2474379 wikitext text/x-wiki avjzkuh3722nfv03dvynuwbev95kx15 2474380 2474379 2026-10-27T23:42:58Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for ffmpeg: Actually apply the -speed argument ([[gerrit:1359411|1.48.0-wmf.1]], [[gerrit:1359412|1.47.0-wmf.24]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474380 wikitext text/x-wiki kcxnr4e9csnl02nju7gtp0dw7etli7s 2474381 2474380 2026-10-27T23:43:54Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2474381 wikitext text/x-wiki mnk6jsqkw24odr1cyar0fgfu9jaq1l5 2474382 2474381 2026-10-27T23:49:33Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for ffmpeg: Actually apply the -speed argument ([[gerrit:1359411|1.48.0-wmf.1]], [[gerrit:1359412|1.47.0-wmf.24]]) (duration: 11m 08s) 2474382 wikitext text/x-wiki gtdtgmvli4ncoq9q7ghmbjz02wy1nhz 2474385 2474382 2026-10-28T00:00:11Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync 2474385 wikitext text/x-wiki 7glrrq9jphf7eqfz14qhvkfwyljks6r 2474386 2474385 2026-10-28T00:00:18Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync 2474386 wikitext text/x-wiki 7idw8xi29zaofy8nc9uaqkdu82oo7ec 2474387 2474386 2026-10-28T00:00:22Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync 2474387 wikitext text/x-wiki 4rdb0uxg3jxtijnx0zb9woxr8fnxwuz 2474388 2474387 2026-10-28T00:00:30Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync 2474388 wikitext text/x-wiki ftx18g8qqpksl7tch8vp9z3sa4vi4hl 2474390 2474388 2026-10-28T00:20:06Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply 2474390 wikitext text/x-wiki e3jont48er6a1ziiplh9li8fqrxzbko 2474391 2474390 2026-10-28T00:20:37Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply 2474391 wikitext text/x-wiki 0cb7rhtgkp4uhmufh2puktlr56pccbu 2474393 2474391 2026-10-28T01:01:07Z Stashbot 7414 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply 2474393 wikitext text/x-wiki tg9czr3obgh5mkj0v65cfesl8aotsxd 2474394 2474393 2026-10-28T01:01:43Z Stashbot 7414 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply 2474394 wikitext text/x-wiki efk88d1anoizgym57kzbdphwjszr8cz 2474395 2474394 2026-10-28T01:08:00Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply 2474395 wikitext text/x-wiki 99i2106yy1jlzrqq15hilzxnkac0m6d 2474396 2474395 2026-10-28T01:08:14Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply 2474396 wikitext text/x-wiki qylzqfz28l4jrl1l8uh11z7ekjqvd1h 2474397 2474396 2026-10-28T01:19:40Z Stashbot 7414 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply 2474397 wikitext text/x-wiki otyagwq268j7fcbolyd5p40kff5o611 2474398 2474397 2026-10-28T01:19:45Z Stashbot 7414 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply 2474398 wikitext text/x-wiki 4camtswaesh3lhjq5ez7nrxda823w6y 2474399 2474398 2026-10-28T01:20:55Z Stashbot 7414 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply 2474399 wikitext text/x-wiki gpvx9ciuxpf4yxxewqdpllt7h0nswoj 2474400 2474399 2026-10-28T01:21:30Z Stashbot 7414 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply 2474400 wikitext text/x-wiki 3ubxz0eowehup9g99z89aamyimnpyjf 2474401 2474400 2026-10-28T01:25:54Z Stashbot 7414 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply 2474401 wikitext text/x-wiki 2lf541dqw53g9ve9lf9pybhuem611g5 2474402 2474401 2026-10-28T01:27:02Z Stashbot 7414 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply 2474402 wikitext text/x-wiki 5pdo1xyy26n1unkgie4oqzixl90hhyp 2474403 2474402 2026-10-28T01:32:44Z Stashbot 7414 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply 2474403 wikitext text/x-wiki 3kmrcm5z5rn784atvrnzn4uf6ewlll3 2474404 2474403 2026-10-28T01:33:39Z Stashbot 7414 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply 2474404 wikitext text/x-wiki 9cg6d943432o6dq4675tczf5ll9z79i 2474405 2474404 2026-10-28T02:01:23Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2474405 wikitext text/x-wiki l6rz8ot171wvnsznospnj6jldr2qc38 2474407 2474405 2026-10-28T02:09:00Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) 2474407 wikitext text/x-wiki 2oe4fthxg710z9msopoblv2a2bxpoao 2474425 2474407 2026-10-28T07:22:40Z Stashbot 7414 hashar@deploy1003: Started deploy [gerrit/gerrit@b02d300]: Update Zuul plugin to align CSS styling - T441684 2474425 wikitext text/x-wiki qk0dezfpjieawkh4e5akatjy22ysz4m 2474426 2474425 2026-10-28T07:22:54Z Stashbot 7414 hashar@deploy1003: Finished deploy [gerrit/gerrit@b02d300]: Update Zuul plugin to align CSS styling - T441684 (duration: 00m 14s) 2474426 wikitext text/x-wiki c1ff1g083bohvs93kh8ir1qv7t9bdoj 2474427 2474426 2026-10-28T07:36:56Z Stashbot 7414 hashar: Restarted Gerrit on gerrit1003 2474427 wikitext text/x-wiki c0idepfk3j384af1vqnul13t1h2w48a 2474428 2474427 2026-10-28T08:09:02Z Stashbot 7414 moritzm: failover Ganeti master in eqiad to ganeti1048 2474428 wikitext text/x-wiki p31kf4i9pglvlpyvfo6v4crabw8ne8z 2474430 2474428 2026-10-28T08:12:31Z Stashbot 7414 moritzm: failover Ganeti master in codfw to ganeti2048 2474430 wikitext text/x-wiki mps3eyfp6pt4szpahh4i8geg0d5jxo7 2474431 2474430 2026-10-28T08:16:59Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet 2474431 wikitext text/x-wiki 2rhxm55g4dft925r8zdhxsr45pw7t4c 2474432 2474431 2026-10-28T08:17:03Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2034.codfw.wmnet 2474432 wikitext text/x-wiki 2cln00om91m3pzm82dis5bxajklgfgy 2474433 2474432 2026-10-28T08:17:44Z Stashbot 7414 moritzm: installing clamav security updates 2474433 wikitext text/x-wiki 791ro6219hporkv0qkk2nx1bja4u7xf 2474434 2474433 2026-10-28T08:33:10Z Stashbot 7414 moritzm: installing perl security updates 2474434 wikitext text/x-wiki fwsiwftx6lkvl8mon3dqoh35i3vnuvw 2474435 2474434 2026-10-28T08:35:53Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet 2474435 wikitext text/x-wiki e1kw57p1tsg49qj0oeknaxzif2bah0i 2474436 2474435 2026-10-28T08:35:58Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2034.codfw.wmnet 2474436 wikitext text/x-wiki oukvow5a8yateh1z2yiz87jzocyyukw 2474437 2474436 2026-10-28T08:36:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet 2474437 wikitext text/x-wiki 3r1pk74kiyk2ra3nq51k7it7ukyqgk5 2474438 2474437 2026-10-28T08:36:57Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet 2474438 wikitext text/x-wiki pvz1oxey7gokyiwu550vambvy03w3m2 2474439 2474438 2026-10-28T08:40:26Z Stashbot 7414 vgutierrez: repooling cp2049 2474439 wikitext text/x-wiki 6gcabb7xb990ve6cgmasp4g4zvyj8mi 2474440 2474439 2026-10-28T08:44:49Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2474440 wikitext text/x-wiki khfz51xw3ovm8mvvjnwirwou97uzm8q 2474441 2474440 2026-10-28T08:45:02Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet 2474441 wikitext text/x-wiki p3tsv1hmyvhps0x291phyvma7j85e3z 2474442 2474441 2026-10-28T08:45:11Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet 2474442 wikitext text/x-wiki bxcvk6y0av6kutb1k8ij55asnt4wxe7 2474443 2474442 2026-10-28T08:46:32Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2474443 wikitext text/x-wiki qubt3t259utwboe8r1xejuuskafa76c 2474444 2474443 2026-10-28T08:48:39Z Stashbot 7414 moritzm: failover Ganeti master in codfw02 to ganeti2034 2474444 wikitext text/x-wiki 6s32zyq2qcz8sfgf04dfaecrr2b205c 2474445 2474444 2026-10-28T08:48:44Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29 2474445 wikitext text/x-wiki 8a71s6hkq6rky4p3g3tla0dzn5b64zw 2474446 2474445 2026-10-28T08:48:52Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29 2474446 wikitext text/x-wiki dsdmlwncjuwech6eq8leb056157qadi 2474447 2474446 2026-10-28T08:49:20Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29 2474447 wikitext text/x-wiki 5waiubwlf4dg205easetmm0mflt3x4d 2474448 2474447 2026-10-28T08:49:29Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29 2474448 wikitext text/x-wiki karqvsgup9cpaji8edmgqbyzfg8e443 2474458 2474448 2026-10-28T08:58:03Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29 2474458 wikitext text/x-wiki iyp9y4ndjbabsu7x3kmn8x8zlnt2pzr 2474459 2474458 2026-10-28T08:58:11Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29 2474459 wikitext text/x-wiki d5394sdri713hum32xio86vwvq1ty9e 2474460 2474459 2026-10-28T09:10:29Z Stashbot 7414 javiermonton@deploy1003: Started scap deploy-service pageview-trending-relative-next: testing checks for Flink deployments. 2474460 wikitext text/x-wiki jzxkqahe2adwn580l0v0z7f3yvvk1be 2474461 2474460 2026-10-28T09:10:34Z Stashbot 7414 javiermonton@deploy1003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next 2474461 wikitext text/x-wiki iukml4g1b5n0ii47o0hodz45g74polk 2474462 2474461 2026-10-28T09:12:36Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet 2474462 wikitext text/x-wiki 6sbgdoret02375npq5j34c71yzwe2h3 2474463 2474462 2026-10-28T09:14:27Z Stashbot 7414 javiermonton@deploy1003: Finished scap deploy-service pageview-trending-relative-next: testing checks for Flink deployments. (duration: 03m 58s) 2474463 wikitext text/x-wiki 22aa9upf8n2an7cpxyjlvlkyrhbb6j7 2474464 2474463 2026-10-28T09:22:02Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474464 wikitext text/x-wiki d3w51w9i82debbrjijfaun0ceibefao 2474465 2474464 2026-10-28T09:22:07Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474465 wikitext text/x-wiki etmy9mnpt4is6zi48jyppj4cfz7qte1 2474466 2474465 2026-10-28T09:22:11Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2474466 wikitext text/x-wiki olx7dc4cdv628xb2b1smbqr5apx7l9y 2474467 2474466 2026-10-28T09:22:16Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474467 wikitext text/x-wiki m573hx5smpttvk9mlbzqmsdmmnka2g7 2474468 2474467 2026-10-28T09:22:20Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474468 wikitext text/x-wiki 4j1lbxmeff2jscwr1i5aqh9e8tnkecl 2474469 2474468 2026-10-28T09:22:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2033.codfw.wmnet 2474469 wikitext text/x-wiki p8obeeikxbfg5sbxhn8rv6vftsmlq3a 2474470 2474469 2026-10-28T09:22:34Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474470 wikitext text/x-wiki jzxijkg8q3v7io41cey9rsu12271xf4 2474471 2474470 2026-10-28T09:22:48Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474471 wikitext text/x-wiki otr3vgeiocknjfqgoxh6c25z83w0taq 2474472 2474471 2026-10-28T09:23:14Z Stashbot 7414 moritzm: installing jq security updates 2474472 wikitext text/x-wiki jq0cz60bjnulkm0cj8hgd5len6e1evm 2474473 2474472 2026-10-28T09:24:56Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2474473 wikitext text/x-wiki 3qpcdzez8xa0fh7kiiwbkvwze873u12 2474474 2474473 2026-10-28T09:25:40Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2474474 wikitext text/x-wiki 90o2nov9gwk4s5cyiuu5il08ukckv39 2474475 2474474 2026-10-28T09:30:32Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2033.codfw.wmnet 2474475 wikitext text/x-wiki d6ukh6hftz5yatma5l42diu09clwskb 2474476 2474475 2026-10-28T09:30:36Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet 2474476 wikitext text/x-wiki 5tundpoctycfyp8t39l8wgq867g5r26 2474478 2474476 2026-10-28T09:40:01Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for Fix two bugs on PurgeScoreCache (T440358) ([[gerrit:1359767|1.47.0-wmf.24]], [[gerrit:1359768|1.48.0-wmf.1]]) 2474478 wikitext text/x-wiki tsz4m0h66573fd8aeboz6vojcugufl0 2474480 2474478 2026-10-28T09:44:28Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for Fix two bugs on PurgeScoreCache (T440358) ([[gerrit:1359767|1.47.0-wmf.24]], [[gerrit:1359768|1.48.0-wmf.1]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474480 wikitext text/x-wiki fpltroq8kyz4m4u9yteb5u18c8xn4lj 2474481 2474480 2026-10-28T09:45:37Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2474481 wikitext text/x-wiki p2w0ap219s7k5bxgxhiyky7ljiv4hdm 2474482 2474481 2026-10-28T09:47:54Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474482 wikitext text/x-wiki iyaf7f97p0pp8d3kgsxf0bye2sfqvk7 2474483 2474482 2026-10-28T09:47:59Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474483 wikitext text/x-wiki mfbet772ef2prbpb9sv577694mrf1b5 2474484 2474483 2026-10-28T09:48:04Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474484 wikitext text/x-wiki 0qlx8izqn7dkgberd32k8miojsfnhq3 2474485 2474484 2026-10-28T09:48:11Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474485 wikitext text/x-wiki ceeq1btlh8uokp7bq28bq6cc82vgy6p 2474486 2474485 2026-10-28T09:49:56Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474486 wikitext text/x-wiki n79e7czp11jzo0aqdvp31vx5uicdjpv 2474487 2474486 2026-10-28T09:50:02Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2474487 wikitext text/x-wiki 7tcrudw5512rxr1qz6whtb4rj4j1x7n 2474488 2474487 2026-10-28T09:50:07Z Stashbot 7414 moritzm: installing nodejs security updates on bookworm 2474488 wikitext text/x-wiki ckympvwnjtt4a0nva6kus3vaq3fve2h 2474489 2474488 2026-10-28T09:50:28Z Stashbot 7414 javiermonton@deploy1003: Started scap deploy-service pageview-trending-relative-next: Test Flink checks in spiderpig 2474489 wikitext text/x-wiki rympj9ihqhn2hqo07g7qmp6wfghqtt7 2474490 2474489 2026-10-28T09:50:32Z Stashbot 7414 javiermonton@deploy1003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next 2474490 wikitext text/x-wiki 74p2qzffhq9f1rko1s7xw8kfoh4l0vy 2474491 2474490 2026-10-28T09:51:20Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for Fix two bugs on PurgeScoreCache (T440358) ([[gerrit:1359767|1.47.0-wmf.24]], [[gerrit:1359768|1.48.0-wmf.1]]) (duration: 11m 20s) 2474491 wikitext text/x-wiki dsmgn5ud9hbgvsfa2jhu2q96b3eytf6 2474492 2474491 2026-10-28T09:51:49Z Stashbot 7414 javiermonton@deploy1003: Finished scap deploy-service pageview-trending-relative-next: Test Flink checks in spiderpig (duration: 01m 21s) 2474492 wikitext text/x-wiki h08pf97i35ryc6ngze3elofw6fekkda 2474493 2474492 2026-10-28T09:52:12Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user amire80 2474493 wikitext text/x-wiki 20mi0t4q3jmc47ocwi9msfbu5xwmxrv 2474494 2474493 2026-10-28T09:52:24Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user amire80 2474494 wikitext text/x-wiki cxxuos3zl0rpwtyr5t79tqkb1cagyf2 2474495 2474494 2026-10-28T09:55:10Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user amire80 2474495 wikitext text/x-wiki glc2lwl0pmmo1isnil2c1i4e0zupy3l 2474496 2474495 2026-10-28T09:55:23Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user amire80 2474496 wikitext text/x-wiki 3zo9ouct4crskuccw5lwcrxna7ou7xk 2474497 2474496 2026-10-28T10:07:21Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2474497 wikitext text/x-wiki eo2vtntrs5ndrpqyfk5tekbue5puxxe 2474498 2474497 2026-10-28T10:07:53Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2474498 wikitext text/x-wiki 67xa520mnn4u6oxzm9eyi4tljzpwxgi 2474499 2474498 2026-10-28T10:08:08Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2474499 wikitext text/x-wiki 7h99qu5uxh136thxdxp1dr7wqn7rbcy 2474500 2474499 2026-10-28T10:08:35Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2474500 wikitext text/x-wiki r620d9zk11fik1xgq39mecubvovj7x3 2474501 2474500 2026-10-28T10:08:53Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2474501 wikitext text/x-wiki ngtwt4oqynrlc601wlk8hk0dbafqqws 2474502 2474501 2026-10-28T10:09:14Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2474502 wikitext text/x-wiki pus6atpoi3nt7quf5uwpr03gd63vqwv 2474503 2474502 2026-10-28T10:21:38Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2047.codfw.wmnet 2474503 wikitext text/x-wiki 3qi4aqu875p5tlm5srp0ee2v7vw6jmu 2474504 2474503 2026-10-28T10:24:33Z Stashbot 7414 moritzm: installing Perl security updates 2474504 wikitext text/x-wiki an39cnowmpu0ey0qye80ypdb458uuki 2474505 2474504 2026-10-28T10:24:43Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2047.codfw.wmnet 2474505 wikitext text/x-wiki 8kru7bcbxlgph6lib16tg326y04qb1q 2474506 2474505 2026-10-28T10:26:47Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org 2474506 wikitext text/x-wiki mzp8tdfozc5wltncsi8hzgrosqf86ch 2474507 2474506 2026-10-28T10:30:04Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2047.codfw.wmnet 2474507 wikitext text/x-wiki a13e188tyv3g1wc1qlpe08tn1kv89dk 2474508 2474507 2026-10-28T10:30:33Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2047.codfw.wmnet 2474508 wikitext text/x-wiki 7ot1uxk18ify26vp16tbatxop5jfbqp 2474509 2474508 2026-10-28T10:32:27Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org 2474509 wikitext text/x-wiki piza509a780ztjfbd8buc6o9y08qjac 2474510 2474509 2026-10-28T10:33:09Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2474510 wikitext text/x-wiki qn7p98txz0wh1k59382z1xnmaepbyjh 2474511 2474510 2026-10-28T10:34:21Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2474511 wikitext text/x-wiki gvqmsm23p8yujha79i1zdinrn2rem0s 2474512 2474511 2026-10-28T10:36:01Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet 2474512 wikitext text/x-wiki sgflq67weuxc31aj5hz84fd7e3193i1 2474513 2474512 2026-10-28T10:40:46Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet 2474513 wikitext text/x-wiki 8hv0gd9tej4rgr44sncipaqk5lprssz 2474514 2474513 2026-10-28T10:42:35Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica 2474514 wikitext text/x-wiki 7gm27wrssiuaamafptj6exdvpsfyve2 2474515 2474514 2026-10-28T10:42:58Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica 2474515 wikitext text/x-wiki 8p7rij2rviiqpa5rb8pridnau9s9wf0 2474517 2474515 2026-10-28T10:46:09Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet 2474517 wikitext text/x-wiki 4mcx9eu6bqcal37xiw0cabdpu9szeob 2474518 2474517 2026-10-28T10:46:15Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet 2474518 wikitext text/x-wiki tsc51nwi77e4x5wfdrny6xedq3xjl75 2474519 2474518 2026-10-28T10:46:29Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user west1 2474519 wikitext text/x-wiki k4yh2w75w81whsx8nai7v2j7510dw4g 2474520 2474519 2026-10-28T10:46:39Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user west1 2474520 wikitext text/x-wiki kgtr6hpm0zp5dv4gmef8nm3r0ib7zab 2474521 2474520 2026-10-28T10:49:12Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5005.eqsin.wmnet with OS trixie 2474521 wikitext text/x-wiki aih5qunuir79b4znnbpjh46x5545afy 2474523 2474521 2026-10-28T10:50:15Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5005.eqsin.wmnet (push config only) 2474523 wikitext text/x-wiki mp24to0nr41896bf00ueqe9y4vuxq51 2474524 2474523 2026-10-28T10:50:24Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5005.eqsin.wmnet (push config only) 2474524 wikitext text/x-wiki hr37waoz9ms2bjv96jviowb4cjg0gj0 2474525 2474524 2026-10-28T10:55:31Z Stashbot 7414 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 6 hosts with reason: migrate row a/b network uplinkks from CRs to new Nokia spines 2474525 wikitext text/x-wiki 4up5aauf3hlmsimmji31vrvkwt17f7s 2474526 2474525 2026-10-28T10:57:11Z Stashbot 7414 topranks: shut down cr1-eqiad links connecting asw2-a-eqiad to shift traffic to cr2-eqiad equivalent T441014 2474526 wikitext text/x-wiki 6zsy8qbpgrh1zgnn9a84eq2wn6p8of4 2474527 2474526 2026-10-28T11:02:42Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user arinaigum 2474527 wikitext text/x-wiki f5z1v3xkpblpob0qx46c9gwapzqi74b 2474528 2474527 2026-10-28T11:03:05Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user arinaigum 2474528 wikitext text/x-wiki 9xvw5ncfd2roc348rmesedctjtiei3v 2474530 2474528 2026-10-28T11:06:58Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user joelyrookewmde 2474530 wikitext text/x-wiki f9a0gg28fbr4ppdd4ime0p2sorlejon 2474531 2474530 2026-10-28T11:07:07Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user joelyrookewmde 2474531 wikitext text/x-wiki npybsj0x67m7dni2zfbr8vw44pkxyy3 2474532 2474531 2026-10-28T11:08:15Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user joelyrookewmde 2474532 wikitext text/x-wiki jw9xkp3fou870825k47ywyp26i1jy20 2474533 2474532 2026-10-28T11:08:24Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user joelyrookewmde 2474533 wikitext text/x-wiki 7f1nm7m49ree19osukkuf4pr8wyxyhh 2474535 2474533 2026-10-28T11:12:06Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user resquito 2474535 wikitext text/x-wiki q7k581pjbwa5cdcigkd3ll6wa7pbz5u 2474536 2474535 2026-10-28T11:12:15Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user resquito 2474536 wikitext text/x-wiki c0fczgpwswqafe14ai3apme6afrfkcx 2474537 2474536 2026-10-28T11:14:47Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5005.eqsin.wmnet (remove config only) 2474537 wikitext text/x-wiki 6vo7e07fncts76uqdhz2mjjuiholl68 2474538 2474537 2026-10-28T11:14:59Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5005.eqsin.wmnet (remove config only) 2474538 wikitext text/x-wiki bkmz5y40sqqkw2dujtb5pibqffa1xok 2474539 2474538 2026-10-28T11:18:29Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2474539 wikitext text/x-wiki k1teztl37ozbarcryxynke5ce8bv4vq 2474540 2474539 2026-10-28T11:18:54Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5005.eqsin.wmnet with reason: host reimage 2474540 wikitext text/x-wiki q1gwcsqh25f2osikqntt0vi067qhpy2 2474541 2474540 2026-10-28T11:19:08Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2474541 wikitext text/x-wiki 81e9s2su0m2b6dhq3t422qao94w8edj 2474543 2474541 2026-10-28T11:22:59Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.backup.offsite 2474543 wikitext text/x-wiki p49q5ni1tohog9zcy64fk2ch0hq342j 2474544 2474543 2026-10-28T11:23:04Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0) 2474544 wikitext text/x-wiki f999w0fvjxlj75dvqwhrl3j0u14cgms 2474545 2474544 2026-10-28T11:23:32Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5005.eqsin.wmnet with reason: host reimage 2474545 wikitext text/x-wiki rcxuvp1qhypejx5fo3g9yup5fgrdbny 2474546 2474545 2026-10-28T11:25:35Z Stashbot 7414 hnowlan@dns1004: START - running authdns-update 2474546 wikitext text/x-wiki g8nnrhp7bf6qhwtyf29v7qj5wenvbi8 2474547 2474546 2026-10-28T11:27:08Z Stashbot 7414 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2474547 wikitext text/x-wiki j4cfmnycjang9lvlu3wjtzqxsjp0nr1 2474548 2474547 2026-10-28T11:27:26Z Stashbot 7414 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2474548 wikitext text/x-wiki t2xf1gopd69zyo3dkyqms4bvcyrf2jr 2474549 2474548 2026-10-28T11:27:58Z Stashbot 7414 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2474549 wikitext text/x-wiki p49x9gvh1g0arl6sowbk69ac2zkxk0c 2474550 2474549 2026-10-28T11:28:36Z Stashbot 7414 hnowlan@dns1004: END - running authdns-update 2474550 wikitext text/x-wiki qb8darjyfi7yopsgrum1b39r5souhld 2474551 2474550 2026-10-28T11:28:41Z Stashbot 7414 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2474551 wikitext text/x-wiki qcjo342u602z180qi8y57s9dekurs9e 2474552 2474551 2026-10-28T11:28:52Z Stashbot 7414 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply 2474552 wikitext text/x-wiki 6g2xuon4eiuzzrrqeoxhu6khc7nih3l 2474553 2474552 2026-10-28T11:29:36Z Stashbot 7414 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply 2474553 wikitext text/x-wiki 6htyacbbvs5yyq46tsbdwhxui6vhxah 2474555 2474553 2026-10-28T11:35:01Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.backup.offsite 2474555 wikitext text/x-wiki nskjqqen89wcwz272hgojpihfj8nrmo 2474556 2474555 2026-10-28T11:35:06Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0) 2474556 wikitext text/x-wiki k86pveeo7y8hqs4f9gy8zhr3h8ll8fj 2474560 2474556 2026-10-28T11:39:41Z Stashbot 7414 hnowlan@dns1004: START - running authdns-update 2474560 wikitext text/x-wiki pd2d3krn89t63qvyiw3r4fc5o1d8ah1 2474561 2474560 2026-10-28T11:40:15Z Stashbot 7414 mvolz@deploy1003: Started scap deploy-service citoid: update 2474561 wikitext text/x-wiki koibcl8w0p9lthazzbwbwu8fc22n1yv 2474562 2474561 2026-10-28T11:40:20Z Stashbot 7414 mvolz@deploy1003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid 2474562 wikitext text/x-wiki 98ok600w69m60q5vg086habmyylbq6i 2474563 2474562 2026-10-28T11:42:48Z Stashbot 7414 hnowlan@dns1004: END - running authdns-update 2474563 wikitext text/x-wiki a9vvw117u9g4wo7yytxq8jxb9410ivv 2474564 2474563 2026-10-28T11:46:18Z Stashbot 7414 mvolz@deploy1003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid 2474564 wikitext text/x-wiki b9dck7om99e5q51t232akylo4vjbdad 2474565 2474564 2026-10-28T11:46:22Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5005.eqsin.wmnet with OS trixie 2474565 wikitext text/x-wiki qb01i9avsnbost0cdkks327yieo49gx 2474566 2474565 2026-10-28T11:52:11Z Stashbot 7414 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model damaging --old (T440358) 2474566 wikitext text/x-wiki sd1nwz15jew9u4jp4vk910ld1gnchaj 2474567 2474566 2026-10-28T11:56:56Z Stashbot 7414 hnowlan@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase 2474567 wikitext text/x-wiki 3eain81cdgq4qfayrl96vom9qozea3t 498 16087 2474354 2473120 2026-10-27T21:05:33Z Stashbot 7414 andrewbogott: redeploying from git 'main' branch in /home/root/quarry on quarry-bastion.quarry.eqiad1.wikimedia.cloud 2474354 wikitext text/x-wiki 2ysfblw90oqlqnwowgda5rwgi31sb9k 0 17290 2474247 2473950 2026-10-27T16:32:07Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/132 (T442495) 2474247 wikitext text/x-wiki rwttrtmmmcwz40xzkzckkt0llq2fthj 2474276 2474247 2026-10-27T17:16:48Z Stashbot 7414 dancy: Rearming keyholder on deployment-deploy06.deployment-prep 2474276 wikitext text/x-wiki j2vm4hbzxdphkbimatrxzmefmzwgw55 2474309 2474276 2026-10-27T18:41:23Z Stashbot 7414 dancy: journalctl --vacuum-size 256M on deployment-mediawiki15.deployment-prep.eqiad1.wikimedia.cloud to free up space 2474309 wikitext text/x-wiki brhbcujlpsltaw9azwqpe0gwk95knyr 2474310 2474309 2026-10-27T18:41:45Z Stashbot 7414 dancy: journalctl --vacuum-size 256M on deployment-jobrunner06.deployment-prep.eqiad1.wikimedia.cloud to free up space 2474310 wikitext text/x-wiki 4o1q07lizj0uc2zcdfr093z8jwv2amw 2474351 2474310 2026-10-27T20:57:23Z Stashbot 7414 dancy: Updated docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1359390 and friends 2474351 wikitext text/x-wiki 0to6nbix8l6xstnaiyizb3klr0hmsdd 2474365 2474351 2026-10-27T21:55:43Z Stashbot 7414 Southparkfan: Beta Cluster: remove PHP 8.3 config remnants on mediawiki and jobrunner hosts; packages were in dpkg 'rc' state (i.e. not installed, but config files are present) - T442482 2474365 wikitext text/x-wiki 5056qnrfd6dfsqbqp22kvui0h0ndvnw 2474369 2474365 2026-10-27T22:10:45Z Stashbot 7414 Southparkfan: Beta Cluster: install mediawiki-libs-shellbox "2026-10-22-141212-video85" tag - T442482 2474369 wikitext text/x-wiki ja40chpvjymqxoysfovulkx8ednjgb2 2474371 2474369 2026-10-27T23:04:24Z Stashbot 7414 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1359407 2474371 wikitext text/x-wiki o5t9ux1pzjx15gmxdll5je723ca29if 2474375 2474371 2026-10-27T23:30:03Z Stashbot 7414 James_F: Zuul: [labs/tools/wikiinfo] Add mediawiki-i18n-check template, for T441762 2474375 wikitext text/x-wiki tlsuzzlzcgrs9bzyboxyd8dgzn6yw3l 2474376 2474375 2026-10-27T23:33:51Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/134 (T441915, T442649) 2474376 wikitext text/x-wiki dbf8oqj7qj0mjau2umbbzx9i94rcb3p 498 30942 2474166 2473816 2026-10-27T14:41:07Z Stashbot 7414 andrewbogott: "openstack role create secgroupadmin" for T442554 2474166 wikitext text/x-wiki i2muu9dccp4ty7myuqm70nfr59uqkws 2474168 2474166 2026-10-27T14:41:45Z Stashbot 7414 andrewbogott: openstack role add --project paws --user magnum secgroupadmin T442554 2474168 wikitext text/x-wiki o59ykmf5xbfll4q8gtgz7h5zbtx0dka 498 32945 2474321 2318281 2026-10-27T19:43:50Z Stashbot 7414 taavi: arm keyholder 2474321 wikitext text/x-wiki oktkd8cx0a5bst3coifdspmky0h96d3 0 33560 2474221 2441111 2026-10-27T15:55:36Z JWodstrcil (WMF) 24826 add restart instrucitons for Phabricator 2474221 wikitext text/x-wiki jucjj9jd1y41ssttn1egt1fi3pimwm0 2474223 2474221 2026-10-27T16:01:13Z JWodstrcil (WMF) 24826 2474223 wikitext text/x-wiki mdcy44jg7cqtw6pp6mcz2pzfhw9cm25 498 55577 2474116 2470807 2026-10-27T13:59:25Z Stashbot 7414 taavi: migrate tools.paws to https://wikitech.wikimedia.org/wiki/Tool:Containers#Redirect_container 2474116 wikitext text/x-wiki j1p22frx4xiu91lm98l67tcjyqi4rwn 2474522 2474116 2026-10-28T10:49:20Z Stashbot 7414 dcaro: deleted old 127e cluster 2474522 wikitext text/x-wiki 307sm4nc9n9dogralaud4etwtm7uj1f 498 441911 2474278 2441348 2026-10-27T17:27:25Z Stashbot 7414 mutante: codesearch10: re-enabling puppet to apply 1358873 (hound containers with --init) - restarts all hound-* containers, search incomplete for ~1h while indexes rebuild. Fixes leaked zombie git processes (houndd as PID 1 did not reap them; prod hosts reached 160k+ and ran out of threads) T26819 2474278 wikitext text/x-wiki h0dwnbunf2rpasi4zos39fbb5p55vv7 2474279 2474278 2026-10-27T17:27:57Z Stashbot 7414 mutante: codesearch10: re-enabling puppet to apply 1358873 (hound containers with --init) - restarts all hound-* containers, search incomplete for ~1h while indexes rebuild. Fixes leaked zombie git processes (houndd as PID 1 did not reap them; prod hosts reached 160k+ and ran out of threads) T268199 2474279 wikitext text/x-wiki 9twfmuuvt174lkjgnqdmfxyj8bp990f 2474374 2474279 2026-10-27T23:23:38Z Stashbot 7414 mutante: - up and running and now has 0 git zombie processes - since it now runs with docker --init to tini can reap git zombies, which hound as pid 1 would not do. before it was about 1000 new git processes per day. in prod it ran out of pids within days. 2474374 wikitext text/x-wiki 38q9j230aigd77m8t3nvfxb2wfjfwj3 498 442806 2474135 1872405 2026-10-27T14:10:43Z Stashbot 7414 taavi: migrate to php 8.4 2474135 wikitext text/x-wiki lzo7e55bnvenlibhv8e9uuomkoipgun 498 443628 2474071 2473959 2026-10-27T13:06:36Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591155 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474071 wikitext text/x-wiki dhm1krjy2mat0pj5bsq33iu28aqen3x 2474105 2474071 2026-10-27T13:51:52Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108700 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5) 2474105 wikitext text/x-wiki q155zcoqxmm0l12i3mjreyjducfwntn 2474129 2474105 2026-10-27T14:09:16Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784252 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016) 2474129 wikitext text/x-wiki 1xaa86bhuzsu1qgyzbaxf4xl4gaxfsn 2474149 2474129 2026-10-27T14:16:01Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783783 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d) 2474149 wikitext text/x-wiki 174tlran7uf4hyz7rukve7df2q6hye9 2474219 2474149 2026-10-27T15:53:08Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588912 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106) 2474219 wikitext text/x-wiki 4x458a3t2w7gbs6wk3r95pnn4lozn44 2474232 2474219 2026-10-27T16:17:23Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846790 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c) 2474232 wikitext text/x-wiki ddl9nw291ujmmg3f1p1o7irvj4clq05 2474251 2474232 2026-10-27T16:36:55Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112168 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020) 2474251 wikitext text/x-wiki swzqzk8a74w9kf2z5syrqz8ssbj9l77 2474408 2474251 2026-10-28T02:32:34Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021123 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf) 2474408 wikitext text/x-wiki qpxh0ygtfrvtedlmo9hwitv1bx3i5m6 2474418 2474408 2026-10-28T05:18:23Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879022 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99) 2474418 wikitext text/x-wiki jno4j2gerrl45c555qysrna71787ya7 2474421 2474418 2026-10-28T05:41:32Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987512 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc) 2474421 wikitext text/x-wiki f3z2oeq2pdqi58am5zz8muanh03bg2p 2474455 2474421 2026-10-28T08:55:21Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788312 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474455 wikitext text/x-wiki 3don7jrd8vcgfpgpufwcbzzc649slnq 0 443686 2474542 2404995 2026-10-28T11:20:37Z Mvolz (WMF) 2813 2474542 wikitext text/x-wiki fdek33y5m4bltzfmlo6c9s25z2acsli 0 444275 2474392 2473462 2026-10-28T00:52:07Z RLazarus (WMF) 15215 remove API Gateway, now decommissioned 2474392 wikitext text/x-wiki 6if94cjb1nhdm5aebu0l1uff49hq4sw 498 446211 2474069 2473929 2026-10-27T13:06:28Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508590936 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474069 wikitext text/x-wiki 85jc27h3y68aeth02ikoo3i4y3pho7l 2474452 2474069 2026-10-28T08:54:56Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788418 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474452 wikitext text/x-wiki mqlo6xw9ejihx0xcyez26cqwmn3eit7 0 449160 2474389 2473594 2026-10-28T00:02:22Z Dexbot 30554 Bot: Updating the report 2474389 wikitext text/x-wiki k4ojow68hpkcq6xufga594xjdexh9i2 498 451089 2474320 2002364 2026-10-27T19:39:57Z Stashbot 7414 taavi: port to php 8.4 2474320 wikitext text/x-wiki fpf1z5mtlm9ajfyzblru89zxc1f1h00 498 452320 2474141 2373109 2026-10-27T14:11:36Z Stashbot 7414 taavi: migrate to php 8.4, filed T442540 2474141 wikitext text/x-wiki rzskdtowh1w3ljlofv3dla3sswccvot 498 452890 2474072 2473958 2026-10-27T13:07:07Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591099 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474072 wikitext text/x-wiki 9dt6xdgi9jx7oi4bbtxe67a8vlq3ke6 2474133 2474072 2026-10-27T14:09:51Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784076 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016) 2474133 wikitext text/x-wiki qbtr37nxtjy5ld8zkgfbnqpcjqhk945 2474148 2474133 2026-10-27T14:15:49Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783532 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d) 2474148 wikitext text/x-wiki ld5awkahpmwygpinfiogul0ala020l1 2474220 2474148 2026-10-27T15:53:30Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588887 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106) 2474220 wikitext text/x-wiki 0f7a2tunhvmxax1jyazfqe82fmciztu 2474234 2474220 2026-10-27T16:18:20Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846798 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c) 2474234 wikitext text/x-wiki 1l5xc2zii0ro33eagrn568tn53mk8ju 2474253 2474234 2026-10-27T16:37:39Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112186 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020) 2474253 wikitext text/x-wiki fxmbdqli7le89ecd2cznh1odqxddpvv 2474273 2474253 2026-10-27T17:10:33Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40547448775 (https://github.com/cluebotng/component-configs/commits/d8e5c2b81841263635a002cdb005aaa7328042b8) 2474273 wikitext text/x-wiki 4k15tx7lnxfnrna52e2f0vx4n4cnxj4 2474409 2474273 2026-10-28T02:32:56Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021329 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf) 2474409 wikitext text/x-wiki cdd9ttrdexu4tzmviuufayqdv61ie0f 2474420 2474409 2026-10-28T05:19:09Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879060 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99) 2474420 wikitext text/x-wiki f0ng1x42yev3d57dqzo3ra5ma19mld0 2474422 2474420 2026-10-28T05:42:21Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987481 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc) 2474422 wikitext text/x-wiki 4hvc9rjl8gy9htr8pveq8zz9m1if9tr 2474457 2474422 2026-10-28T08:56:02Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788346 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474457 wikitext text/x-wiki 1lkjet4jsy0szt04oemdtdiw3atejsq 116 453426 2474411 2473646 2026-10-28T04:47:38Z Phabbanbot 37210 20261028.1319 was disabled by AntiCompositeNumber 2474411 wikitext text/x-wiki 9cnp7dmalwss1vlvtbpwregpuvw3sja 2474412 2474411 2026-10-28T04:49:39Z Phabbanbot 37210 261028.1308 was disabled by AntiCompositeNumber 2474412 wikitext text/x-wiki nehiggogcsops8vppg543cxv7ceuncw 2474413 2474412 2026-10-28T04:50:34Z Phabbanbot 37210 20261028.1319 was disabled by VeritasSapientiae 2474413 wikitext text/x-wiki bac7423fmchkfxdhm1v2137utfwe53d 2474414 2474413 2026-10-28T04:50:43Z Phabbanbot 37210 261028.1307 was disabled by AntiCompositeNumber 2474414 wikitext text/x-wiki bevdu6skpnalqffweunz7cdjpnttt3g 2474416 2474414 2026-10-28T04:55:56Z Phabbanbot 37210 20261028.1319 was disabled by VeritasSapientiae 2474416 wikitext text/x-wiki ggt86ot4z7kldh6xvg8sc3ww13kxeeo 116 453906 2474534 2473647 2026-10-28T11:09:15Z Gitlabaccountapprovalbot 37332 tolland was rejected. 2474534 wikitext text/x-wiki 3ek4612kyziw7sl7el49rm0k3dwttu7 0 457955 2474146 2473811 2026-10-27T14:15:04Z ASanford-WMF 52089 /* 4. Add the public keys to puppet repo for production access */ Clarify instructions for requesting first-time production access vs replacing existing keys 2474146 wikitext text/x-wiki szzu49a7y5vzad9izt4umtrtpw7sy7i 2474267 2474146 2026-10-27T16:53:38Z Matma Rex 562 Add Windows notes, some formatting tweaks 2474267 wikitext text/x-wiki 7hgebk8xvjl422fa5po12dolup64chn 498 458787 2474066 2473776 2026-10-27T13:06:00Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591043 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474066 wikitext text/x-wiki 1s3wb2ngugug1fq8v8rrue172pz3id4 2474101 2474066 2026-10-27T13:50:48Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108521 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5) 2474101 wikitext text/x-wiki 9cutz9w1mfcnmohdtwn8mhvw88vy5dt 2474147 2474101 2026-10-27T14:15:15Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783500 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d) 2474147 wikitext text/x-wiki dvsl5yjvhfet3ndp54vcy0t377of5x5 2474250 2474147 2026-10-27T16:36:51Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112336 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020) 2474250 wikitext text/x-wiki sz0gxvsyz4ykymddyrwpovvbogjlp7a 2474419 2474250 2026-10-28T05:18:35Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636878933 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99) 2474419 wikitext text/x-wiki nz6ggm89zdo1kv44ie0r2d8whkkdgh0 2474451 2474419 2026-10-28T08:54:43Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788367 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474451 wikitext text/x-wiki ioy8q51bh0gld2h9yqwhq8uwaqr9nhy 0 459043 2474237 2417150 2026-10-27T16:23:44Z AStein-WMF 39485 2474237 wikitext text/x-wiki 7wd04ie80h6mz7kt5e6af8xuz9elbhh 0 459115 2474246 2464210 2026-10-27T16:31:04Z AndrewBogott (WMF) 51047 /* node images */ 2474246 wikitext text/x-wiki 1dfx3enzzl575kani58i9v62cvc0092 498 459243 2474070 2473902 2026-10-27T13:06:30Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508590861 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474070 wikitext text/x-wiki e4e7hzzvw8eddox2ljau7lsed2s6xzi 2474191 2474070 2026-10-27T15:02:49Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40526418448 (https://github.com/cluebotng/component-configs/commits/9c1e7dfb720d9ef57fe0e066dcebe9ccd5701eaf) 2474191 wikitext text/x-wiki ozb3b2xn0cqgelx5694k0je7a89lzfx 2474195 2474191 2026-10-27T15:07:51Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40527035651 (https://github.com/cluebotng/component-configs/commits/7dd9e7d26ac0141af54d06f81163a130e5a402d3) 2474195 wikitext text/x-wiki q5t37gtsz9tszkc9z6pmlzgu1fx51lg 2474224 2474195 2026-10-27T16:05:22Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40536663681 (https://github.com/cluebotng/component-configs/commits/d02fac71cd9a7c00d0b2cba2a245b95677c46c4d) 2474224 wikitext text/x-wiki rpwxww4hxt4hnjoogropw4i3z8m8nfc 2474269 2474224 2026-10-27T16:58:59Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40545550695 (https://github.com/cluebotng/component-configs/commits/9a4c6ecd52da46f16ac808838c87f2b4608e2987) 2474269 wikitext text/x-wiki rh5dd0xxhzhpd8uxrqsfwd8iltjf282 2474415 2474269 2026-10-28T04:54:19Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40632642704 (https://github.com/cluebotng/component-configs/commits/611facf7667c39a7f5b1712e5b174684a0649ea4) 2474415 wikitext text/x-wiki o1h2kx0qt3fyz9bpy7z6o6a742h6cva 2474424 2474415 2026-10-28T06:30:12Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40643749649 (https://github.com/cluebotng/component-configs/commits/6aa169a5ad67427223a008319e43e00be9db16c5) 2474424 wikitext text/x-wiki imq92c8pg9361k9uyyzxxe4qwq5tstw 2474456 2474424 2026-10-28T08:55:47Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788397 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474456 wikitext text/x-wiki amtiwysizwuc3mwiqfw7y66ko8j9egv 498 459244 2474067 2473863 2026-10-27T13:06:06Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591039 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474067 wikitext text/x-wiki m7dgk4wqpbl6c6z9859x3qwmclgmuq3 2474128 2474067 2026-10-27T14:09:13Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517784751 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016) 2474128 wikitext text/x-wiki faqkiwjuru9qz38mdf6zj0c3iud9xi5 2474142 2474128 2026-10-27T14:12:49Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517885090 (https://github.com/cluebotng/component-configs/commits/fbcf68ea18c187fbb8af37ed04a674837d121903) 2474142 wikitext text/x-wiki cxso11d4zbcmu607nqr0dh1shwkd44h 2474233 2474142 2026-10-27T16:17:47Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846848 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c) 2474233 wikitext text/x-wiki abdu62vus3k0j3hlxg3wqnqe4zry6yy 2474453 2474233 2026-10-28T08:54:59Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788352 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474453 wikitext text/x-wiki s011knf8jqrymx2ssvsy5z41dglxhtq 2474477 2474453 2026-10-28T09:39:51Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40665456675 (https://github.com/cluebotng/component-configs/commits/cbd8bb224c15c6dacdf93632ef29a11586a0b50b) 2474477 wikitext text/x-wiki ftppfybr6ukrtmbqhwka4c6phda65o5 498 459255 2474068 2473657 2026-10-27T13:06:10Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591062 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474068 wikitext text/x-wiki ko7rzl4occh65tbjudv7taucfmrxi1x 2474449 2474068 2026-10-28T08:54:22Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788301 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474449 wikitext text/x-wiki dfwg8kex6rycfchkjvijho0rtf70bxr 498 459263 2474064 2473658 2026-10-27T13:05:58Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508592543 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474064 wikitext text/x-wiki 4unu5zy691x7vo1eyj70oalh70klnq7 2474454 2474064 2026-10-28T08:55:01Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788309 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474454 wikitext text/x-wiki m4zu3kvvzoz2jhhefylkk8tx8cn4bct 0 459290 2474222 2473287 2026-10-27T15:55:37Z APizzata-WMF 47941 2474222 wikitext text/x-wiki hqqh1986ove770tx5xj7cqk8t2mouaq 2474317 2474222 2026-10-27T19:14:01Z SNwachukwu (WMF) 39521 Update deployed item and deployer 2474317 wikitext text/x-wiki svgx14dcjvovciv1jrhdl3a55pb7y90 498 459362 2474065 2473656 2026-10-27T13:05:59Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591170 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) 2474065 wikitext text/x-wiki 5fapmjs97rbfkgnuw90p926ym3rkgp1 2474450 2474065 2026-10-28T08:54:32Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788311 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) 2474450 wikitext text/x-wiki 499yyj59q7uticf5vwtq2ccek9qq48i 0 459699 2474326 2473586 2026-10-27T19:53:58Z Stashbot 7414 eileen: cv upgraded from b8a8dd6a to 77eb9b15 2474326 wikitext text/x-wiki gyrhq3xzxi4p4742uvi46qmobv4mjb3 2474327 2474326 2026-10-27T20:03:05Z Stashbot 7414 eileen: civicrm upgraded from 7980eec7 to 1b87ea3d 2474327 wikitext text/x-wiki 5xj35knjrfww6ys4fhcu6j3z7h3d4em 2474406 2474327 2026-10-28T02:03:27Z Stashbot 7414 eileen: civicrm upgraded from 1b87ea3d to 8b129c5d 2474406 wikitext text/x-wiki 3c7lnc0uhsgyb3qq710clr92htjntjh 2474410 2474406 2026-10-28T02:51:47Z Stashbot 7414 eileen: civicrm upgraded from 8b129c5d to 0d103d9e 2474410 wikitext text/x-wiki 17t5il04irk5zhh9p5p68bygebggg4w 2474417 2474410 2026-10-28T05:07:36Z Stashbot 7414 eileen: civicrm upgraded from 55c473fb to e42d41de 2474417 wikitext text/x-wiki 3xea80cds101i7o5y9tz7nnck0yeh7y 2474423 2474417 2026-10-28T06:01:30Z Stashbot 7414 eileen: civicrm upgraded from e42d41de to 9ec2f297 2474423 wikitext text/x-wiki 82k824y22yizfg6euv2jnerfuyydpqm 0 460417 2474055 2472275 2026-10-27T12:50:54Z BWojtowicz-WMF 42268 Edit summary after migration to llmisvc 2474055 wikitext text/x-wiki oiqb5v1f7bq8aw39fka9546l8rhzhln 2474056 2474055 2026-10-27T12:52:59Z BWojtowicz-WMF 42268 Edit "What it offers" not after llmisvc migration 2474056 wikitext text/x-wiki qhny8lqlxa7o0n16s96z193n7ji31t4 2474059 2474056 2026-10-27T12:58:16Z BWojtowicz-WMF 42268 Update Available Models section after llmisvc migration 2474059 wikitext text/x-wiki 6vlfkwbh5vkuymfuete6k9i5ksugjf9 2474060 2474059 2026-10-27T13:01:16Z BWojtowicz-WMF 42268 Edit Endpoints section after llmisvc migration 2474060 wikitext text/x-wiki 4v3twsw0kpctnt2kaenakr521ywl03k 2474087 2474060 2026-10-27T13:24:27Z BWojtowicz-WMF 42268 Edit deployment information section after llmisvc migration 2474087 wikitext text/x-wiki qg87viauu5nlldfnb5x6gst6wmk460s 0 460521 2474231 2472399 2026-10-27T16:16:27Z APaskulin (WMF) 14880 Link to wiki page 2474231 wikitext text/x-wiki mr6vm9is43mz803oi7nikx0ljvv396a 0 460627 2474126 2473625 2026-10-27T14:06:38Z AKhatun (WMF) 50461 Update logstash link 2474126 wikitext text/x-wiki def8qttpynzwegc3t7fev69zglq4tq5 2474127 2474126 2026-10-27T14:07:26Z AKhatun (WMF) 50461 Fix link 2474127 wikitext text/x-wiki kndhf7zp9eu0lvf55qg2etq5sa4m1l1 2474164 2474127 2026-10-27T14:35:47Z AKhatun (WMF) 50461 /* Lambda */ 2474164 wikitext text/x-wiki df6al592g7ax9nlr96uye5wvq3w5c0v 2474236 2474164 2026-10-27T16:23:37Z AKhatun (WMF) 50461 /* Lambda */ Finish alerts runbook 2474236 wikitext text/x-wiki couckghsndouv0wr0t0hkncqqnm3ksn 2474238 2474236 2026-10-27T16:24:21Z AKhatun (WMF) 50461 /* Lambda */ 2474238 wikitext text/x-wiki mjlkd2m33kzmne1uxegyn9g7r3mdk44 2474298 2474238 2026-10-27T18:07:46Z AKhatun (WMF) 50461 /* Deployment */ Add roll restart command 2474298 wikitext text/x-wiki 1uso9beoj17nlpx099w90wjlni59zzk 2474306 2474298 2026-10-27T18:15:34Z AKhatun (WMF) 50461 /* Deployment */ more k8s commands 2474306 wikitext text/x-wiki rj2z4m3817p71c4w0m9fyhcztopdzn9 2474308 2474306 2026-10-27T18:34:38Z AKhatun (WMF) 50461 /* Deployment */ 2474308 wikitext text/x-wiki anlbncvnit4qgmersqs18p9rwrzbgg7 0 460646 2474110 2473554 2026-10-27T13:55:37Z BKing (WMF) 29017 /* Milestones */ 2474110 wikitext text/x-wiki nsktr28vih28cxpfotxyq3i9ddweby6 0 460688 2474479 2472630 2026-10-28T09:42:28Z GGoncalves-WMF 42848 /* What is KAPOW? */ Remove mention of old heuristics, the link at the bottom of the page is good enough. 2474479 wikitext text/x-wiki oxiasp3acg4696yvxgvuedjkfhagcgx 0 460791 2474383 2468734 2026-10-27T23:51:06Z This, that and the other 921 /* What are other differences between the legacy dumps and the alternatives? */ add bullet point linking to [[MediaWiki Content File Exports#Differences with the legacy XML Dumps]] 2474383 wikitext text/x-wiki lizumolfzbwptav0vne9z0b1ozhr4l4 2474384 2474383 2026-10-27T23:52:09Z This, that and the other 921 /* What are other differences between the legacy dumps and the alternatives? */ clarify 2474384 wikitext text/x-wiki pb9e4b7vsb2praydg7f0efiz355tdc8 0 460839 2474052 2471995 2026-10-27T12:44:30Z Saper 135 +detailed project status 2474052 wikitext text/x-wiki k46qf6g9sc25co7081fih5bc88ptuuh 1 460842 2474176 2473875 2026-10-27T14:44:07Z Kuldeepburjbhalaike 37135 /* Toolforge (no PHP) */ 2474176 wikitext text/x-wiki gj4qwcyqslg9eddmiig5lc76h86xqrv 0 460881 2474053 2473893 2026-10-27T12:48:28Z Taavi-WMF 41365 Taavi-WMF moved page [[Talk:Incidents/2026-10-21 PHP security update/Project status]] to [[Incidents/2026-10-21 PHP security update/Project status]]: not a talk page really 2473893 wikitext text/x-wiki qcdstv996bgyj02ay5oy0jc7nonu6mn 2474112 2474053 2026-10-27T13:56:51Z AndrewBogott (WMF) 51047 /* fully restored */ Added many projects that were crossed off on the doc but didn't make it here yet 2474112 wikitext text/x-wiki ielyc7sfyb6dr7io2bpe4ce3mfkshrm 2474113 2474112 2026-10-27T13:57:44Z AndrewBogott (WMF) 51047 /* fully restored */ sorted 2474113 wikitext text/x-wiki jdvtopysskqn4revac95mgg9mephjcw 2474346 2474113 2026-10-27T20:35:20Z Taavi-WMF 41365 restore search 2474346 wikitext text/x-wiki 503hgkfqbgl3b4i2d2mqmoqnoiku3zy 2474429 2474346 2026-10-28T08:10:11Z Taavi-WMF 41365 mediawiki-quickstart 2474429 wikitext text/x-wiki sbk5bxnwv8mooafbvxt8wz4amu0midc 1 460893 2474054 2026-10-27T12:48:28Z Taavi-WMF 41365 Taavi-WMF moved page [[Talk:Incidents/2026-10-21 PHP security update/Project status]] to [[Incidents/2026-10-21 PHP security update/Project status]]: not a talk page really 2474054 wikitext text/x-wiki qb3w1nrgm4al74kdnpas4gg5e1glf3x 0 460894 2474271 2026-10-27T17:07:16Z FNegri-WMF 32595 Import list from https://phabricator.wikimedia.org/P97005 2474271 wikitext text/x-wiki q45wqlw1307eiz2zc8ey383qjpzl089 2474296 2474271 2026-10-27T18:07:22Z FNegri-WMF 32595 FNegri-WMF moved page [[Incidents/2026-10-21 PHP security update/Community/Deprecated tools]] to [[Incidents/2026-10-21 PHP security update/Community/Tools running deprecated PHP versions]]: Use a more descriptive title 2474271 wikitext text/x-wiki q45wqlw1307eiz2zc8ey383qjpzl089 2474305 2474296 2026-10-27T18:14:21Z FNegri-WMF 32595 Sort list alphabetically 2474305 wikitext text/x-wiki ckutsw5g9mc27a1m2i7w8ikwmavubbb 3 460895 2474275 2026-10-27T17:11:30Z StrikerBot 8475 Welcome to Toolforge! 2474275 wikitext text/x-wiki me2p85m5k4czo55wd1jznbtfhhb4sff 0 460896 2474297 2026-10-27T18:07:22Z FNegri-WMF 32595 FNegri-WMF moved page [[Incidents/2026-10-21 PHP security update/Community/Deprecated tools]] to [[Incidents/2026-10-21 PHP security update/Community/Tools running deprecated PHP versions]]: Use a more descriptive title 2474297 wikitext text/x-wiki m4ypixvfd0d0mg5diuc6tceoc9pb8tz 3 460897 2474325 2026-10-27T19:52:07Z StrikerBot 8475 Welcome to Toolforge! 2474325 wikitext text/x-wiki jeklytgv5fidksrzt17jppcbfgvh2oi 0 460898 2474516 2026-10-28T10:45:40Z DCaro (WMF) 18271 Created page with "{{Toolforge nav admin}} {{Requires admin permissions banner}} The '''MaintainDBUsersPopulationFailing''' happens when maintain-dbusers had unexpected issues when populating account (fetching accounts from the different systems and creating entries in it's own database). == Debugging == Maintain-dbusers lives in one of the cloudcontrol nodes, to find out which: <syntaxhighlight lang=shell-session> 02:41 PM ~/Work/wikimedia/operations-puppet (production|✔) dcaro@acme$..." 2474516 wikitext text/x-wiki 18c50eckamqeux5jdg7o8m8q0p6auam