Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.12 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2441067 2441051 2026-07-27T12:06:08Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers (T432325) 2441067 wikitext text/x-wiki b2zt9hipv1nulfipzf3iuwaplkrmpi7 2441243 2441067 2026-07-27T16:59:09Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2441243 wikitext text/x-wiki f9j49nqh2q5ip5u7uvww6tjz63lv3r2 2441248 2441243 2026-07-27T17:12:19Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2441248 wikitext text/x-wiki klwd5lan1zpeoaxsh6buhfsbijupelv 2441250 2441248 2026-07-27T17:18:20Z Stashbot 7414 dcaro: increase object storage object number quota to 1M (T433281) 2441250 wikitext text/x-wiki l7hou4d75p9b975zzl8uthd7oq4ii7j 2441412 2441250 2026-07-28T06:55:51Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs 2441412 wikitext text/x-wiki 6u0mmji3760ykfrtoq53qd0q0tzergn 2441413 2441412 2026-07-28T06:56:17Z Stashbot 7414 godog: resume nfs workers roll restart - T432325 2441413 wikitext text/x-wiki a63q51xyz5nlebbvqpuoqiws65ied5o 2441451 2441413 2026-07-28T08:25:45Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, t 2441451 wikitext text/x-wiki tr6odvsok1c057iovv9dns3bep2ykt6 0 3420 2441426 1887933 2026-07-28T07:20:36Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Obsolete:History of job queue runners at WMF]] 2441426 wikitext text/x-wiki 4ui92mu6yqrxlqa95oc8lieseqrn9hc 0 4108 2441094 2440938 2026-07-27T13:17:17Z Lucas Werkmeister (WMDE) 10037 /* {{Deployment_day|date=2026-07-29}} */ add one-off deployment window for [[phab:T231755|T231755]] 2441094 wikitext text/x-wiki h0t2mpjd7fuppqi9m911tjy2a1pu10k 2441264 2441094 2026-07-27T17:41:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1318149]] to Monday, July 27 UTC late backport window 2441264 wikitext text/x-wiki euaadu42dlf5mn91x3go6s722abqfsi 2441265 2441264 2026-07-27T17:42:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1318217]] to Monday, July 27 UTC late backport window 2441265 wikitext text/x-wiki i4xktkyrlwr0mehjkg5q5udb48suvnk 2441333 2441265 2026-07-27T21:02:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1306988]] to Tuesday, July 28 UTC afternoon backport window 2441333 wikitext text/x-wiki 819oqadu0exx0p3yvj4syig4sdbvir7 2441334 2441333 2026-07-27T21:02:42Z ScheduleDeploymentBot 37566 Add [[gerrit:1312694]] to Tuesday, July 28 UTC afternoon backport window 2441334 wikitext text/x-wiki apdo0d9rthqps0mwghrf2fd7ov5nh5o 110 6477 2441223 1887833 2026-07-27T16:11:24Z BCornwall-WMF 31521 BCornwall-WMF moved page [[History of job queue runners at WMF/Jobrunner]] to [[Obsolete:History of job queue runners at WMF/Jobrunner]] 1887833 wikitext text/x-wiki gacz1lpknvomogyabn781r6uwmzy4w0 0 6479 2441454 2426109 2026-07-28T08:31:26Z MLechvien-WMF 49197 Adding ownership information to facilitate routing of the tasks 2441454 wikitext text/x-wiki ih2mwb798vj0e8zcfbfit02kvxt7kya 498 6726 2441237 2441024 2026-07-27T16:40:04Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2441237 wikitext text/x-wiki 9iz8k3rhs39ul593e07aqn5idye0cz0 2441242 2441237 2026-07-27T16:53:14Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2441242 wikitext text/x-wiki olbzf0ebi5dy26n6hmvqlasw5yarto1 0 7919 2441068 2441062 2026-07-27T12:08:50Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 38 hosts 2441068 wikitext text/x-wiki g9jea7xbnnwgueln2m4evpuwi7n2mq0 2441069 2441068 2026-07-27T12:10:57Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 38 hosts 2441069 wikitext text/x-wiki thv45fayh27t86dec3yf7waypjfdhgd 2441070 2441069 2026-07-27T12:12:25Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1038: es1038 repool 2441070 wikitext text/x-wiki loigbnys2b9it56g9u2jwfhc7ohwdz0 2441071 2441070 2026-07-27T12:13:54Z Stashbot 7414 XioNoX: rebooting cr4-ulsfo for upgrade - T431752 2441071 wikitext text/x-wiki n39pjzgdwaxf0cb2nc5j1on62aqblfi 2441072 2441071 2026-07-27T12:17:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 2 hosts 2441072 wikitext text/x-wiki l45y4wnga9srerwketfbq9o31fhqpa3 2441073 2441072 2026-07-27T12:17:29Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 2 hosts 2441073 wikitext text/x-wiki 6q5f3qpkg5ztiv0j5arlg47e2fat27n 2441074 2441073 2026-07-27T12:20:32Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1318098|Enable CheckUser SI special page on dewiki and ukwiki (T432835 T433226)]] 2441074 wikitext text/x-wiki 3gy0wtsj3fz1ln1jwvodn2vo9vr6qav 2441075 2441074 2026-07-27T12:22:15Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1318098|Enable CheckUser SI special page on dewiki and ukwiki (T432835 T433226)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441075 wikitext text/x-wiki d3o13f9wr8b25yifon03tej8sateg50 2441076 2441075 2026-07-27T12:25:52Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2441076 wikitext text/x-wiki pyyrvdyeb4g4jl80m348we54rl1vwr6 2441077 2441076 2026-07-27T12:28:44Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2441077 wikitext text/x-wiki 432fsdglf50w0hdstbgdzwihdleplhc 2441078 2441077 2026-07-27T12:30:03Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318098|Enable CheckUser SI special page on dewiki and ukwiki (T432835 T433226)]] (duration: 09m 32s) 2441078 wikitext text/x-wiki secqgyjuoe10t68egk47nzhkavw9smo 2441079 2441078 2026-07-27T12:30:47Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2441079 wikitext text/x-wiki 060k1x3atmb2uv93gxcc0wiog02tnw6 2441080 2441079 2026-07-27T12:30:50Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.wipe-cache url-downloader.eqiad.wikimedia.org on all recursors 2441080 wikitext text/x-wiki ev9q2z99svdphahvu8q7zufp8j0phjo 2441081 2441080 2026-07-27T12:30:54Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) url-downloader.eqiad.wikimedia.org on all recursors 2441081 wikitext text/x-wiki 5ag88knprwdzub310om9zr75a9hpq5x 2441082 2441081 2026-07-27T12:43:38Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Maintenance 2441082 wikitext text/x-wiki hlwp8vzg4nsjnuayngrpeqojfi0mjye 2441083 2441082 2026-07-27T12:43:58Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance 2441083 wikitext text/x-wiki k7w4n62tqat4v2oytj950sea2onbd6e 2441084 2441083 2026-07-27T12:44:06Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1201 (T431660)', diff saved to https://phabricator.wikimedia.org/P95186 and previous config saved to /var/cache/conftool/dbconfig/20260727-124404-cwilliams.json 2441084 wikitext text/x-wiki tfnypgwst9gas0plvpayqsqg1z29mmg 2441085 2441084 2026-07-27T12:48:09Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2441085 wikitext text/x-wiki qtwjljcwvictdo1t50qglq7i4lura3l 2441086 2441085 2026-07-27T12:51:08Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1201: Maintenance 2441086 wikitext text/x-wiki hm3xpftnswdw4qvv778ndcatqhobr2u 2441088 2441086 2026-07-27T13:03:01Z Stashbot 7414 XioNoX: repool cr4-ulsfo - T431752 2441088 wikitext text/x-wiki 5fnagjs8vvf3la1ujcx8ui9bbwm0ogz 2441089 2441088 2026-07-27T13:06:43Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1315843|Enable WikibaseLexeme REST API on beta wikidata (T430943)]], [[gerrit:1315842|Remove obsolete wmgWikibaseRestApiEnabled setting (T302959 T324999 T383774)]] 2441089 wikitext text/x-wiki qkk256ncmf73p06zmfroj41obpv3e0b 2441090 2441089 2026-07-27T13:07:05Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool ulsfo [reason: router upgrade finished, T431752] 2441090 wikitext text/x-wiki 8wt2k8zhpa5mzfo70xql49lzq3hscm8 2441091 2441090 2026-07-27T13:07:08Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool ulsfo [reason: router upgrade finished, T431752] 2441091 wikitext text/x-wiki 9lkg58jv0c5ibaiigsu810htxwv287d 2441092 2441091 2026-07-27T13:08:29Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, sihe: Backport for [[gerrit:1315843|Enable WikibaseLexeme REST API on beta wikidata (T430943)]], [[gerrit:1315842|Remove obsolete wmgWikibaseRestApiEnabled setting (T302959 T324999 T383774)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441092 wikitext text/x-wiki cbzxm5xtklrhz366ynray8hnh4ix5bo 2441093 2441092 2026-07-27T13:14:22Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, sihe: Continuing with deployment 2441093 wikitext text/x-wiki 2iw0fa8g1sxx3czs24p6b1whbvt7any 2441095 2441093 2026-07-27T13:18:46Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1315843|Enable WikibaseLexeme REST API on beta wikidata (T430943)]], [[gerrit:1315842|Remove obsolete wmgWikibaseRestApiEnabled setting (T302959 T324999 T383774)]] (duration: 11m 57s) 2441095 wikitext text/x-wiki 5c0kv8zgemf77ds2ykbaweyqugtc5y5 2441097 2441095 2026-07-27T13:27:35Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window doen 2441097 wikitext text/x-wiki niy7o6gok5j1jyxxsy1rakxtbp0rq69 2441098 2441097 2026-07-27T13:42:12Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1201: Maintenance 2441098 wikitext text/x-wiki syqvt101f953ufsbeq500i7npp47l7u 2441102 2441098 2026-07-27T13:46:50Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 34 hosts 2441102 wikitext text/x-wiki 6graq5itfe9jm2vkw6it3g0ck1n7t2h 2441104 2441102 2026-07-27T13:50:12Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 34 hosts 2441104 wikitext text/x-wiki 380btp9r4nopfyjafod8qshj11wsujx 2441105 2441104 2026-07-27T13:52:24Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1020.eqiad.wmnet with OS bookworm 2441105 wikitext text/x-wiki tksnbazq2bdcndrscbzic4g4ipkjfex 2441106 2441105 2026-07-27T13:52:53Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance 2441106 wikitext text/x-wiki ldzowiw7lqbpg6mvp7zc8pucx1rriu1 2441107 2441106 2026-07-27T13:52:59Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2010.codfw.wmnet with OS bookworm 2441107 wikitext text/x-wiki 43br0yqpjxpshnvvy32jbc76cviz65x 2441108 2441107 2026-07-27T13:53:02Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2175 (T431660)', diff saved to https://phabricator.wikimedia.org/P95191 and previous config saved to /var/cache/conftool/dbconfig/20260727-135300-cwilliams.json 2441108 wikitext text/x-wiki 7e00p1i223cfwliqn8a635j5iawg1k6 2441109 2441108 2026-07-27T13:54:07Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance 2441109 wikitext text/x-wiki 95fqq5oz5dybzc8wcq3wkna81j4i6gb 2441110 2441109 2026-07-27T13:54:15Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2156 (T431660)', diff saved to https://phabricator.wikimedia.org/P95192 and previous config saved to /var/cache/conftool/dbconfig/20260727-135413-cwilliams.json 2441110 wikitext text/x-wiki gqz76rl8332cilx7rjbpt6gsi26n5un 2441112 2441110 2026-07-27T13:55:46Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1020 2441112 wikitext text/x-wiki se5dig2n8781r7mr4fzmhx4oqmoupmp 2441113 2441112 2026-07-27T13:55:48Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1020 2441113 wikitext text/x-wiki lhe1w0jdl41gwbinvjx7v2mtopwku9y 2441115 2441113 2026-07-27T13:56:07Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance 2441115 wikitext text/x-wiki spwrl5imamapxw36xotfxjpqtwhj17j 2441116 2441115 2026-07-27T13:56:15Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2155 (T431660)', diff saved to https://phabricator.wikimedia.org/P95193 and previous config saved to /var/cache/conftool/dbconfig/20260727-135613-cwilliams.json 2441116 wikitext text/x-wiki 3dzapt96yvhcs9ejmmcq3pg3jlm2x6d 2441117 2441116 2026-07-27T13:57:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2010 2441117 wikitext text/x-wiki 0wanp6mj7lgdt51yeeg8j16uxitbepm 2441118 2441117 2026-07-27T13:58:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2441118 wikitext text/x-wiki rlhhl7sf791et5wra454tflu0792tfd 2441119 2441118 2026-07-27T13:58:53Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2441119 wikitext text/x-wiki 2g3rgj8oo9a4wjpgjqtz1pm2zwk7g8y 2441121 2441119 2026-07-27T13:59:23Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2175: Maintenance 2441121 wikitext text/x-wiki hyu0yccy4znw1nvetcx9rbu6okecejj 2441122 2441121 2026-07-27T13:59:37Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance 2441122 wikitext text/x-wiki 8au1ciq86ygel0sz1s087h9gvew3qha 2441123 2441122 2026-07-27T13:59:45Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2157 (T431660)', diff saved to https://phabricator.wikimedia.org/P95194 and previous config saved to /var/cache/conftool/dbconfig/20260727-135943-cwilliams.json 2441123 wikitext text/x-wiki k2tc10ghqjd2zkptvqwps0zfso8zxeu 2441124 2441123 2026-07-27T14:00:35Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2156: Maintenance 2441124 wikitext text/x-wiki nns275y7gvfuvlwtjxgef1773f4j69m 2441125 2441124 2026-07-27T14:01:11Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1136.eqiad.wmnet 2441125 wikitext text/x-wiki dpxtkqv0qvtnspnyuwhlbp9nheoe9bw 2441126 2441125 2026-07-27T14:01:13Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1136.eqiad.wmnet 2441126 wikitext text/x-wiki qfonycb3saer2dxctld43gso98idya3 2441127 2441126 2026-07-27T14:01:15Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1136.eqiad.wmnet 2441127 wikitext text/x-wiki sxx6ohem5j8pla6jnu7813qu7rf7twh 2441128 2441127 2026-07-27T14:01:47Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 11 hosts 2441128 wikitext text/x-wiki dptlshq3cn4ghuqsbqx2cajv4dpk4sp 2441130 2441128 2026-07-27T14:02:58Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2155: Maintenance 2441130 wikitext text/x-wiki o5w6q24uf4lj3xwh59gdegmz6bt5l0z 2441131 2441130 2026-07-27T14:03:05Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 11 hosts 2441131 wikitext text/x-wiki cibde9t89l57j2cqst91cp03ywjynvc 2441132 2441131 2026-07-27T14:04:57Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2175: Maintenance 2441132 wikitext text/x-wiki kgvk9l3yoratnnwfvz9n8qh9xopivsh 2441133 2441132 2026-07-27T14:05:26Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2157: Maintenance 2441133 wikitext text/x-wiki 7mecetnh0umvz285z0to1ik76d70l7s 2441135 2441133 2026-07-27T14:08:28Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2155: Maintenance 2441135 wikitext text/x-wiki s6g6rxrqleyuprh6b6gdvf9ncil4vy4 2441137 2441135 2026-07-27T14:13:19Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2441137 wikitext text/x-wiki bza6inzazsm6j2bo9zhdpg3bsy41yde 2441138 2441137 2026-07-27T14:13:29Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2441138 wikitext text/x-wiki 7l1omz0vma5rx5lc483z27ozg3ad2ir 2441139 2441138 2026-07-27T14:14:00Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2441139 wikitext text/x-wiki 38i3oqgos4tnzo68eow85wrxrgfi3vl 2441140 2441139 2026-07-27T14:14:11Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2441140 wikitext text/x-wiki 4bv6knld2yghqnumn82f6baywo3u5gf 2441141 2441140 2026-07-27T14:14:24Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2441141 wikitext text/x-wiki 9qn50dlrhkxjz41eskt9kpmfwlezrp6 2441142 2441141 2026-07-27T14:14:41Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2441142 wikitext text/x-wiki akuy2ewyg6jirq4t1nkii3b0xkrs1j6 2441143 2441142 2026-07-27T14:14:59Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2441143 wikitext text/x-wiki 722tx7v34p0tq5lpsiknyeignu9hf7u 2441145 2441143 2026-07-27T14:15:07Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2441145 wikitext text/x-wiki 11as7hoviklewsp06tmc89eerhgnnf1 2441147 2441145 2026-07-27T14:16:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2441147 wikitext text/x-wiki h0l58jt5980lh01rn11t5d6ddfg68cs 2441148 2441147 2026-07-27T14:17:04Z Stashbot 7414 sukhe: sudo gnt-instance reboot urldownloader1005.wikimedia.org 2441148 wikitext text/x-wiki tu6enou8xchfnuymdqnbc7xmnc0tro4 2441149 2441148 2026-07-27T14:17:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1020.eqiad.wmnet with reason: host reimage 2441149 wikitext text/x-wiki gantfegfaosjwkmmk4jksao3n7ot50o 2441150 2441149 2026-07-27T14:20:01Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply 2441150 wikitext text/x-wiki ew4074hxp468v5fr4hnzyzja314ghs3 2441151 2441150 2026-07-27T14:20:38Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply 2441151 wikitext text/x-wiki rm7eg0nmm2apppqk6jswx4kckyw3egm 2441152 2441151 2026-07-27T14:23:32Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[1031,2024]*: Upgrade Cassandra to 5.0.8 (canary) - eevans@cumin1003 2441152 wikitext text/x-wiki mt8jh4jljtlb7xc5amfb6hnufhynd3o 2441153 2441152 2026-07-27T14:23:53Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1020.eqiad.wmnet with reason: host reimage 2441153 wikitext text/x-wiki ghzljmkyg8vjem7krpaff6gc2v1pbb3 2441154 2441153 2026-07-27T14:24:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2010 - bking@cumin2003" 2441154 wikitext text/x-wiki 1x90ibbtd5xj23cggo2ys1scpuyaxh0 2441155 2441154 2026-07-27T14:24:16Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2010 - bking@cumin2003" 2441155 wikitext text/x-wiki 43z9l9fowehlqct1jgzrdshh8xsvkhf 2441156 2441155 2026-07-27T14:24:18Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441156 wikitext text/x-wiki 9j9y2x2neiy81rnp6vvkymfotkh5npa 2441157 2441156 2026-07-27T14:24:19Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2010.codfw.wmnet 94.16.192.10.in-addr.arpa 4.9.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441157 wikitext text/x-wiki o7y2x9j7v4hn2q0efunk44sn8t4b62q 2441158 2441157 2026-07-27T14:24:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2010.codfw.wmnet 94.16.192.10.in-addr.arpa 4.9.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441158 wikitext text/x-wiki ad5ihlqia08m561qw5riq4gp2jalk4o 2441159 2441158 2026-07-27T14:24:22Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2010 2441159 wikitext text/x-wiki kwwypgxe1h3604ovpc8mg0ilruh7mq8 2441160 2441159 2026-07-27T14:24:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2010 2441160 wikitext text/x-wiki r89i4tms0yjeodfv6ceh46tttvo0tsj 2441161 2441160 2026-07-27T14:24:32Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2010 2441161 wikitext text/x-wiki ebe8hrzk5bu57fs2qjlyoreuqglwl1h 2441162 2441161 2026-07-27T14:33:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2175: Maintenance 2441162 wikitext text/x-wiki 9ulka94qcn280nzhlulxrakvya8sr9n 2441164 2441162 2026-07-27T14:34:29Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2155: Maintenance 2441164 wikitext text/x-wiki lwv9engzitdmbf4ypvnxbpoxokn78kz 2441166 2441164 2026-07-27T14:41:49Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[1031,2024]*: Upgrade Cassandra to 5.0.8 (canary) - eevans@cumin1003 2441166 wikitext text/x-wiki 8inluycqs4jbr20e0oduaw3gkximsyx 2441167 2441166 2026-07-27T14:42:43Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2010.codfw.wmnet with reason: host reimage 2441167 wikitext text/x-wiki hhwp2ah5tv7qkgg97qtm9zengfzqkfv 2441168 2441167 2026-07-27T14:48:11Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2010.codfw.wmnet with reason: host reimage 2441168 wikitext text/x-wiki m3k4n2gllax5kbn2vbbvnsxx29jj50t 2441169 2441168 2026-07-27T14:51:38Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2156: Maintenance 2441169 wikitext text/x-wiki 40kha8nrzctf1f9prbxns44dhcstmej 2441170 2441169 2026-07-27T14:51:50Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1020.eqiad.wmnet with OS bookworm 2441170 wikitext text/x-wiki 5ughufx3g6b1j3laq9k3h98126d9ire 2441171 2441170 2026-07-27T14:52:00Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance 2441171 wikitext text/x-wiki n43aloesbiaybjuwtq35pzz5fu6kmr5 2441172 2441171 2026-07-27T14:52:07Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2157: Maintenance 2441172 wikitext text/x-wiki seolmcn3jh7xprustaq4yb2mfgeajbj 2441173 2441172 2026-07-27T14:52:09Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2177 (T431660)', diff saved to https://phabricator.wikimedia.org/P95208 and previous config saved to /var/cache/conftool/dbconfig/20260727-145206-cwilliams.json 2441173 wikitext text/x-wiki lhleftrzb775vgjkxrmz2osopvekwpg 2441174 2441173 2026-07-27T14:52:29Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance 2441174 wikitext text/x-wiki ot0it1mijrv0j2qvyly1vjd94aocqne 2441175 2441174 2026-07-27T14:52:38Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2171 (T431660)', diff saved to https://phabricator.wikimedia.org/P95209 and previous config saved to /var/cache/conftool/dbconfig/20260727-145236-cwilliams.json 2441175 wikitext text/x-wiki imchgnk2csqiuqifo0skxm5q3g3eqpn 2441176 2441175 2026-07-27T14:58:28Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2171: Maintenance 2441176 wikitext text/x-wiki 5q9173gtc9f2v6aazi78jv7zcm0thvv 2441177 2441176 2026-07-27T14:58:32Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2177: Maintenance 2441177 wikitext text/x-wiki p4niyu21jv0glvuxglh5dz6bk0jfiu5 2441178 2441177 2026-07-27T15:04:09Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2177: Maintenance 2441178 wikitext text/x-wiki bzc5ibda5fj7fbazbweut61ofkobuou 2441179 2441178 2026-07-27T15:08:23Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2177: Maintenance 2441179 wikitext text/x-wiki 6tfug7uiw6dxwjuw70xspozweo0chct 2441180 2441179 2026-07-27T15:12:33Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2010.codfw.wmnet with OS bookworm 2441180 wikitext text/x-wiki pfhiqzc7jtu8dxoyx1dj2fok0zqalyr 2441181 2441180 2026-07-27T15:15:26Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2441181 wikitext text/x-wiki 025agh2usb3ixwl8ekr5yajvdsc0wgk 2441182 2441181 2026-07-27T15:15:33Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 06s) 2441182 wikitext text/x-wiki luwwke8snn6gfh6veq0bhhowj8guh9q 2441183 2441182 2026-07-27T15:15:37Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1318166|SpecialWantedFiles: Simplify query plan (T431518)]] 2441183 wikitext text/x-wiki brp949xrfvbulr9azgkw9i88jmsvjyz 2441184 2441183 2026-07-27T15:15:45Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2441184 wikitext text/x-wiki 801cuj70b7e1787s1kleuq20i0i6pq0 2441185 2441184 2026-07-27T15:15:54Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s) 2441185 wikitext text/x-wiki 22wmumn3fnt2n8ft5d0fc8s8zd4levt 2441186 2441185 2026-07-27T15:17:23Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1318166|SpecialWantedFiles: Simplify query plan (T431518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441186 wikitext text/x-wiki ebynhuczaj2ou894mettbgperdzf6tm 2441187 2441186 2026-07-27T15:18:15Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with deployment 2441187 wikitext text/x-wiki hsodu7j2vjxefrpn6ow5qh1g7sd4azo 2441188 2441187 2026-07-27T15:19:04Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2175: Maintenance 2441188 wikitext text/x-wiki 23rrdo80umyjgfvoe4oiz3n6192tsjl 2441189 2441188 2026-07-27T15:19:59Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2155: Maintenance 2441189 wikitext text/x-wiki 756xscicnu8106s695c3yrk70yzv5le 2441190 2441189 2026-07-27T15:21:18Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards 2441190 wikitext text/x-wiki emu9q9vh9djjjixi4jq6s7a4ehgg9ll 2441191 2441190 2026-07-27T15:22:34Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318166|SpecialWantedFiles: Simplify query plan (T431518)]] (duration: 07m 00s) 2441191 wikitext text/x-wiki 1i3g9mydieibfeme4jpltjonft9lfse 2441192 2441191 2026-07-27T15:26:08Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards 2441192 wikitext text/x-wiki 7lquf3hu0nila3siecgnsg5hmc1enx1 2441193 2441192 2026-07-27T15:29:07Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards 2441193 wikitext text/x-wiki 5i3nnyrgd6q4im7g20hbevcosbx9gx9 2441194 2441193 2026-07-27T15:31:39Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards 2441194 wikitext text/x-wiki g8zcdf0o7sm6p881yhesys9mxao3gxs 2441195 2441194 2026-07-27T15:32:08Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards 2441195 wikitext text/x-wiki owvzf4xhv4t07sul6im3m24l4koa5ig 2441196 2441195 2026-07-27T15:32:39Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards 2441196 wikitext text/x-wiki n5zx8f8gh4f1qvyqqrr4mkoguyi3o2s 2441198 2441196 2026-07-27T15:34:33Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards 2441198 wikitext text/x-wiki gdgxbv7nhwsxl6hwqkveb9x2nqdm4qa 2441199 2441198 2026-07-27T15:38:26Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance 2441199 wikitext text/x-wiki 6l5ki5h9wxqws7166cf0uzqvwwlj7ru 2441200 2441199 2026-07-27T15:38:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2189 (T431660)', diff saved to https://phabricator.wikimedia.org/P95220 and previous config saved to /var/cache/conftool/dbconfig/20260727-153833-cwilliams.json 2441200 wikitext text/x-wiki fcegh8qlsyy4qlvdqt66j2zzxkiw476 2441201 2441200 2026-07-27T15:38:38Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards 2441201 wikitext text/x-wiki 8s0czhaz9n9kxdy7w4u18o2w6oq0n1a 2441202 2441201 2026-07-27T15:39:21Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance 2441202 wikitext text/x-wiki r7h0yjcbqfe7yx2wovv043acngvyp51 2441203 2441202 2026-07-27T15:39:29Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2172 (T431660)', diff saved to https://phabricator.wikimedia.org/P95222 and previous config saved to /var/cache/conftool/dbconfig/20260727-153927-cwilliams.json 2441203 wikitext text/x-wiki bdzat0z5dkyu3oae087bsw0slduzghz 2441204 2441203 2026-07-27T15:41:15Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards 2441204 wikitext text/x-wiki ks0ng5zg74hagxxb3cxvfxha5i44t58 2441205 2441204 2026-07-27T15:43:22Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards 2441205 wikitext text/x-wiki cxbz4eo2oyrqtqiu92k7lpaur51xtig 2441206 2441205 2026-07-27T15:44:37Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2189: Maintenance 2441206 wikitext text/x-wiki 89bhp3ij1mfyenpdpd7x6ketlgs85r2 2441207 2441206 2026-07-27T15:45:32Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2171: Maintenance 2441207 wikitext text/x-wiki pi8lzuov5tgb6qketpwlfmzaq0v3pkl 2441208 2441207 2026-07-27T15:45:55Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance 2441208 wikitext text/x-wiki o46kjksse3vhedx19gp15mskfdp85zq 2441209 2441208 2026-07-27T15:46:00Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2172: Maintenance 2441209 wikitext text/x-wiki g6pfmo2w8lcpeijfc8af75rampylj0p 2441210 2441209 2026-07-27T15:46:02Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2178 (T431660)', diff saved to https://phabricator.wikimedia.org/P95224 and previous config saved to /var/cache/conftool/dbconfig/20260727-154559-cwilliams.json 2441210 wikitext text/x-wiki ed77sbvk7ngthhafye818g3p33ushgr 2441212 2441210 2026-07-27T15:51:39Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2172: Maintenance 2441212 wikitext text/x-wiki d3tv4ewz1wfgf76ekwesbfm9p08lplk 2441213 2441212 2026-07-27T15:51:47Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2178: Maintenance 2441213 wikitext text/x-wiki e2y19nahf5h1b7oe2z44a7au2m41dbm 2441215 2441213 2026-07-27T15:53:47Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2172: Maintenance 2441215 wikitext text/x-wiki nu3swknx1oswc2yvudt183g8pfu7980 2441216 2441215 2026-07-27T15:53:51Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2177: Maintenance 2441216 wikitext text/x-wiki g8lsitiqo0prs1mfz19fuczbctqq3xt 2441219 2441216 2026-07-27T16:05:56Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance 2441219 wikitext text/x-wiki jr6qckhgxu0wuh2x8ruuztd9gxyz1bb 2441220 2441219 2026-07-27T16:06:05Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2190 (T431660)', diff saved to https://phabricator.wikimedia.org/P95230 and previous config saved to /var/cache/conftool/dbconfig/20260727-160602-cwilliams.json 2441220 wikitext text/x-wiki fn813o0i4mvc5xvcoj5z2ojn99o7p4k 2441227 2441220 2026-07-27T16:13:32Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2190: Maintenance 2441227 wikitext text/x-wiki rc2blibfzry3n2mf7m8j1zyznmf38xn 2441229 2441227 2026-07-27T16:32:33Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: Maintenance 2441229 wikitext text/x-wiki eh9ozo2wsx0slzhmknqx04ziqoi5qix 2441230 2441229 2026-07-27T16:33:00Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance 2441230 wikitext text/x-wiki nlep7i0oz9oj0v8ibly1lvryytnzw8q 2441231 2441230 2026-07-27T16:33:09Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2225 (T431660)', diff saved to https://phabricator.wikimedia.org/P95238 and previous config saved to /var/cache/conftool/dbconfig/20260727-163307-cwilliams.json 2441231 wikitext text/x-wiki rttjqi55lml6rre27f8x07slwzst35b 2441232 2441231 2026-07-27T16:38:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2441232 wikitext text/x-wiki 0fp5ze2x1wxht786h0xhzt3mwp963ra 2441233 2441232 2026-07-27T16:38:48Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2441233 wikitext text/x-wiki ndks8o6l0tnm1d69rv86gt9m1zh6hw0 2441234 2441233 2026-07-27T16:39:21Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2172: Maintenance 2441234 wikitext text/x-wiki 5uq5vdo1yams3hphkanbjfwiz17m9fb 2441235 2441234 2026-07-27T16:39:24Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2225: Maintenance 2441235 wikitext text/x-wiki m7nuvhpnrh1bgnih8tbol0u7ltcuvx3 2441236 2441235 2026-07-27T16:39:45Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2178: Maintenance 2441236 wikitext text/x-wiki 0v50f9k0e8ey5fspdbirvyddr1swb5d 2441238 2441236 2026-07-27T16:40:09Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance 2441238 wikitext text/x-wiki 31tusvkengpgh48u82ldn1zi757wjwl 2441239 2441238 2026-07-27T16:40:17Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2211 (T431660)', diff saved to https://phabricator.wikimedia.org/P95242 and previous config saved to /var/cache/conftool/dbconfig/20260727-164015-cwilliams.json 2441239 wikitext text/x-wiki agd9mt3c5gf084tv9bou9t85s8pil9m 2441240 2441239 2026-07-27T16:46:08Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2211: Maintenance 2441240 wikitext text/x-wiki oz7rai9neuye6gv0u9emtp1z65n8h6p 2441241 2441240 2026-07-27T16:52:17Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards 2441241 wikitext text/x-wiki kw21lpxii91dpum0vsnpid1q8m9xees 2441244 2441241 2026-07-27T17:04:36Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2190: Maintenance 2441244 wikitext text/x-wiki ginx6b15717pq6pshq6o8njq9kyf0lx 2441245 2441244 2026-07-27T17:04:46Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance 2441245 wikitext text/x-wiki 5qks73pcw6vq8fgllc65gg4o2312ebk 2441246 2441245 2026-07-27T17:04:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2194 (T431660)', diff saved to https://phabricator.wikimedia.org/P95248 and previous config saved to /var/cache/conftool/dbconfig/20260727-170453-cwilliams.json 2441246 wikitext text/x-wiki kynlyb3mv1phqpf3svijq2fgnrs15l4 2441247 2441246 2026-07-27T17:11:25Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2194: Maintenance 2441247 wikitext text/x-wiki 87283hksfl5rrv5aao3mgc0pkx7l2uw 2441249 2441247 2026-07-27T17:13:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards 2441249 wikitext text/x-wiki jfik93t258ulsoa6oygqvyrnioytzub 2441251 2441249 2026-07-27T17:25:24Z Stashbot 7414 taavi@deploy1003: Started scap sync-world: Backport for [[gerrit:1318202|Undeploy WP25EasterEggs (I) (T418134)]] 2441251 wikitext text/x-wiki ax4bzyxuyglhuaoltr2gspikiwvm0va 2441252 2441251 2026-07-27T17:26:07Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2225: Maintenance 2441252 wikitext text/x-wiki 9rzarwojnrh67tqhfyazagf33mkm3c1 2441253 2441252 2026-07-27T17:26:29Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance 2441253 wikitext text/x-wiki cx18k3sc9xtfitbbsehuhzabpbzez8l 2441254 2441253 2026-07-27T17:26:37Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2226 (T431660)', diff saved to https://phabricator.wikimedia.org/P95256 and previous config saved to /var/cache/conftool/dbconfig/20260727-172636-cwilliams.json 2441254 wikitext text/x-wiki eii9bjdcqj0wr0ar86zn7j8ccs8xzvk 2441255 2441254 2026-07-27T17:27:13Z Stashbot 7414 taavi@deploy1003: taavi: Backport for [[gerrit:1318202|Undeploy WP25EasterEggs (I) (T418134)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441255 wikitext text/x-wiki icwb1n87cu6g7b8q5r3g62cbklhtlmc 2441256 2441255 2026-07-27T17:27:45Z Stashbot 7414 taavi@deploy1003: taavi: Continuing with deployment 2441256 wikitext text/x-wiki 019vb4s85ss92hrwl7juw2pafuwyseu 2441257 2441256 2026-07-27T17:31:57Z Stashbot 7414 taavi@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318202|Undeploy WP25EasterEggs (I) (T418134)]] (duration: 06m 33s) 2441257 wikitext text/x-wiki 10xscul9ariyo15l622zzukuynndvta 2441258 2441257 2026-07-27T17:32:35Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2226: Maintenance 2441258 wikitext text/x-wiki 9r04oi9rphfkg9ao7l1p3bk1wk63j2e 2441259 2441258 2026-07-27T17:36:55Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs1020\.eqiad\.wmnet,dc=eqiad,cluster=wdqs\-main,service=wdqs\-main 2441259 wikitext text/x-wiki 8fgqqtxr92ge6zjthpaa79x5vysrafr 2441260 2441259 2026-07-27T17:37:12Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2211: Maintenance 2441260 wikitext text/x-wiki pf21xultff6hkhnoy5knz20j3u2qu6g 2441261 2441260 2026-07-27T17:37:34Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance 2441261 wikitext text/x-wiki ote81wgjdebu522h8za03f4diwx9bpz 2441262 2441261 2026-07-27T17:37:42Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2213 (T431660)', diff saved to https://phabricator.wikimedia.org/P95260 and previous config saved to /var/cache/conftool/dbconfig/20260727-173740-cwilliams.json 2441262 wikitext text/x-wiki 4i76zb1xj6u7rp2wrxh4t642mq0htuh 2441263 2441262 2026-07-27T17:38:43Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2010\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main 2441263 wikitext text/x-wiki mxny0vj35nj1s8tywfv1kr589x7vgpf 2441266 2441263 2026-07-27T17:43:30Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Maintenance 2441266 wikitext text/x-wiki tn2h3ltygha5m879fo98y412u7ejkvj 2441267 2441266 2026-07-27T17:46:45Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1021.eqiad.wmnet with OS bookworm 2441267 wikitext text/x-wiki 5hah75rpg56w84aahdef5rvsz36zdh3 2441268 2441267 2026-07-27T17:48:07Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2011.codfw.wmnet with OS bookworm 2441268 wikitext text/x-wiki 876eixolh5ndlf5fb5gthh4dtmoiu7h 2441269 2441268 2026-07-27T17:51:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1021 2441269 wikitext text/x-wiki 6630e640yqemnb3601qirvik9mcdcyb 2441270 2441269 2026-07-27T17:51:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1021 2441270 wikitext text/x-wiki mudeohpq478umcg8p41vg2aw9b347yv 2441271 2441270 2026-07-27T17:51:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2011 2441271 wikitext text/x-wiki 3uotimjsakxpv5lqfx8mf0yqxdv981k 2441272 2441271 2026-07-27T17:52:22Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2441272 wikitext text/x-wiki a5231zdmm51rtd3wek5zo8bv5536dak 2441273 2441272 2026-07-27T17:56:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2011 - bking@cumin2003" 2441273 wikitext text/x-wiki q0mkrtnbjj4ymlshaq24h50q7q5tras 2441274 2441273 2026-07-27T17:56:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2011 - bking@cumin2003" 2441274 wikitext text/x-wiki gfm8lycyhdjo3mkwk75qltqdxpk2jag 2441275 2441274 2026-07-27T17:56:43Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441275 wikitext text/x-wiki kw2jbgsb856p1ok77x7hy3t1uhf7mtz 2441276 2441275 2026-07-27T17:56:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2011.codfw.wmnet 37.32.192.10.in-addr.arpa 7.3.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441276 wikitext text/x-wiki llqfigps43vpacix3rghwxzixym98nf 2441277 2441276 2026-07-27T17:56:46Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2011.codfw.wmnet 37.32.192.10.in-addr.arpa 7.3.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441277 wikitext text/x-wiki 8a3815w41kkat5gl0lsnt0ad9ijj3f9 2441278 2441277 2026-07-27T17:56:48Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2011 2441278 wikitext text/x-wiki c3axfu9e0wcl1nkmmiuc1ieie43oolr 2441279 2441278 2026-07-27T17:56:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2011 2441279 wikitext text/x-wiki kofvahsf77kbs56t69e8zr9f41yrg3k 2441280 2441279 2026-07-27T17:57:01Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2011 2441280 wikitext text/x-wiki 7ry8tk7uajfk48fhvc4j02uf83atu34 2441281 2441280 2026-07-27T18:02:26Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2194: Maintenance 2441281 wikitext text/x-wiki 58rvg0z40jusjhw3sn4hopwua596dff 2441282 2441281 2026-07-27T18:02:50Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance 2441282 wikitext text/x-wiki lssvl92zxyuoqz0mezicklrx87yva6u 2441283 2441282 2026-07-27T18:02:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2227 (T431660)', diff saved to https://phabricator.wikimedia.org/P95265 and previous config saved to /var/cache/conftool/dbconfig/20260727-180256-cwilliams.json 2441283 wikitext text/x-wiki dqgtmn772batozuq1w7hdr66npyym1r 2441284 2441283 2026-07-27T18:09:27Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2227: Maintenance 2441284 wikitext text/x-wiki qhi4q2pwz7vrgn70c42zlv1kp4wjqw0 2441285 2441284 2026-07-27T18:12:34Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1021.eqiad.wmnet with reason: host reimage 2441285 wikitext text/x-wiki qflhdks1wibca2rqr853fvhm3v99snf 2441286 2441285 2026-07-27T18:14:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2011.codfw.wmnet with reason: host reimage 2441286 wikitext text/x-wiki mr7h25aq5nx9xb231k2dlmnh3z0fnh3 2441287 2441286 2026-07-27T18:17:54Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1021.eqiad.wmnet with reason: host reimage 2441287 wikitext text/x-wiki rb5b3v9t45zsjsgllj0i0aa1i0xf0k3 2441288 2441287 2026-07-27T18:19:16Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2226: Maintenance 2441288 wikitext text/x-wiki lkm7u6m48uynzgje1hkx7fe7ix8refj 2441289 2441288 2026-07-27T18:19:38Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance 2441289 wikitext text/x-wiki nns9aoz3j4xe9i66dsxyanaxwadcjwh 2441290 2441289 2026-07-27T18:19:47Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2238 (T431660)', diff saved to https://phabricator.wikimedia.org/P95271 and previous config saved to /var/cache/conftool/dbconfig/20260727-181944-cwilliams.json 2441290 wikitext text/x-wiki poyz3myr6t0z6n1h2gg02mk7t9ckzn6 2441291 2441290 2026-07-27T18:21:51Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2011.codfw.wmnet with reason: host reimage 2441291 wikitext text/x-wiki piuc33llra2fshzl1dk1b6bkmnzgpa9 2441292 2441291 2026-07-27T18:26:01Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2238: Maintenance 2441292 wikitext text/x-wiki 0o7vy9h3ixj2m1j7x30c9u03by5hphd 2441293 2441292 2026-07-27T18:32:02Z Stashbot 7414 musikanimal@deploy1003: Started scap sync-world: Backport for [[gerrit:1318211|CodeMirrorWikiEditor: don't autofocus from live preview when RTP is open]] 2441293 wikitext text/x-wiki pbg29fem2xkxa6sfpalz6lufqscgnwq 2441294 2441293 2026-07-27T18:33:45Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Backport for [[gerrit:1318211|CodeMirrorWikiEditor: don't autofocus from live preview when RTP is open]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441294 wikitext text/x-wiki 2uwslj6twkhaknf0xqemb3ia9yqdjtd 2441295 2441294 2026-07-27T18:34:32Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2213: Maintenance 2441295 wikitext text/x-wiki nabeba5al1j0q2e9qngpc60md9cbt9s 2441296 2441295 2026-07-27T18:34:52Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Continuing with deployment 2441296 wikitext text/x-wiki dd3nzlel34vxmlaqa4xju6hlg2jgqlp 2441297 2441296 2026-07-27T18:34:54Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance 2441297 wikitext text/x-wiki fg8m0wn54u1fi2aksnpb5zn2kapl58i 2441298 2441297 2026-07-27T18:35:02Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2223 (T431660)', diff saved to https://phabricator.wikimedia.org/P95275 and previous config saved to /var/cache/conftool/dbconfig/20260727-183500-cwilliams.json 2441298 wikitext text/x-wiki db5itd4jlgozdtionggwettii4l871f 2441299 2441298 2026-07-27T18:39:07Z Stashbot 7414 musikanimal@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318211|CodeMirrorWikiEditor: don't autofocus from live preview when RTP is open]] (duration: 07m 05s) 2441299 wikitext text/x-wiki nfi25rx0h063vtvax5it3958pmneczm 2441300 2441299 2026-07-27T18:40:49Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2223: Maintenance 2441300 wikitext text/x-wiki mwiapnp3zzal23xuy6ixe0949y7r3pw 2441301 2441300 2026-07-27T18:43:00Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2011.codfw.wmnet with OS bookworm 2441301 wikitext text/x-wiki sptbchichofcrm47vaa8crc2i1lht17 2441302 2441301 2026-07-27T18:44:55Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1021.eqiad.wmnet with OS bookworm 2441302 wikitext text/x-wiki 15f8pv17qnfx0u2s0p6r84onvx4rsr2 2441306 2441302 2026-07-27T18:55:57Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply 2441306 wikitext text/x-wiki soygfhsbb5peaxlzdnjd6o1z3uel1xo 2441307 2441306 2026-07-27T18:57:19Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply 2441307 wikitext text/x-wiki n013et0zgbxmv67ki5t0pgpsskvsv89 2441308 2441307 2026-07-27T18:57:24Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2227: Maintenance 2441308 wikitext text/x-wiki b50aaenufodjha3jvzujn3gz4mmo0fd 2441309 2441308 2026-07-27T18:57:27Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply 2441309 wikitext text/x-wiki e9vtqajzhvt3896r6jje93gfuvx8mgh 2441310 2441309 2026-07-27T18:58:06Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply 2441310 wikitext text/x-wiki jeytslogy4cn4g9vmtx4ik9p1nggtkp 2441311 2441310 2026-07-27T19:12:49Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2238: Maintenance 2441311 wikitext text/x-wiki 4j1az7cajqssp1sacspxktyoegwtgte 2441312 2441311 2026-07-27T19:15:05Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1316171|logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile]], [[gerrit:1316172|logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices]] 2441312 wikitext text/x-wiki 1ydfztpsedw2ody33dx07lh8s3825ez 2441313 2441312 2026-07-27T19:16:49Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1316171|logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile]], [[gerrit:1316172|logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441313 wikitext text/x-wiki 05rx7h0nlcbutytqvyg6btq59i2qdvy 2441314 2441313 2026-07-27T19:23:20Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2441314 wikitext text/x-wiki m0oglst8kzhsmzc829lmuzou3myo5f1 2441315 2441314 2026-07-27T19:26:43Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: Maintenance 2441315 wikitext text/x-wiki c960xo4wv24hzsjwd2dc8q6666kg87u 2441316 2441315 2026-07-27T19:27:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance 2441316 wikitext text/x-wiki ran7wq73inmb56dj89yj3yn0ciowf8x 2441317 2441316 2026-07-27T19:27:14Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2228 (T431660)', diff saved to https://phabricator.wikimedia.org/P95285 and previous config saved to /var/cache/conftool/dbconfig/20260727-192711-cwilliams.json 2441317 wikitext text/x-wiki a7mt73kkzkcdwdmured8yshnxc5veed 2441318 2441317 2026-07-27T19:27:31Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1316171|logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile]], [[gerrit:1316172|logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices]] (duration: 12m 26s) 2441318 wikitext text/x-wiki rjbdn6ctu6s1iirnq2bnakfhn7p9i1j 2441319 2441318 2026-07-27T19:33:04Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2228: Maintenance 2441319 wikitext text/x-wiki ap58wgoko8k5wevurbxmldllgkefqf7 2441320 2441319 2026-07-27T19:47:44Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2441320 wikitext text/x-wiki 0xl5ekjjxhobl653f1tt4tauij8qbjp 2441321 2441320 2026-07-27T19:47:52Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 46s) 2441321 wikitext text/x-wiki mv78iwxwpfl98mgyuftxz93jq9usin1 2441323 2441321 2026-07-27T20:07:22Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1318149|Article Guidance: clean up old wikidata config (T421250)]], [[gerrit:1318217|ArticleGuidance: Enable wgArticleGuidanceWikidataConnectEnabled in prod (T421250)]] 2441323 wikitext text/x-wiki d94k4uesoodzudntlmbyucrfncr9f52 2441324 2441323 2026-07-27T20:09:07Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1318149|Article Guidance: clean up old wikidata config (T421250)]], [[gerrit:1318217|ArticleGuidance: Enable wgArticleGuidanceWikidataConnectEnabled in prod (T421250)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441324 wikitext text/x-wiki 7eh5eiwycpovc4w5t61gkd7efwt29u8 2441325 2441324 2026-07-27T20:11:14Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2441325 wikitext text/x-wiki imlajz1est570oa0d2a2wwwidrks8sj 2441326 2441325 2026-07-27T20:15:27Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318149|Article Guidance: clean up old wikidata config (T421250)]], [[gerrit:1318217|ArticleGuidance: Enable wgArticleGuidanceWikidataConnectEnabled in prod (T421250)]] (duration: 08m 03s) 2441326 wikitext text/x-wiki ibksagwk14599gbtmmfuc1yn8udi36w 2441327 2441326 2026-07-27T20:18:50Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: Maintenance 2441327 wikitext text/x-wiki be13hzdc8w95dpcezk7o0ognxsnydu2 2441329 2441327 2026-07-27T20:51:27Z Stashbot 7414 mutante: zuul1001 - re-enabled puppet - revert "cherry-picked" gerrit:1314120 - T431003 2441329 wikitext text/x-wiki myg8kk6qhwvd6r9cawik15opu8b43l1 2441330 2441329 2026-07-27T20:53:55Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1021.eqiad.wmnet, repooling source-only afterwards 2441330 wikitext text/x-wiki slx107ioxh4bdcrdflmdcn0l8brzghi 2441331 2441330 2026-07-27T20:58:11Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1022.eqiad.wmnet with OS bookworm 2441331 wikitext text/x-wiki mwvcilgt41w5qmm1vfh50m38n21d8g3 2441332 2441331 2026-07-27T21:01:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1022 2441332 wikitext text/x-wiki dnxy0mlz74rp15b8uh5ddh338fgtxcd 2441335 2441332 2026-07-27T21:05:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2441335 wikitext text/x-wiki jfoi9q2ry5syxygb5l0ollyi12oukm4 2441336 2441335 2026-07-27T21:10:37Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling source-only afterwards 2441336 wikitext text/x-wiki plnmozbkra3ab08bn36wcj19bmnzcr1 2441337 2441336 2026-07-27T21:11:15Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2441337 wikitext text/x-wiki a4q5vi34z17sz6sooozom6vkmm0sml9 2441338 2441337 2026-07-27T21:15:45Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1004] - vriley@cumin1003" 2441338 wikitext text/x-wiki l3z9311hjfwjhl7yj64g92r8vunpwwv 2441339 2441338 2026-07-27T21:15:49Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1004] - vriley@cumin1003" 2441339 wikitext text/x-wiki s31zn5161gvfqmj6eukdmgaq4qe3zab 2441340 2441339 2026-07-27T21:15:51Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441340 wikitext text/x-wiki cimxpo2avauy7ykfz87c1mua91ukun6 2441341 2441340 2026-07-27T21:16:52Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1004 2441341 wikitext text/x-wiki l40ropp1y4f1oxso0t313oqfzstwrr8 2441342 2441341 2026-07-27T21:17:52Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1004 2441342 wikitext text/x-wiki 4v33dm62fvqloxagosjfeu0o6iyfnga 2441343 2441342 2026-07-27T21:19:34Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441343 wikitext text/x-wiki 2gq3ijmpmcufeu6ihjom06fvh3ajgaf 2441344 2441343 2026-07-27T21:21:35Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441344 wikitext text/x-wiki d9uw03digxjvkps2hom3redhjy3z2gl 2441345 2441344 2026-07-27T21:22:07Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441345 wikitext text/x-wiki sl5lbcdxfo4gjhh6nxtlvy0ag1qpeap 2441346 2441345 2026-07-27T21:31:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441346 wikitext text/x-wiki 8vzygf0afgolvflgxaaqtekbtebzllg 2441347 2441346 2026-07-27T21:34:42Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS bookworm 2441347 wikitext text/x-wiki 2hqbu7yl2b1jqlm634yb3iiirjhbanp 2441349 2441347 2026-07-27T21:41:07Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2441349 wikitext text/x-wiki pvsmq6xvhk3n86d4j04xyumtrds6rmq 2441350 2441349 2026-07-27T21:41:12Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2441350 wikitext text/x-wiki apuah52ao38usoq5df44jq1noerfab1 2441351 2441350 2026-07-27T21:44:24Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441351 wikitext text/x-wiki 3t8kvbj0laia5tzes4ytmsity74q0ft 2441352 2441351 2026-07-27T21:44:26Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1022.eqiad.wmnet 239.48.64.10.in-addr.arpa 9.3.2.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441352 wikitext text/x-wiki 7a0dsu186kfkig7x9k92uveqkv5h2aw 2441353 2441352 2026-07-27T21:44:28Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1022.eqiad.wmnet 239.48.64.10.in-addr.arpa 9.3.2.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441353 wikitext text/x-wiki jk3uwurjpefcvuqn8hbal43tsreogyh 2441354 2441353 2026-07-27T21:44:30Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1022 2441354 wikitext text/x-wiki 9gzwn71695yenqhm6lgn4wsohkh0lip 2441355 2441354 2026-07-27T21:45:43Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1022 2441355 wikitext text/x-wiki a4fckkl498z4wxfuyf4jh19r9lhk3h3 2441356 2441355 2026-07-27T21:45:45Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1022 2441356 wikitext text/x-wiki lth1eojwpm4n8nwu1hha6unvj5lzkjo 2441358 2441356 2026-07-27T21:55:11Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2011\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main 2441358 wikitext text/x-wiki lvac4oxx4xcs5ie123jt2ho9j1xpbfl 2441360 2441358 2026-07-27T21:57:05Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2012.codfw.wmnet with OS bookworm 2441360 wikitext text/x-wiki brzd1unaze68xoscumcudwwr2i9levg 2441361 2441360 2026-07-27T22:00:35Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2012 2441361 wikitext text/x-wiki kbkwznw4e031ard3vjot81wyf2pct0u 2441362 2441361 2026-07-27T22:01:47Z Stashbot 7414 RScout-WMF: Deployed security fix for T431819 2441362 wikitext text/x-wiki dokm7rva06zsr0mmcj80f12lz7eerlo 2441363 2441362 2026-07-27T22:07:29Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1022.eqiad.wmnet with reason: host reimage 2441363 wikitext text/x-wiki h3j5g621e1sabgk6ib1bn1ckt086eww 2441364 2441363 2026-07-27T22:14:15Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1022.eqiad.wmnet with reason: host reimage 2441364 wikitext text/x-wiki rsvdgzkwba4u2l7zdtzus01ypn6sn00 2441365 2441364 2026-07-27T22:22:18Z Stashbot 7414 sbassett: Deployed security patch for T431819 2441365 wikitext text/x-wiki npvscpjsd2dj9o202emtaa54lvsapvy 2441366 2441365 2026-07-27T22:32:48Z Stashbot 7414 sbassett: Deployed security fix for T432789 2441366 wikitext text/x-wiki 368nv3f6107uh0es4bkx7w8r04c0bex 2441368 2441366 2026-07-27T22:37:25Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS bullseye 2441368 wikitext text/x-wiki 1f57oz8zmz1c0p3i06shoy6w7n045lj 2441369 2441368 2026-07-27T22:38:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1022.eqiad.wmnet with OS bookworm 2441369 wikitext text/x-wiki 91io7pdx6tmy34c6wl19jtid64adqpp 2441370 2441369 2026-07-27T22:39:47Z Stashbot 7414 maryum: Deploy security fix for T432877 2441370 wikitext text/x-wiki ifveijlx3fxqtjwpmolbh5wsvztee1g 2441371 2441370 2026-07-27T23:25:51Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2441371 wikitext text/x-wiki kjg42bf2mc1hsu059lohxhjbuahl7o0 2441372 2441371 2026-07-27T23:25:58Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2441372 wikitext text/x-wiki b4tulgtrle0q62gu0vriyd7t4gcgc34 2441373 2441372 2026-07-27T23:26:47Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2441373 wikitext text/x-wiki 24yk9e74uwldunfsvsx4dz4mp6l5mge 2441374 2441373 2026-07-27T23:27:07Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS bullseye 2441374 wikitext text/x-wiki hsemiwa38xlqf9wdwibdsrqzoxiz62v 2441375 2441374 2026-07-27T23:28:03Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2441375 wikitext text/x-wiki f8dg7h6zhb38anfal9s0210j2jfcvaf 2441376 2441375 2026-07-27T23:50:08Z Stashbot 7414 Amir1: mass deleting vp8 transcodes 2441376 wikitext text/x-wiki 085ft0f5wlb1vhwir8v8prpicp9mxm2 2441378 2441376 2026-07-28T00:24:27Z Stashbot 7414 dzahn@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie 2441378 wikitext text/x-wiki 5zq5tg90cjbqr1ikdwx504tuzp56ux9 2441379 2441378 2026-07-28T00:26:27Z Stashbot 7414 mutante: attempting reimage with trixie on zuul1004 re-purposed physical hardware - dcops reported install issue - host was in busybox shell (T427353) 2441379 wikitext text/x-wiki r60fnkyb64obsfzzxfezhfyg6bi1a6y 2441380 2441379 2026-07-28T00:47:26Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2441380 wikitext text/x-wiki oykahbyt143zgw53ob6dzzsxmb61i52 2441381 2441380 2026-07-28T00:47:46Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2441381 wikitext text/x-wiki 0a9wexw731gbv755oinylvi0zwvjw5r 2441382 2441381 2026-07-28T00:47:58Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2441382 wikitext text/x-wiki bqsld8saa88slabn8g9khfh536lvmpk 2441383 2441382 2026-07-28T00:48:15Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2441383 wikitext text/x-wiki cd5jufu1fpitxpj8cvs0kz7b8nqe2dz 2441384 2441383 2026-07-28T01:08:07Z Stashbot 7414 pt1979@cumin2003: START - Cookbook sre.dns.netbox 2441384 wikitext text/x-wiki ieaq7n59k1lme7xmi4wlduyja2uer9l 2441385 2441384 2026-07-28T01:12:44Z Stashbot 7414 pt1979@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt for new switches in eqsin - pt1979@cumin2003" 2441385 wikitext text/x-wiki aq28vcf3aassc2dpwiusg8fgq7ifd4s 2441386 2441385 2026-07-28T01:12:49Z Stashbot 7414 pt1979@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt for new switches in eqsin - pt1979@cumin2003" 2441386 wikitext text/x-wiki g78ewm0m26ft6xrfzfnla1dm9ev3f44 2441387 2441386 2026-07-28T01:12:50Z Stashbot 7414 pt1979@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441387 wikitext text/x-wiki t4d4pvmeur34g4blmrtk4y614w0urni 2441388 2441387 2026-07-28T01:24:21Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.network.tls for network device asw1-603-eqsin 2441388 wikitext text/x-wiki bpckigkto9lhw8v9o9jod9djc7hsy8j 2441389 2441388 2026-07-28T01:25:27Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-603-eqsin 2441389 wikitext text/x-wiki rg6q97v9gk4j2fncx3voz5uj9focy0h 2441390 2441389 2026-07-28T01:43:29Z Stashbot 7414 dzahn@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie 2441390 wikitext text/x-wiki 78onbg9cbd3fdtqhlfa96ypobneukej 2441391 2441390 2026-07-28T01:43:46Z Stashbot 7414 dzahn@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie 2441391 wikitext text/x-wiki h7cp1pmaxnfx71h1bxyjblv1gsf98dl 2441395 2441391 2026-07-28T02:15:51Z Stashbot 7414 dzahn@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host zuul1004.eqiad.wmnet with OS trixie 2441395 wikitext text/x-wiki 1cx1l0s09lqvirxjz9nzdz61f9j4err 2441396 2441395 2026-07-28T02:16:06Z Stashbot 7414 dzahn@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie 2441396 wikitext text/x-wiki 4kqleh8nwcj84l3xb2bhktblnj8vgzk 2441397 2441396 2026-07-28T02:31:17Z Stashbot 7414 dzahn@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage 2441397 wikitext text/x-wiki 2kjpjir1sspmvol12dyymvzqy1hhwp7 2441398 2441397 2026-07-28T02:37:58Z Stashbot 7414 dzahn@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage 2441398 wikitext text/x-wiki kn9nxb1cjthx7gld6xwwaj4kfyo4478 2441400 2441398 2026-07-28T02:55:40Z Stashbot 7414 dzahn@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dzahn@cumin1003" 2441400 wikitext text/x-wiki 48852efnxwz5plu9008yadrwg254vee 2441401 2441400 2026-07-28T02:57:45Z Stashbot 7414 dzahn@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dzahn@cumin1003" 2441401 wikitext text/x-wiki oqpj7ztoq18oto45o702elgv09jbwth 2441402 2441401 2026-07-28T02:57:47Z Stashbot 7414 dzahn@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie 2441402 wikitext text/x-wiki g81soyjuuak06inah6dgh7e2d4ewivb 2441403 2441402 2026-07-28T03:03:24Z Stashbot 7414 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.13 refs T430832 2441403 wikitext text/x-wiki e7e6jxgpv5r2zqgpisghvvpwpktefbf 2441405 2441403 2026-07-28T03:39:27Z Stashbot 7414 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.13 refs T430832 (duration: 36m 06s) 2441405 wikitext text/x-wiki kqlabxisogxktyflb8ydg05poigpd9e 2441406 2441405 2026-07-28T04:02:41Z Stashbot 7414 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.10 (duration: 02m 34s) 2441406 wikitext text/x-wiki jayel1sw5bpe55gkzg8n5js3dyj67jk 2441410 2441406 2026-07-28T06:39:32Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2235].codfw.wmnet,db[1164,1217,1228].eqiad.wmnet with reason: m5 master switch T432967 2441410 wikitext text/x-wiki d11uhl02bmw45ur8q273i206nla6eog 2441411 2441410 2026-07-28T06:44:04Z Stashbot 7414 marostegui: Failover m5 from db1164 to db1228 - T432967 2441411 wikitext text/x-wiki kme81u8c5xdp45rizfk1zbarzjkgwkl 2441414 2441411 2026-07-28T07:02:13Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance 2441414 wikitext text/x-wiki 6qpzfecko7it62cjmbg8nhwvl26ud78 2441415 2441414 2026-07-28T07:02:21Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2206 (T431660)', diff saved to https://phabricator.wikimedia.org/P95292 and previous config saved to /var/cache/conftool/dbconfig/20260728-070219-cwilliams.json 2441415 wikitext text/x-wiki bw91l70sxz9x2ea5mtum8vyxr2lm504 2441417 2441415 2026-07-28T07:08:49Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2206: Maintenance 2441417 wikitext text/x-wiki h31lw9g6u0uy1zuhtds316gm53mvxfq 2441419 2441417 2026-07-28T07:15:58Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance 2441419 wikitext text/x-wiki gilplw864weadth8wffk9bq6ue1q1ki 2441420 2441419 2026-07-28T07:16:06Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2168 (T431660)', diff saved to https://phabricator.wikimedia.org/P95294 and previous config saved to /var/cache/conftool/dbconfig/20260728-071604-cwilliams.json 2441420 wikitext text/x-wiki mtdb1pwt9d4e20i1t6b7tl5d816fjbz 2441422 2441420 2026-07-28T07:16:34Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance 2441422 wikitext text/x-wiki l8ek56utjhv6n5zt23n3qt637vreehd 2441423 2441422 2026-07-28T07:16:42Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2154 (T431660)', diff saved to https://phabricator.wikimedia.org/P95295 and previous config saved to /var/cache/conftool/dbconfig/20260728-071640-cwilliams.json 2441423 wikitext text/x-wiki qqs23ro3te8fu1goo5lo0sef491sk6y 2441429 2441423 2026-07-28T07:22:12Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2168: Maintenance 2441429 wikitext text/x-wiki 6pofhbolj1r4v5iypmbmeanhyd810yh 2441430 2441429 2026-07-28T07:22:47Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2154: Maintenance 2441430 wikitext text/x-wiki sijg51l13jnm5kqqci9xr8uxwn87nh9 2441431 2441430 2026-07-28T07:50:10Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2441431 wikitext text/x-wiki 8t5jxsn2uc57dmqp80drhn8tdr2u45g 2441432 2441431 2026-07-28T07:50:26Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2441432 wikitext text/x-wiki i03sbph5lp2i5uasppwok8fo0yyea43 2441434 2441432 2026-07-28T07:59:52Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2206: Maintenance 2441434 wikitext text/x-wiki r4m641gjmcvci58hbcxrlr1ds892uc7 2441435 2441434 2026-07-28T08:00:02Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance 2441435 wikitext text/x-wiki mkdxosauxz84io5agh0f2xzqmkzp8ka 2441436 2441435 2026-07-28T08:00:10Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2210 (T431660)', diff saved to https://phabricator.wikimedia.org/P95305 and previous config saved to /var/cache/conftool/dbconfig/20260728-080008-cwilliams.json 2441436 wikitext text/x-wiki ogpk4fvrcf2dspt3fjg5zo0r1axy6rh 2441437 2441436 2026-07-28T08:05:36Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: router upgrade, T431749] 2441437 wikitext text/x-wiki m1t1afjvsznihmnksmfe9kdbg6qn18w 2441438 2441437 2026-07-28T08:05:41Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: router upgrade, T431749] 2441438 wikitext text/x-wiki im4dzvn6qt13x7kjiqm1tt3kberd30g 2441439 2441438 2026-07-28T08:06:27Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2210: Maintenance 2441439 wikitext text/x-wiki rw5vyamultcw274fbl6oep7fy55qsay 2441440 2441439 2026-07-28T08:06:48Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-drmrs,cr1-drmrs IPv6,cr1-drmrs.mgmt with reason: router upgrade 2441440 wikitext text/x-wiki fla100gizf9pb63xyfyquoiiovsdykk 2441441 2441440 2026-07-28T08:09:19Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Maintenance 2441441 wikitext text/x-wiki riytbc7uaewdlzcgtygjtdj7tbwasp6 2441442 2441441 2026-07-28T08:09:40Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance 2441442 wikitext text/x-wiki j801kuy5kf9b0qr8vo6gyd427rrc8d8 2441443 2441442 2026-07-28T08:09:49Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2182 (T431660)', diff saved to https://phabricator.wikimedia.org/P95307 and previous config saved to /var/cache/conftool/dbconfig/20260728-080947-cwilliams.json 2441443 wikitext text/x-wiki t3lnn1wj4im1om2bncmdknw2crt3jzg 2441444 2441443 2026-07-28T08:10:15Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Maintenance 2441444 wikitext text/x-wiki ev4jyugz623lo9s0fnon9mrlsh698hp 2441445 2441444 2026-07-28T08:10:37Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance 2441445 wikitext text/x-wiki 20bb2kp8a0qqo1m14i6eyeslk33boi2 2441446 2441445 2026-07-28T08:10:46Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2161 (T431660)', diff saved to https://phabricator.wikimedia.org/P95309 and previous config saved to /var/cache/conftool/dbconfig/20260728-081044-cwilliams.json 2441446 wikitext text/x-wiki 6tpkml3yj2d17pgbnh0gcbgrpeb2u2k 2441447 2441446 2026-07-28T08:12:03Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2210: Maintenance 2441447 wikitext text/x-wiki armp1q48hwa1ml15id4rp5t475pmlc3 2441448 2441447 2026-07-28T08:16:07Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2182: Maintenance 2441448 wikitext text/x-wiki 6gd7xajyh62dlzwcbvoicesg924szy9 2441449 2441448 2026-07-28T08:17:07Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2161: Maintenance 2441449 wikitext text/x-wiki s0temfz8k0whnp24in4xxfrev082mic 2441450 2441449 2026-07-28T08:21:44Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2182: Maintenance 2441450 wikitext text/x-wiki qyauzwfgxnua1crx7hvqxg4yynmzgf3 2441452 2441450 2026-07-28T08:29:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2182: Maintenance 2441452 wikitext text/x-wiki paa36vungcpj5ocnn5s1dvs27k0nq6e 2441453 2441452 2026-07-28T08:31:02Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2210: Maintenance 2441453 wikitext text/x-wiki rpsj2o9sode7nubg2m4guw7kzcf1xr5 2441455 2441453 2026-07-28T08:33:17Z Stashbot 7414 XioNoX: draining cr1-drmrs - T431749 2441455 wikitext text/x-wiki eh2uvhst77bo42o0r5p0h9av3pdr500 2441456 2441455 2026-07-28T08:35:56Z Stashbot 7414 XioNoX: rebooting cr1-drmrs - T431749 2441456 wikitext text/x-wiki c5oztm19ohr8it6ul0kueht7b5vxx31 2441457 2441456 2026-07-28T08:42:35Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2441457 wikitext text/x-wiki ekidjli664ml1nb58ev8mx5hc16fufq 2441458 2441457 2026-07-28T08:47:20Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker 2441458 wikitext text/x-wiki fv5nnhv5urb24lrfn3r3eh7dr4z3jr0 2441459 2441458 2026-07-28T08:47:22Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet 2441459 wikitext text/x-wiki 9c2snug1z8mv5aslxhfl39fsup20gc7 2441460 2441459 2026-07-28T08:48:11Z Stashbot 7414 XioNoX: un-drain cr1-drmrs - T431749 2441460 wikitext text/x-wiki o3bd84yw08ypwpmy8bke86h18t53qnq 2441461 2441460 2026-07-28T08:57:30Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet 2441461 wikitext text/x-wiki butqutze9rilrgklq33fwi2qagl4zvo 2441462 2441461 2026-07-28T09:04:09Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet 2441462 wikitext text/x-wiki 4lhu5v3hoi4ehcuwhzcl81k5s0ogj5x 2441463 2441462 2026-07-28T09:04:11Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet 2441463 wikitext text/x-wiki j1ae263w3sz35uz2ujp7lyr8mgn8f05 2441464 2441463 2026-07-28T09:04:15Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet 2441464 wikitext text/x-wiki f4om1qc89jvu58jo0ez6rqd4qubh8ur 2441465 2441464 2026-07-28T09:06:09Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Maintenance 2441465 wikitext text/x-wiki mk3mdc1f2rn0mtz7bdi1ukuxatzsvzh 2441466 2441465 2026-07-28T09:06:32Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance 2441466 wikitext text/x-wiki dsgdr2d73l4kp5an38g30n45kjqo8ze 2441467 2441466 2026-07-28T09:06:41Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2163 (T431660)', diff saved to https://phabricator.wikimedia.org/P95320 and previous config saved to /var/cache/conftool/dbconfig/20260728-090638-cwilliams.json 2441467 wikitext text/x-wiki hreqqwpagnquul5k2x79qk96jed9ns3 2441468 2441467 2026-07-28T09:06:59Z Stashbot 7414 XioNoX: draining cr2-drmrs - T431749 2441468 wikitext text/x-wiki p2x8hwg3uk01mjfwcrutyoafvmbb97a 2441469 2441468 2026-07-28T09:10:46Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: router upgrade 2441469 wikitext text/x-wiki eb4bw62gjf87idfhev95gr54vtulcjs 2441470 2441469 2026-07-28T09:11:06Z Stashbot 7414 XioNoX: rebooting cr2-drmrs - T431749 2441470 wikitext text/x-wiki 0x4kmpmj9s98evjlcpitkifjd4mpks7 2441471 2441470 2026-07-28T09:12:46Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Maintenance 2441471 wikitext text/x-wiki 4haeea5c13xduy5hz0ltx10ikep9dxi 2441472 2441471 2026-07-28T09:14:24Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet 2441472 wikitext text/x-wiki juum3t8gy232lcgibilxzxt00ok8ok8 2441473 2441472 2026-07-28T09:15:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2182: Maintenance 2441473 wikitext text/x-wiki gpfsazqz15abqd8sf9kyu9btyqg7de6 2441474 2441473 2026-07-28T09:16:25Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f4-codfw 2441474 wikitext text/x-wiki 5ghh0fxwj9am99rqbpx25t9fh2mkezk 2441475 2441474 2026-07-28T09:16:31Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2210: Maintenance 2441475 wikitext text/x-wiki oyezm0q6afwz2j01xenlkrn965r8ib4 2441476 2441475 2026-07-28T09:17:26Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f4-codfw 2441476 wikitext text/x-wiki k0sz14tl9nhi4yirrqkwmsjyslme1vh 2441477 2441476 2026-07-28T09:17:29Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-e5-codfw 2441477 wikitext text/x-wiki npj8bki84k375k4hiqbhiux23t2g7bl 2441478 2441477 2026-07-28T09:17:35Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-codfw 2441478 wikitext text/x-wiki tsrhhqqqk33s788qu5nae2u9txweq11 2441479 2441478 2026-07-28T09:17:38Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-e2-codfw 2441479 wikitext text/x-wiki 4di2as94skv6op9ast8j3s4f3j2ezt3 2441480 2441479 2026-07-28T09:17:50Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e2-codfw 2441480 wikitext text/x-wiki jn1g0r0x10ztqwgoro24g8ldb860hnl 2441481 2441480 2026-07-28T09:17:52Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-e4-codfw 2441481 wikitext text/x-wiki bnq7y57f8l83bliesyuaklurnw5q06n 2441482 2441481 2026-07-28T09:18:23Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2163: Maintenance 2441482 wikitext text/x-wiki 63okpofx3cisve1ep2t51y9loc09idb 2441483 2441482 2026-07-28T09:18:31Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e4-codfw 2441483 wikitext text/x-wiki 4f8wiuwu92m63clgssq93ulnv1d4ibj 2441484 2441483 2026-07-28T09:18:33Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f2-codfw 2441484 wikitext text/x-wiki a20mu628fgfw9qnzw9fbw849yvjtdvh 2441485 2441484 2026-07-28T09:19:24Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f2-codfw 2441485 wikitext text/x-wiki ojwvpadbplwk3vt90e62kmspixlcl2m 2441486 2441485 2026-07-28T09:20:55Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet 2441486 wikitext text/x-wiki q6ikfnh0z373rdfosamqj2zkpz1sv7n 2441487 2441486 2026-07-28T09:20:57Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet 2441487 wikitext text/x-wiki thj8od2b6qzxs0kbtykmglz9k0mqi4g 2441488 2441487 2026-07-28T09:21:01Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet 2441488 wikitext text/x-wiki m3r0yoxs6o4ii2rtwe3cei759ds8vsa 2441490 2441488 2026-07-28T09:22:29Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d8-eqiad 2441490 wikitext text/x-wiki ia44cuyric49al6ra46en3pfy0cik90 2441491 2441490 2026-07-28T09:22:42Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d8-eqiad 2441491 wikitext text/x-wiki diws66z4r7qr183w86s1tm42ywqrord 2441492 2441491 2026-07-28T09:22:55Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet 2441492 wikitext text/x-wiki cdoo3garuu9dula7a1d365ldz3fxx0m 2441493 2441492 2026-07-28T09:22:57Z Stashbot 7414 klausman@cumin1003: END (ERROR) - Cookbook sre.ganeti.reboot-vm (exit_code=97) for VM ml-serve-ctrl2001.codfw.wmnet 2441493 wikitext text/x-wiki ahbw2tq45c623ywyl0vz9rw6e60yuxn 2441494 2441493 2026-07-28T09:30:15Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2441494 wikitext text/x-wiki 1exikjlkf2mllt198i5zmu26fzt1lo2 2441495 2441494 2026-07-28T09:30:26Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2441495 wikitext text/x-wiki 7xrcm6u294tbptoymsi58dhk9a58pfo 2441496 2441495 2026-07-28T09:30:38Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: router upgrade, T431749] 2441496 wikitext text/x-wiki ji3fq91ffw8qcfzpwibmpg6qsanhmjg 2441497 2441496 2026-07-28T09:30:40Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Maintenance 2441497 wikitext text/x-wiki fm4btv0i1jeg532n18sltyqc5lchwpp 2441498 2441497 2026-07-28T09:30:43Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: router upgrade, T431749] 2441498 wikitext text/x-wiki c7rragvfvrqktopywppgk7ddjf7c7mr 2441499 2441498 2026-07-28T09:31:09Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet 2441499 wikitext text/x-wiki hy9rljvjjdrxmrtt30xu62xlgyzv6mf 2441500 2441499 2026-07-28T09:32:26Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2441500 wikitext text/x-wiki d3fpvshooez8zbgcd7erty4kxbq7mn1 2441501 2441500 2026-07-28T09:37:08Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet 2441501 wikitext text/x-wiki exbnjpmqrkki5my1uzazlu8qxt2dpi0 2441502 2441501 2026-07-28T09:37:11Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet 2441502 wikitext text/x-wiki f0iz7vs8kwvai7y3cqlpefkdaue0379 2441503 2441502 2026-07-28T09:37:13Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker 2441503 wikitext text/x-wiki t9ixz5bs7j8rfq3nnrvfqafnc7we8o7 2441505 2441503 2026-07-28T10:16:09Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Maintenance 2441505 wikitext text/x-wiki eemurzcy7hgmcnn46d8tpjc0qg1bvw0 2441506 2441505 2026-07-28T10:20:04Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1137.eqiad.wmnet 2441506 wikitext text/x-wiki 7rj3mm10drti584d28u1ns8we2g6uqa 2441507 2441506 2026-07-28T10:20:08Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1137.eqiad.wmnet 2441507 wikitext text/x-wiki h1o7jvxec2sa7b0xtbdythuftu30yzu 2441508 2441507 2026-07-28T10:20:36Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply 2441508 wikitext text/x-wiki pbuyhzr197tyu2kp51c8e1tl4gj61ch 2441509 2441508 2026-07-28T10:20:41Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1137.eqiad.wmnet 2441509 wikitext text/x-wiki 2t8nloq3nzyr3sn7d46srf49rcmwd1o 2441510 2441509 2026-07-28T10:21:12Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply 2441510 wikitext text/x-wiki a7z5dyc9erziaz25cclai1348d3yfo8 2441511 2441510 2026-07-28T10:21:23Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2441511 wikitext text/x-wiki pm0ppj2ng2n7plimmw5w70h6bjkg3k4 2441512 2441511 2026-07-28T10:21:27Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2441512 wikitext text/x-wiki ingpywyzp3rjo6o819t28a32fgrn8r0 2441513 2441512 2026-07-28T10:25:24Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1307506|logging: Switch the wmfconfig processor to Monolog 3's type (LogRecord) (T397070)]] 2441513 wikitext text/x-wiki nr4a870p0t12q94o222abn57so3h22s 2441514 2441513 2026-07-28T10:27:37Z Stashbot 7414 jforrester@deploy1003: jforrester: Backport for [[gerrit:1307506|logging: Switch the wmfconfig processor to Monolog 3's type (LogRecord) (T397070)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441514 wikitext text/x-wiki jrjpaj3x3icu26kxknhvicv21xyat2o 2441515 2441514 2026-07-28T10:28:34Z Stashbot 7414 jforrester@deploy1003: jforrester: Continuing with deployment 2441515 wikitext text/x-wiki e2a7bcv6qc8iiuxdyf05fbio3ex5g32 2441516 2441515 2026-07-28T10:34:25Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1137.eqiad.wmnet with OS trixie 2441516 wikitext text/x-wiki kfme02nr8sv9o8xvj2xsvtwbvphd33w 2441517 2441516 2026-07-28T10:34:37Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-codfw 2441517 wikitext text/x-wiki qaysxx2vr9c8bcxerw99nn6cmxoc39m 2441518 2441517 2026-07-28T10:34:43Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2001.codfw.wmnet 2441518 wikitext text/x-wiki 8mt3j5qhdx9jqdyz8ck9wb2k09lui2r 2441519 2441518 2026-07-28T10:34:53Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1137 2441519 wikitext text/x-wiki piv2vrw601wjiz4m92tvbnuj6dwyb9q 2441520 2441519 2026-07-28T10:34:57Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307506|logging: Switch the wmfconfig processor to Monolog 3's type (LogRecord) (T397070)]] (duration: 09m 31s) 2441520 wikitext text/x-wiki f6198hhhpbjwguub1252ejmh6s71bc1 2441521 2441520 2026-07-28T10:35:05Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2441521 wikitext text/x-wiki oeua43pxngdtujvrhmnxc0futupncon 2441522 2441521 2026-07-28T10:39:49Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2001.codfw.wmnet 2441522 wikitext text/x-wiki db3v7wwkmqq6m09928ny0ee77ssrrtx 2441523 2441522 2026-07-28T10:47:11Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2001.codfw.wmnet 2441523 wikitext text/x-wiki gwcyn6xxjq46ddg5jmcckg4a6l8avtb 2441524 2441523 2026-07-28T10:47:12Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2001.codfw.wmnet 2441524 wikitext text/x-wiki dd0p1wij2f6qlx3bl2jmubx8qldos1d 2441525 2441524 2026-07-28T10:47:18Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2002.codfw.wmnet 2441525 wikitext text/x-wiki jnvfopo0ot2o2bjafe2kjz5a4gub002 2441526 2441525 2026-07-28T10:52:26Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2002.codfw.wmnet 2441526 wikitext text/x-wiki rzav03sbfx84gd41vzizsdfdl4fr9k2 2441527 2441526 2026-07-28T10:53:54Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1137 - jiji@cumin1003" 2441527 wikitext text/x-wiki kdk7uwy5yhput9c50sp3aqvvlozucih 2441528 2441527 2026-07-28T10:56:46Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance 2441528 wikitext text/x-wiki 46lkrd5vetja7erkjowrefjgumsylwv 2441529 2441528 2026-07-28T10:56:54Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2219 (T431660)', diff saved to https://phabricator.wikimedia.org/P95330 and previous config saved to /var/cache/conftool/dbconfig/20260728-105652-cwilliams.json 2441529 wikitext text/x-wiki 8dmdzgfx9odd1qy6erq3iz0uzk6b1f4 2441530 2441529 2026-07-28T10:57:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance 2441530 wikitext text/x-wiki javwbqwe2jh0227vkfm4ti3aaxugo4p 2441531 2441530 2026-07-28T10:57:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2208 (T431660)', diff saved to https://phabricator.wikimedia.org/P95331 and previous config saved to /var/cache/conftool/dbconfig/20260728-105711-cwilliams.json 2441531 wikitext text/x-wiki l7y6tqo6gkdbxo48xpniuyrgzknv95u 2441532 2441531 2026-07-28T10:57:42Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance 2441532 wikitext text/x-wiki c6gsy49r8u0hflmsqk9kk9wyeo2ewlv 2441533 2441532 2026-07-28T10:57:50Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2164 (T431660)', diff saved to https://phabricator.wikimedia.org/P95332 and previous config saved to /var/cache/conftool/dbconfig/20260728-105749-cwilliams.json 2441533 wikitext text/x-wiki 5mewz8ep8gcbayi2lllw34zb8grj4ta 2441534 2441533 2026-07-28T11:03:30Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2208: Maintenance 2441534 wikitext text/x-wiki oliykbixdy8zyoj0bv2iifna3g1kwsi 2441535 2441534 2026-07-28T11:03:57Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2002.codfw.wmnet 2441535 wikitext text/x-wiki pdz6qv7ip4l3i6btz3nps8rmgp3ab87 2441536 2441535 2026-07-28T11:03:59Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2002.codfw.wmnet 2441536 wikitext text/x-wiki 1epsn60uafa1c3vpnb5tuz8602xllf7 2441537 2441536 2026-07-28T11:04:02Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2164: Maintenance 2441537 wikitext text/x-wiki 6eldo2jn7kjbzuxk8olpl5hho2bbfk3 2441538 2441537 2026-07-28T11:04:05Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2003.codfw.wmnet 2441538 wikitext text/x-wiki krxqv0evxg5w0hx37ib0m05q5vqpiu7 2441539 2441538 2026-07-28T11:04:25Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2219: Maintenance 2441539 wikitext text/x-wiki 1e27e1dt0avejik9q3m1hjo4y4wsmwe 2441540 2441539 2026-07-28T11:10:01Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2219: Maintenance 2441540 wikitext text/x-wiki 6byl9xggbo3kc26rxolugsdnb3u2gdm 2441541 2441540 2026-07-28T11:11:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2219: Maintenance 2441541 wikitext text/x-wiki mwm4bgqh7cagzrm12t09nh00viiq9kp 2441542 2441541 2026-07-28T11:14:13Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2003.codfw.wmnet 2441542 wikitext text/x-wiki dukklxtvhzi4ni9clwjjzuj9xq0l7b0 2441543 2441542 2026-07-28T11:19:47Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2003.codfw.wmnet 2441543 wikitext text/x-wiki cez40us810sk27jnsrll6yhurselvpa 2441544 2441543 2026-07-28T11:19:49Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2003.codfw.wmnet 2441544 wikitext text/x-wiki 9msqd52gfc3ibtrzpo62k4el5qclneh 2441545 2441544 2026-07-28T11:19:55Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2004.codfw.wmnet 2441545 wikitext text/x-wiki adbquq5jco9i8p4xbzqbov7v67bpn52 2441547 2441545 2026-07-28T11:25:04Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2004.codfw.wmnet 2441547 wikitext text/x-wiki 03o1iqvregf6ubs6kxy9qbscz8qryca 2441548 2441547 2026-07-28T11:30:02Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1137 - jiji@cumin1003" 2441548 wikitext text/x-wiki isibtmq9qu065y6wq8di50l1q8nkbtk 2441549 2441548 2026-07-28T11:30:04Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441549 wikitext text/x-wiki ejbpji8usinaxd098rfiqcgiirntvu7 2441550 2441549 2026-07-28T11:30:06Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1137.eqiad.wmnet 192.32.64.10.in-addr.arpa 2.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441550 wikitext text/x-wiki 5nabbcee63bf6bzcfdb26zsdffi62q9 2441551 2441550 2026-07-28T11:30:08Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1137.eqiad.wmnet 192.32.64.10.in-addr.arpa 2.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441551 wikitext text/x-wiki 4uy3xt2gf1ithhtfizom8pc4lmiei68 2441552 2441551 2026-07-28T11:30:10Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1137 2441552 wikitext text/x-wiki 5jyog70b6t2fupqfepn4xkazopf7vie 2441553 2441552 2026-07-28T11:30:47Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1137 2441553 wikitext text/x-wiki sa1y0jqagsnhkwrvs4b1mj0w6ijixlp 2441554 2441553 2026-07-28T11:30:48Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1137 2441554 wikitext text/x-wiki bpjmxltdxwoaxkiz7rvziresyvti7g1 2441555 2441554 2026-07-28T11:35:25Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2004.codfw.wmnet 2441555 wikitext text/x-wiki fynwbrqb8bk2ygwz1avdwlw6omdbi2g 2441556 2441555 2026-07-28T11:35:27Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2004.codfw.wmnet 2441556 wikitext text/x-wiki ggs4949bsnvzgepb0ys5z8vcojs9tld 2441557 2441556 2026-07-28T11:35:32Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet 2441557 wikitext text/x-wiki c2isxocl61euskxpy4dlrz5rc65cmap 2441558 2441557 2026-07-28T11:40:39Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet 2441558 wikitext text/x-wiki 7jjxb0oi68pky11be649rcyxxyraoht 2441559 2441558 2026-07-28T11:45:48Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet 2441559 wikitext text/x-wiki 1eox6bgnayfr6f5jj3ah5jylc557s2t 2441560 2441559 2026-07-28T11:45:49Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet 2441560 wikitext text/x-wiki 4mlbb2chk78d5dzt2lo3awnt1kygp9c 2441561 2441560 2026-07-28T11:45:55Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2006.codfw.wmnet 2441561 wikitext text/x-wiki nz5i9nzvnktn87f7av10x5ty65qhj2z 2441562 2441561 2026-07-28T11:47:21Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1137.eqiad.wmnet with reason: host reimage 2441562 wikitext text/x-wiki ju0x4zjxq3c7f0oxxxo5cgftau2nwz4 2441563 2441562 2026-07-28T11:50:53Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Maintenance 2441563 wikitext text/x-wiki 1ejyz21umfg27yy3q6iszkolh3gi2qz 2441564 2441563 2026-07-28T11:51:13Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance 2441564 wikitext text/x-wiki h0to32p9dcj8xrgcnvsiu9cjixdyfhw 2441565 2441564 2026-07-28T11:51:21Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2166 (T431660)', diff saved to https://phabricator.wikimedia.org/P95342 and previous config saved to /var/cache/conftool/dbconfig/20260728-115119-cwilliams.json 2441565 wikitext text/x-wiki efmk63kynsegm94mjyaplcxq4lurs9c 2441566 2441565 2026-07-28T11:51:28Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2208: Maintenance 2441566 wikitext text/x-wiki tpc8ac6zsslhgtlk6dfau3651b7o92v 2441567 2441566 2026-07-28T11:51:48Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance 2441567 wikitext text/x-wiki sad78hkq5lr67e51k3cs0zs27prdf37 2441568 2441567 2026-07-28T11:51:57Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2218 (T431660)', diff saved to https://phabricator.wikimedia.org/P95344 and previous config saved to /var/cache/conftool/dbconfig/20260728-115155-cwilliams.json 2441568 wikitext text/x-wiki kazrvlq904x33raxfn2qnj9rfdijk6n 2441569 2441568 2026-07-28T11:52:27Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1137.eqiad.wmnet with reason: host reimage 2441569 wikitext text/x-wiki ad7j246741ullo9yzydw3f08krh6m83 2441574 2441569 2026-07-28T11:56:03Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2006.codfw.wmnet 2441574 wikitext text/x-wiki ohjt5ptwblymbxvxeih8zia2swbtnqh 2441575 2441574 2026-07-28T11:57:02Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2219: Maintenance 2441575 wikitext text/x-wiki 9714ckt0bt8iwh2l67s6zl0mg6oqlkx 2441578 2441575 2026-07-28T11:57:28Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2166: Maintenance 2441578 wikitext text/x-wiki i9qbhkh7clruikjlah4teifbuzvktjo 2441579 2441578 2026-07-28T11:58:03Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2218: Maintenance 2441579 wikitext text/x-wiki mh9lhu5ijukh52wyllsqlllmm7fsnft 110 16009 2441218 2401042 2026-07-27T15:56:32Z HNowlan (WMF) 15760 Add outdated info template. 2441218 wikitext text/x-wiki 09wn5voma7eu0timzvgnkegp3dh6kw2 2441221 2441218 2026-07-27T16:11:24Z BCornwall-WMF 31521 BCornwall-WMF moved page [[History of job queue runners at WMF]] to [[Obsolete:History of job queue runners at WMF]] 2441218 wikitext text/x-wiki 09wn5voma7eu0timzvgnkegp3dh6kw2 111 16069 2441225 1887841 2026-07-27T16:11:25Z BCornwall-WMF 31521 BCornwall-WMF moved page [[Talk:History of job queue runners at WMF/Jobrunner]] to [[Obsolete talk:History of job queue runners at WMF/Jobrunner]] 120492 wikitext text/x-wiki 3gfbcpjow8aoslq6vayna4py6g74yxe 12 18678 2441546 2405192 2026-07-28T11:21:57Z Meno25 1446 update redirected link 2441546 wikitext text/x-wiki aeu55itmdof14l0k73btxvs3dd9jvr6 0 24248 2441359 2431146 2026-07-27T21:55:44Z ADenisse-WMF 31462 /* Operations */ 2441359 wikitext text/x-wiki c17in27x3d2t0rxnn7s7b7lrrm6nkvs 498 30942 2441211 2440714 2026-07-27T15:51:08Z Stashbot 7414 andrewbogott: temporarily muting ceph slow ops alerts ("ceph health mute BLUESTORE_SLOW_OP_ALERT --sticky") for T431659 2441211 wikitext text/x-wiki qa4ull3kdco25flnranizog3b88jpe6 2441214 2441211 2026-07-27T15:52:46Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds 2441214 wikitext text/x-wiki ouc98zj06jvptfjl2likbplmjwdvpuk 2441228 2441214 2026-07-27T16:26:54Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) 2441228 wikitext text/x-wiki d64d4gg2e8a4twcxxr46gsnpqnpryfs 2441303 2441228 2026-07-27T18:49:54Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds 2441303 wikitext text/x-wiki 1ibdx2bjdqg82rv7mxd91jiarxdrc5z 2441304 2441303 2026-07-27T18:49:57Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) 2441304 wikitext text/x-wiki mnk1a9xthrq643s6comy5a561nswpji 2441305 2441304 2026-07-27T18:50:26Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds 2441305 wikitext text/x-wiki cwal9y6awkggxztqtinf0sefrvdb5f5 2441367 2441305 2026-07-27T22:33:30Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) 2441367 wikitext text/x-wiki 0ofax5j8vu6ld18advvf1awzj22prjn 2441392 2441367 2026-07-28T01:57:24Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds 2441392 wikitext text/x-wiki qg9loo53hz5oez5rwfvxsn48s7xidbh 2441393 2441392 2026-07-28T02:02:34Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) 2441393 wikitext text/x-wiki ibq2f1jvhzksfrtsln7dq5caqgjgsi2 2441394 2441393 2026-07-28T02:07:40Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds 2441394 wikitext text/x-wiki 5q9bo1z80ho3fxkbfc2xbawsuq6p98l 2441399 2441394 2026-07-28T02:40:14Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) 2441399 wikitext text/x-wiki rx2a0caly82xi8p02y49jniheh0yvxz 2441404 2441399 2026-07-28T03:13:07Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node 2441404 wikitext text/x-wiki spw5unvsqcth34csinrph8tv9h8nas1 2441570 2441404 2026-07-28T11:55:19Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441570 wikitext text/x-wiki 00mbulo7z7cb6migvvfoncx18emev9z 2441571 2441570 2026-07-28T11:55:48Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441571 wikitext text/x-wiki 0z99fajavkymy8fg4u6oyw7gwmgwx6w 2441572 2441571 2026-07-28T11:55:52Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441572 wikitext text/x-wiki 2jd073lq76p0as2vmsul6ss099zlz3l 2441573 2441572 2026-07-28T11:55:55Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441573 wikitext text/x-wiki o7rbagwjqtaoqpu4czrx919rpbwmizp 2441577 2441573 2026-07-28T11:57:25Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441577 wikitext text/x-wiki 2n2oj7loi1vf6yqohx2d2q2lwzn9t54 0 33560 2441111 2439048 2026-07-27T13:54:57Z SLyngshede-WMF 31501 /* IDP */ Remove notes on memcached. We now use the shared redis cluster. 2441111 wikitext text/x-wiki tgkspbpdh6zsw7eqg1mnbvyrhhnl1a5 498 55577 2441418 2435492 2026-07-28T07:15:52Z Stashbot 7414 godog: set up clouddumps nfs audit via network capture - T432583 2441418 wikitext text/x-wiki t6dj9va0o5zpdcwezo5ar44knxqbfj4 10 391699 2441407 1919544 2026-07-28T05:49:50Z Axolitl 56776 /* */ Adding class=skin-invert for dark mode 2441407 wikitext text/x-wiki 6kff1w6s35l7vium5tuwa5ukn24g4g8 2441408 2441407 2026-07-28T05:55:16Z Axolitl 56776 /* */ Changing alt text to "@", adding noinclude tags 2441408 wikitext text/x-wiki hh9b0mct8cfshb2jxtf62596a9q185m 2441409 2441408 2026-07-28T05:58:28Z Axolitl 56776 /* */ Fixed noinclude tags 2441409 wikitext text/x-wiki npffoyq4qxenbxhcz8p618d6pl5whnq 2 440766 2441165 2368614 2026-07-27T14:39:56Z Valerio Bozzolan 10109 +survey info 2441165 wikitext text/x-wiki t8as9rws9l86xg793ml3pxvvrl1a5xn 498 441911 2441348 2431185 2026-07-27T21:39:13Z Stashbot 7414 mutante: - deleting volume "sourcebot-data" - resources back to wmcs - 150GB of gerrit and gitlab repos for sourcebot which we did not end up using T268199 2441348 wikitext text/x-wiki pdy809q2yns10izbb9mnyk4c9rj49yi 0 442653 2441087 2197404 2026-07-27T12:58:17Z LToscano (WMF) 7191 move root to wmfroot 2441087 wikitext text/x-wiki mxfbpv9kpesc4e1nvlo9a19lqziixdl 0 443680 2441433 2429033 2026-07-28T07:53:58Z JWodstrcil (WMF) 24826 add a section about helmBinary update 2441433 wikitext text/x-wiki 2cb1n0uegac8fzo20r6sud8n8z7b1cl 0 445495 2441357 2337661 2026-07-27T21:54:05Z Krinkle 44 /* See also */ 2441357 wikitext text/x-wiki 7xf3md27atv38geww0djgz133scagyv 0 445760 2441217 2389280 2026-07-27T15:54:34Z HNowlan (WMF) 15760 Add details distinguishing between changeprop and the jobqueue. Add jobs list - rule list coming in another edit. 2441217 wikitext text/x-wiki bvjkt5ioqv4udm10gpqoqewqb2ncp3x 2441504 2441217 2026-07-28T09:52:51Z HNowlan (WMF) 15760 Add changeprop rules list. Move "What it does" above the lists. Correct references to restbase 2441504 wikitext text/x-wiki co5eulgq4uax4i66mabs0l0ohwq2evg 0 446801 2441427 1887825 2026-07-28T07:20:38Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Obsolete:History of job queue runners at WMF]] 2441427 wikitext text/x-wiki 4ui92mu6yqrxlqa95oc8lieseqrn9hc 1 446802 2441428 1887842 2026-07-28T07:20:39Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Obsolete:History of job queue runners at WMF/Jobrunner]] 2441428 wikitext text/x-wiki ggtc013y6uddm0di2gvjesqzutdq43m 0 446803 2441425 1887822 2026-07-28T07:20:35Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Obsolete:History of job queue runners at WMF]] 2441425 wikitext text/x-wiki 4ui92mu6yqrxlqa95oc8lieseqrn9hc 10 448135 2441101 2424628 2026-07-27T13:44:27Z JMeybohm (WMF) 16709 Add Infrastructure Upgrade Procedure 2441101 wikitext text/x-wiki hutmippjb1r8r6cufwpk4carnt5jmd2 2441103 2441101 2026-07-27T13:48:05Z JMeybohm (WMF) 16709 Split policies and administation 2441103 wikitext text/x-wiki jayr1a79h9fuk7f00tsn374kcpu5k20 0 449160 2441114 2441057 2026-07-27T13:55:57Z Dexbot 30554 Bot: Updating the report 2441114 wikitext text/x-wiki 3r6l0mmysp8abj0m6c6qx1e3g723cjl 2441120 2441114 2026-07-27T13:58:58Z Dexbot 30554 Bot: Updating the report 2441120 wikitext text/x-wiki qovwz70got76ubb0sqiuan9q31j4wtf 2441129 2441120 2026-07-27T14:01:59Z Dexbot 30554 Bot: Updating the report 2441129 wikitext text/x-wiki 4ex6eyc3a9ar1esoff7lgectrvd5ctc 2441377 2441129 2026-07-28T00:00:32Z Dexbot 30554 Bot: Updating the report 2441377 wikitext text/x-wiki 3q2bpbket4oado2ei6gxlq1jv8bou9q 2441416 2441377 2026-07-28T07:04:27Z Dexbot 30554 Bot: Updating the report 2441416 wikitext text/x-wiki cgsjev32p2gc8xmtg6aop9svgt0bipb 2441421 2441416 2026-07-28T07:16:29Z Dexbot 30554 Bot: Updating the report 2441421 wikitext text/x-wiki 17odrau6ir2hul8h3qqufhs5cax129b 2441424 2441421 2026-07-28T07:19:30Z Dexbot 30554 Bot: Updating the report 2441424 wikitext text/x-wiki 5a633ucoykaag084oeg0sr2iahgo8tk 0 450789 2441096 2424840 2026-07-27T13:23:27Z Lucas Werkmeister (WMDE) 10037 /* Benchmarking */ fix missing space 2441096 wikitext text/x-wiki rpdjzkjjnnkbk0y2md5wq7zizpjss6a 0 451075 2441322 2435971 2026-07-27T20:02:12Z WFan (WMF) 38087 2441322 wikitext text/x-wiki i5ju00wu0ilw1vzqalw1yzqtlkae1jk 116 453906 2441489 2440942 2026-07-28T09:21:33Z Gitlabaccountapprovalbot 37332 @gka was approved. 2441489 wikitext text/x-wiki oii8w1xxwngtpoa9migj2olervkgb9w 0 458856 2441328 2412077 2026-07-27T20:42:21Z Krinkle 44 /* Known limitations */ restore link to [[phab:T405688|T405688: Support shell to mw-experimental pod]] 2441328 wikitext text/x-wiki 72y0bmj3a3t2ke70icfde430nlianyr 0 459044 2441134 2440709 2026-07-27T14:07:35Z BKing (WMF) 29017 /* Banning nodes from the cluster */ 2441134 wikitext text/x-wiki ebndobfv58kmh9ryg4ugfgn51sq59vc 2441136 2441134 2026-07-27T14:09:18Z BKing (WMF) 29017 /* Banning nodes from the cluster */ 2441136 wikitext text/x-wiki 38o1zao1qyvhlwm1xphyoxx43nz1b5a 2441144 2441136 2026-07-27T14:15:04Z BKing (WMF) 29017 2441144 wikitext text/x-wiki abkvw1cn57dow2w9gibqbnhkvc24m9o 2441163 2441144 2026-07-27T14:33:45Z BKing (WMF) 29017 /* Alerts */ 2441163 wikitext text/x-wiki kheqjtac433hktuyn8whyguhwjyo1r6 0 459290 2441576 2439019 2026-07-28T11:57:20Z JAllemandou (WMF) 37709 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ Add patch to deploy 2441576 wikitext text/x-wiki 2yov9cj0ohntuosxzq8eut5jfe2ymot 0 460216 2441099 2424309 2026-07-27T13:43:40Z JMeybohm (WMF) 16709 JMeybohm (WMF) moved page [[Kubernetes/Kubernetes Infrastructure Upgrade Procedure]] to [[Kubernetes/Infrastructure Upgrade Procedure]] 2424309 wikitext text/x-wiki opvrnvcre3xpd2hi4x24yzets0sf3he 0 460231 2441146 2435554 2026-07-27T14:15:47Z BKing (WMF) 29017 /* Extra Precautions for "perma-yellow" */ 2441146 wikitext text/x-wiki s8cehtdza8s56j4jppd10cwtaxsqjor 0 460491 2441100 2026-07-27T13:43:40Z JMeybohm (WMF) 16709 JMeybohm (WMF) moved page [[Kubernetes/Kubernetes Infrastructure Upgrade Procedure]] to [[Kubernetes/Infrastructure Upgrade Procedure]] 2441100 wikitext text/x-wiki 12gzrtxcpu80mg163gj69mi0hg5f2i4 2 460492 2441197 2026-07-27T15:33:31Z DARQ CEE 58064 /* */ 2441197 wikitext text/x-wiki dc95z5c3hosfeumy0qemv8w43i18piw 0 460493 2441222 2026-07-27T16:11:24Z BCornwall-WMF 31521 BCornwall-WMF moved page [[History of job queue runners at WMF]] to [[Obsolete:History of job queue runners at WMF]] 2441222 wikitext text/x-wiki 4ui92mu6yqrxlqa95oc8lieseqrn9hc 0 460494 2441224 2026-07-27T16:11:25Z BCornwall-WMF 31521 BCornwall-WMF moved page [[History of job queue runners at WMF/Jobrunner]] to [[Obsolete:History of job queue runners at WMF/Jobrunner]] 2441224 wikitext text/x-wiki ggtc013y6uddm0di2gvjesqzutdq43m 1 460495 2441226 2026-07-27T16:11:25Z BCornwall-WMF 31521 BCornwall-WMF moved page [[Talk:History of job queue runners at WMF/Jobrunner]] to [[Obsolete talk:History of job queue runners at WMF/Jobrunner]] 2441226 wikitext text/x-wiki leh8dx17sjb1pnnbjvdtdlnzr9n0da8