Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.12 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2440285 2440282 2026-07-23T12:00:58Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2440285 wikitext text/x-wiki 725n8ynh2imez0jfwa5tzgpm7662cfh 2440299 2440285 2026-07-23T12:22:34Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2440299 wikitext text/x-wiki 0korxlhxn16c6idrmeg1t9afc2dv63d 2440303 2440299 2026-07-23T12:30:37Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2440303 wikitext text/x-wiki tau264sjimfsv348zd8fysembip1o2z 2440475 2440303 2026-07-23T16:50:26Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 32 cores, 65536 ram 2440475 wikitext text/x-wiki f4jxuf8qr5no16jpaayj4l92wf8hbto 2440476 2440475 2026-07-23T16:50:33Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 32 cores, 65536 ram 2440476 wikitext text/x-wiki k57q6ln2iy1rixrt0vck2xl9g2jyek9 2440477 2440476 2026-07-23T16:55:12Z Stashbot 7414 dcaro: resize haproxy-7 2440477 wikitext text/x-wiki k8g4dohbbbj60qs8rzvlkl67ltecxvk 2440478 2440477 2026-07-23T16:56:22Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2440478 wikitext text/x-wiki 29ciihamh60ygz3lijmexml7s5osuk6 2440485 2440478 2026-07-23T17:11:25Z Stashbot 7414 dcaro: resize haproxy-8 2440485 wikitext text/x-wiki q74ltveuhapbhig81mk5240s93vqif4 0 4108 2440309 2440090 2026-07-23T12:53:54Z ScheduleDeploymentBot 37566 Add [[gerrit:1311851]] to Thursday, July 23 UTC afternoon backport window 2440309 wikitext text/x-wiki 17mqx3mxfk9j5ulh3aupxu98on8exd3 2440336 2440309 2026-07-23T13:49:02Z Brennen 54136 Remove a couple of Train Log Triage instances - we're experimentally canceling this meeting. 2440336 wikitext text/x-wiki 3wg1s3cty4fretrdegf1lb95dew10yx 2440528 2440336 2026-07-23T18:44:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1314951]] to Thursday, July 23 UTC late backport window 2440528 wikitext text/x-wiki c5t7sh8mse5b0w9e7go1p5stjfxtmji 0 4199 2440540 2420667 2026-07-23T19:40:49Z Mutante 20 update info on Java version and ticket 2440540 wikitext text/x-wiki gu7kzbzn3qh1bkf7j5exnc2vlqyr7ht 498 6726 2440292 2440278 2026-07-23T12:12:17Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2440292 wikitext text/x-wiki n6oze7eaow1u3mle79bxxzb7pinqtmi 2440298 2440292 2026-07-23T12:20:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2440298 wikitext text/x-wiki izzcbr0mdlqjrc9rd096evusz9g1pz3 0 7919 2440293 2440284 2026-07-23T12:12:46Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2207: Repooling 2440293 wikitext text/x-wiki 5bvwkmqrzcrcuctuu80nszmd3rrv4xl 2440294 2440293 2026-07-23T12:12:50Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Repooling 2440294 wikitext text/x-wiki 868sissmc5kkgace5ji3nd7nrhrfok9 2440295 2440294 2026-07-23T12:13:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2207: Repooling 2440295 wikitext text/x-wiki fps96oqrissxje9bbnml0vce27zg40q 2440296 2440295 2026-07-23T12:13:05Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440296 wikitext text/x-wiki s0vgpeyb5jo6gnpaq6czthbz9oogb09 2440300 2440296 2026-07-23T12:28:33Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2440300 wikitext text/x-wiki 6xnp43zj7wq47b0r1y3lh3y604xwo19 2440301 2440300 2026-07-23T12:29:47Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2440301 wikitext text/x-wiki e5wq03ozj1z5rtarinql3zux71laier 2440304 2440301 2026-07-23T12:36:02Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Maintenance 2440304 wikitext text/x-wiki h7e8ihd21ayl37ca35obu67o10pvruc 2440305 2440304 2026-07-23T12:39:45Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2196.codfw.wmnet with reason: Maintenance 2440305 wikitext text/x-wiki r5wowvopaxf7onaolcmwhjebjq4yd7f 2440306 2440305 2026-07-23T12:39:53Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2196 (T431660)', diff saved to https://phabricator.wikimedia.org/P95087 and previous config saved to /var/cache/conftool/dbconfig/20260723-123952-cwilliams.json 2440306 wikitext text/x-wiki fv669chjnba0ol04ezum2w9hcre8auo 2440307 2440306 2026-07-23T12:49:48Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Maintenance 2440307 wikitext text/x-wiki mktuq5q348fg0tz2fuw5c1z2fuzy411 2440310 2440307 2026-07-23T12:58:30Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Repooling 2440310 wikitext text/x-wiki jd174bq5xpohzl9gypvtkuoaedkplef 2440311 2440310 2026-07-23T13:07:16Z Stashbot 7414 sukhe: sukhe@lvs1020:~$ sudo systemctl restart pybal.service 2440311 wikitext text/x-wiki iwwq5xduvuu2l71xzngq59zetdpkgfn 2440312 2440311 2026-07-23T13:14:36Z Stashbot 7414 sukhe: sukhe@lvs1019:~$ sudo systemctl restart pybal.service 2440312 wikitext text/x-wiki 71fl9nqte96hp0kxwosml2awsp1r8nw 2440313 2440312 2026-07-23T13:17:10Z Stashbot 7414 hashar@deploy1003: Started deploy [integration/docroot@2199146]: build: License GPL2.0+ / updating npm dependencies 2440313 wikitext text/x-wiki 5nhkh266cq08c7erghyovfhmwfqaj8s 2440314 2440313 2026-07-23T13:17:25Z Stashbot 7414 hashar@deploy1003: Finished deploy [integration/docroot@2199146]: build: License GPL2.0+ / updating npm dependencies (duration: 00m 14s) 2440314 wikitext text/x-wiki jes8wuwomylx3z3qxbmjurkmdgnlwvl 2440315 2440314 2026-07-23T13:28:56Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1311851|EventStreamConfig: remove stream used in past experiments (T428265)]] 2440315 wikitext text/x-wiki qf7bay0eowlidxr8n6ma1u3g5gj2qg2 2440316 2440315 2026-07-23T13:30:21Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2440316 wikitext text/x-wiki 4n2c8q4ygufvqcq32hyt5mvr2q7mtkt 2440317 2440316 2026-07-23T13:31:12Z Stashbot 7414 kharlan@deploy1003: kharlan, emc-wmf: Backport for [[gerrit:1311851|EventStreamConfig: remove stream used in past experiments (T428265)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2440317 wikitext text/x-wiki 2h318v31dy7qqhrnrzt838n2tbkfy60 2440319 2440317 2026-07-23T13:33:41Z Stashbot 7414 kharlan@deploy1003: kharlan, emc-wmf: Continuing with deployment 2440319 wikitext text/x-wiki 0t1ginafi0rvhc6va3cj71bl9g2zmcf 2440320 2440319 2026-07-23T13:33:46Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2440320 wikitext text/x-wiki pm6gl9otexgkevqbekw6slm70revme4 2440321 2440320 2026-07-23T13:33:51Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2440321 wikitext text/x-wiki gjvhbmcux5zfc0it92yivualym95xem 2440322 2440321 2026-07-23T13:34:54Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2440322 wikitext text/x-wiki 63l10rnpzfe8fjtbv1eiry0yiozb40m 2440323 2440322 2026-07-23T13:35:00Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2440323 wikitext text/x-wiki i5rwtnzkp4y0g3wu3cb1p5gsyp9sqf8 2440324 2440323 2026-07-23T13:35:16Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Maintenance 2440324 wikitext text/x-wiki abdltpcp0xm1gxgzwywfml66hd7zbgu 2440325 2440324 2026-07-23T13:36:20Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1037 hosts 2440325 wikitext text/x-wiki ectw9cnz5xfc6d8zsrlsktdnj8t81mz 2440326 2440325 2026-07-23T13:38:04Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1311851|EventStreamConfig: remove stream used in past experiments (T428265)]] (duration: 09m 07s) 2440326 wikitext text/x-wiki 49fn4ffnqq8adfogx31vqxdn42l9v7r 2440327 2440326 2026-07-23T13:38:45Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2440327 wikitext text/x-wiki o1fgcqch66n452qgre1umx5al7gxqoj 2440328 2440327 2026-07-23T13:39:37Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440328 wikitext text/x-wiki 2qygo3mdfdwyzzrniyh4p4yx49vmgw9 2440329 2440328 2026-07-23T13:43:21Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2440329 wikitext text/x-wiki dauw5md9t9mh3kuuoz3j0wmdgwccili 2440330 2440329 2026-07-23T13:43:24Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply 2440330 wikitext text/x-wiki hcfgg362d3vjr3f5sh6dp4r1p6089mg 2440332 2440330 2026-07-23T13:44:30Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2231.codfw.wmnet with reason: Maintenance 2440332 wikitext text/x-wiki 6peirpfkhjzt4uzh0rc9tb5pmenwrkf 2440333 2440332 2026-07-23T13:44:38Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2231 (T431660)', diff saved to https://phabricator.wikimedia.org/P95097 and previous config saved to /var/cache/conftool/dbconfig/20260723-134436-cwilliams.json 2440333 wikitext text/x-wiki qqq30he6dw0q6o6xx533fvvjokxgfoe 2440337 2440333 2026-07-23T13:49:06Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for eqiad row a b new switches - cmooney@cumin1003" 2440337 wikitext text/x-wiki daucvtxd42elgrk272dtiaukfne95di 2440338 2440337 2026-07-23T13:49:21Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for eqiad row a b new switches - cmooney@cumin1003" 2440338 wikitext text/x-wiki jzue5lhiwgsl94d9bwio1gfxh7wcnhm 2440339 2440338 2026-07-23T13:49:22Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440339 wikitext text/x-wiki 5c15il29ujef72f5iu2hlxnre993cm8 2440340 2440339 2026-07-23T13:50:31Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2440340 wikitext text/x-wiki isrqctyhb958cfi33pib4o8j01n2ny2 2440341 2440340 2026-07-23T13:50:38Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2440341 wikitext text/x-wiki r227o7l3r4qo0rrl0mvy8d6cfss6wzh 2440342 2440341 2026-07-23T13:50:44Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2440342 wikitext text/x-wiki 79azxjenwsd2zvrx6brv8lni9glcnnc 2440343 2440342 2026-07-23T13:50:50Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2440343 wikitext text/x-wiki 6y1v6jhh5bpys4sulscvkgaxtmzn0of 2440344 2440343 2026-07-23T13:50:51Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2231: Maintenance 2440344 wikitext text/x-wiki 3i4uwosa9q6iay136xwuz2ikpzz9uan 2440345 2440344 2026-07-23T13:51:06Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2440345 wikitext text/x-wiki gpdhllwgks36zmv1l7bhqdewfprp26w 2440346 2440345 2026-07-23T13:51:11Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2440346 wikitext text/x-wiki 8qk1liyko66xd0cidqzajg4hxq4z040 2440347 2440346 2026-07-23T13:51:13Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2440347 wikitext text/x-wiki 72um5tskx97w1csno7ilrk7o4r6uml3 2440348 2440347 2026-07-23T13:51:27Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2440348 wikitext text/x-wiki 32aqt2g2r9b3lmjf1iveptdtukgc3ke 2440349 2440348 2026-07-23T13:51:36Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2440349 wikitext text/x-wiki ffhtok9kg91tqdalvm2i7uc5f6lrkra 2440350 2440349 2026-07-23T13:51:44Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2440350 wikitext text/x-wiki 6kfvc5b0afncgi6acyofxlmeujcmmlg 2440351 2440350 2026-07-23T13:51:49Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2440351 wikitext text/x-wiki hhxccedpppkluvk88g33ejtngh4izhp 2440352 2440351 2026-07-23T13:52:42Z Stashbot 7414 logmsgbot: kharlan Deployed security patch for T432948 2440352 wikitext text/x-wiki gsogt88gkaxermqv231yvvdv2tta44b 2440353 2440352 2026-07-23T13:53:30Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2440353 wikitext text/x-wiki 5znp9nkzu8885a7gi7yybf6o47wv468 2440354 2440353 2026-07-23T13:55:07Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1071.eqiad.wmnet 2440354 wikitext text/x-wiki s7gl8l50q653fwhs5l1s848ax5pkafg 2440355 2440354 2026-07-23T13:55:10Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet 2440355 wikitext text/x-wiki 923qzruz3srrcmvk77fhn5xjopkw1na 2440356 2440355 2026-07-23T13:55:45Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet 2440356 wikitext text/x-wiki 5itjhx2qqh87tqjxtws7jnm7p4ru6wi 2440357 2440356 2026-07-23T13:57:38Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1071.eqiad.wmnet with OS trixie 2440357 wikitext text/x-wiki qkk9ak32rjjztj1408uoe7c79o9y34x 2440358 2440357 2026-07-23T13:57:43Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for eqiad row a b new switches - cmooney@cumin1003" 2440358 wikitext text/x-wiki h13l1vqynjqlyqwu19rtv5b6suh0j6z 2440359 2440358 2026-07-23T13:57:47Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for eqiad row a b new switches - cmooney@cumin1003" 2440359 wikitext text/x-wiki gm2bu92q09l00zp1da36zue6f64bud9 2440360 2440359 2026-07-23T13:57:49Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440360 wikitext text/x-wiki ktl2zt78ydjuiuwg8aaoutwgwp8tse8 2440361 2440360 2026-07-23T13:58:06Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1071 2440361 wikitext text/x-wiki dkis28u7v1gx7uj1zpyhv3ofbzua8xq 2440362 2440361 2026-07-23T13:59:01Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2440362 wikitext text/x-wiki 8nnyybqjs0s4i8tvisbaxpngiwvydyp 2440363 2440362 2026-07-23T14:06:48Z Stashbot 7414 elukey: upload python3-docker-report 0.0.19 to apt.wikimedia.org for bookworm and trixie 2440363 wikitext text/x-wiki d34754zzkzhpf5980wutu3kyvb44ogq 2440364 2440363 2026-07-23T14:07:31Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: lsw1-b8-codfw JunOS upgrade 2440364 wikitext text/x-wiki fd82lknqzjxdvw8bjkaq6auaj16wj3s 2440365 2440364 2026-07-23T14:10:04Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b8-codfw,lsw1-b8-codfw IPv6,lsw1-b8-codfw.mgmt,ssw1-a[1,8]-codfw with reason: lsw1-b8-codfw JunOS upgrade 2440365 wikitext text/x-wiki 3x83zd0gjpskux3unksikwbr8zk0i5r 2440366 2440365 2026-07-23T14:10:39Z Stashbot 7414 topranks: depool dns2004 before lsw1-b8-codfw switch maintenance T430929 2440366 wikitext text/x-wiki mkodp3tmy0x464b31bg6hixhjzy21ho 2440367 2440366 2026-07-23T14:12:52Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2440367 wikitext text/x-wiki 99eeqensxypmh05xuknuqdobh5n3q1t 2440368 2440367 2026-07-23T14:13:00Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s) 2440368 wikitext text/x-wiki oszwflnpyg8xqks1mijzzkp4j9382rb 2440369 2440368 2026-07-23T14:14:36Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2008.codfw.wmnet with OS bookworm 2440369 wikitext text/x-wiki hhee99c9e26ommtwq7zt515c37j53ob 2440370 2440369 2026-07-23T14:14:41Z Stashbot 7414 cmooney@cumin1003: conftool action : set/pooled=no; selector: name=dns2004.wikimedia.org 2440370 wikitext text/x-wiki 63eh08rs39oo37nkomahyll76yzgqi5 2440371 2440370 2026-07-23T14:14:57Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2008 2440371 wikitext text/x-wiki d361dxc96zi9aeem4ogx2r52oypic7m 2440372 2440371 2026-07-23T14:14:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2008 2440372 wikitext text/x-wiki dhrz636871mc3ad4l3g1wrb4icg1k2b 2440373 2440372 2026-07-23T14:16:05Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2035-2036,2087-2090,2286-2291].codfw.wmnet 2440373 wikitext text/x-wiki 89tt5z7qtz17doaq01iu31kmnbzo63w 2440374 2440373 2026-07-23T14:17:41Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2050.codfw.wmnet 2440374 wikitext text/x-wiki 3w9srrc3rbiez0mgexc1esom0n6tx9h 2440375 2440374 2026-07-23T14:18:12Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1014.eqiad.wmnet with OS bookworm 2440375 wikitext text/x-wiki cxbg8xw5zyblewr7dxaol24i4rscjfx 2440376 2440375 2026-07-23T14:19:22Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool db2163: codfw rack B8 depool for maintenance 2440376 wikitext text/x-wiki d9r2incnawu7an9jodvg8z4wxac1u1s 2440377 2440376 2026-07-23T14:19:40Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1014 2440377 wikitext text/x-wiki 2dk7r41ikbt073nzq27cqb84mmj1pvj 2440378 2440377 2026-07-23T14:19:46Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2440378 wikitext text/x-wiki c15uwr0davnns99n6dy6heasx9jxgdw 2440379 2440378 2026-07-23T14:19:56Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: codfw rack B8 depool for maintenance 2440379 wikitext text/x-wiki dgf9esq93gbi8y5ukazvt6kzpr6hmqj 2440380 2440379 2026-07-23T14:20:01Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool db2164: codfw rack B8 depool for maintenance 2440380 wikitext text/x-wiki rf55jq68noqqwflyo1fpnv0tehttq9r 2440381 2440380 2026-07-23T14:20:26Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2164: codfw rack B8 depool for maintenance 2440381 wikitext text/x-wiki 3xbmblmwv1t0ag49har9poqx65cdyyn 2440382 2440381 2026-07-23T14:20:30Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool db2189: codfw rack B8 depool for maintenance 2440382 wikitext text/x-wiki czxgm6xg4zabwwiepvx1baiftksqtez 2440383 2440382 2026-07-23T14:20:44Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2050.codfw.wmnet 2440383 wikitext text/x-wiki 3z0dvwsf721d5hc3ni2xpyb4et14nvl 2440384 2440383 2026-07-23T14:20:46Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet 2440384 wikitext text/x-wiki mbfr9nq04suzcarei2ngffafopaz6xp 2440385 2440384 2026-07-23T14:21:25Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: codfw rack B8 depool for maintenance 2440385 wikitext text/x-wiki pyrfunl17ckzeh5hs92actvb6dbxy9j 2440386 2440385 2026-07-23T14:21:29Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool db2249: codfw rack B8 depool for maintenance 2440386 wikitext text/x-wiki 5g0cwkqmojv3gf9asvhw7ggfrgrjn1j 2440387 2440386 2026-07-23T14:21:33Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1071 - jiji@cumin1003" 2440387 wikitext text/x-wiki 2q0jjsed36py6yctilr32ne4m5k0kth 2440388 2440387 2026-07-23T14:21:39Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1071 - jiji@cumin1003" 2440388 wikitext text/x-wiki 11cvsg0g2yklqcghsmy6t9xcz9myjff 2440389 2440388 2026-07-23T14:21:40Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440389 wikitext text/x-wiki d6prbpkdllu6b4pgmb0p2yvw7t5yqt9 2440390 2440389 2026-07-23T14:21:42Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1071.eqiad.wmnet 166.48.64.10.in-addr.arpa 6.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440390 wikitext text/x-wiki hnjqctah36dohi0ays0hemzyqpqtqy7 2440391 2440390 2026-07-23T14:21:43Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1071.eqiad.wmnet 166.48.64.10.in-addr.arpa 6.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440391 wikitext text/x-wiki 3pfupk4huqehqz6xo6gm1572delvi4m 2440392 2440391 2026-07-23T14:21:45Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1071 2440392 wikitext text/x-wiki kko9btv32eqqav654kusljbe5op155c 2440393 2440392 2026-07-23T14:21:46Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2249: codfw rack B8 depool for maintenance 2440393 wikitext text/x-wiki 57l9iqaxadm996loxiatca5gc4bgz0h 2440394 2440393 2026-07-23T14:21:48Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.depool depool es2052: codfw rack B8 depool for maintenance 2440394 wikitext text/x-wiki 9j1zz3hfsx0xgqzfed6zsiroz7m03uv 2440395 2440394 2026-07-23T14:22:02Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2052: codfw rack B8 depool for maintenance 2440395 wikitext text/x-wiki pm8gcvzv8cin7l6s8cx2vgc0bwb37vc 2440396 2440395 2026-07-23T14:22:35Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore scholarly data after Bookworm reimage) xfer scholarly_articles from wdqs2024.codfw.wmnet -> wdqs2016.codfw.wmnet, repooling source-only afterwards 2440396 wikitext text/x-wiki 4h2ui66kbflp2v5312bc4zmj13cpw4f 2440397 2440396 2026-07-23T14:23:24Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2035-2036,2087-2090,2286-2291].codfw.wmnet 2440397 wikitext text/x-wiki 41ypfjlktks0e2minzlluo1g8solh50 2440398 2440397 2026-07-23T14:23:25Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1071 2440398 wikitext text/x-wiki 0vr20em6lqvr0rrva41a4ejbvux0qbw 2440399 2440398 2026-07-23T14:23:27Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1071 2440399 wikitext text/x-wiki 04af1kq1lgj5p3fvwkep2mvxfgt84vb 2440400 2440399 2026-07-23T14:25:48Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet 2440400 wikitext text/x-wiki houwo2uku3cg6fseqnyp57okppch68e 2440401 2440400 2026-07-23T14:30:30Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1001.eqiad.wmnet with OS trixie 2440401 wikitext text/x-wiki nabx6yk5bvedwq4j59kf1mm7qlobmh5 2440402 2440401 2026-07-23T14:37:52Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2231: Maintenance 2440402 wikitext text/x-wiki g4op8y1ebpnlsjvhcjw2jg3k47o3y15 2440403 2440402 2026-07-23T14:38:52Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2008.codfw.wmnet with reason: host reimage 2440403 wikitext text/x-wiki c1vban9lfv1iayoxtt4m2p7oid33c8b 2440404 2440403 2026-07-23T14:41:51Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1071.eqiad.wmnet with reason: host reimage 2440404 wikitext text/x-wiki bqq0omcrbzo6rr3b3cvopt61ahfle6j 2440405 2440404 2026-07-23T14:43:19Z Stashbot 7414 topranks: reboot lsw1-b8-codw to upgrade JunOS T430929 2440405 wikitext text/x-wiki qflk42wttpds7hszi71mi104ysb6css 2440406 2440405 2026-07-23T14:44:02Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2008.codfw.wmnet with reason: host reimage 2440406 wikitext text/x-wiki l0bu8rtt3oayslzcpu4iy40qrnqnzx6 2440407 2440406 2026-07-23T14:47:08Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1001.eqiad.wmnet with reason: host reimage 2440407 wikitext text/x-wiki gxwyapbf93qon9zrt07o4ffzmzbztlm 2440408 2440407 2026-07-23T14:48:06Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1071.eqiad.wmnet with reason: host reimage 2440408 wikitext text/x-wiki s9111zgwmnhjfayzn7f4q3yece0u7dm 2440409 2440408 2026-07-23T14:51:29Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1001.eqiad.wmnet with reason: host reimage 2440409 wikitext text/x-wiki 5bnbtczesxr154cuefi205xr70jktvi 2440410 2440409 2026-07-23T14:51:55Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2016.codfw.wmnet 2440410 wikitext text/x-wiki 254kto7czugi8m5ze4orv3dijv6k4b6 2440411 2440410 2026-07-23T14:56:07Z Stashbot 7414 ryankemper: [WDQS] T430880 Reimaged `wdqs2016` to Bookworm, xferred scholarly_articles from `wdqs2024`, validated updater/readiness/federation, and repooled 2440411 wikitext text/x-wiki dggzcdt6kr95vq3bwkcgm0j78a49tcy 2440412 2440411 2026-07-23T14:57:06Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test2001.codfw.wmnet 2440412 wikitext text/x-wiki eouisj4fd35t6bu1l4314gto3obx6z3 2440413 2440412 2026-07-23T14:59:23Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test2001.codfw.wmnet 2440413 wikitext text/x-wiki i0ubqzyvjxf8cbygevrn38wkcgvenjw 2440414 2440413 2026-07-23T15:01:30Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2035-2036,2087-2090,2286-2291].codfw.wmnet 2440414 wikitext text/x-wiki kfb902vosdks9dav9zqc21towp8xp09 2440415 2440414 2026-07-23T15:01:38Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2035-2036,2087-2090,2286-2291].codfw.wmnet 2440415 wikitext text/x-wiki p35wbp9m93idaooxlj5kapju0izl54c 2440416 2440415 2026-07-23T15:02:30Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.pool pool db2163: codfw rack B8 re-pool after maintenance 2440416 wikitext text/x-wiki n9wcwwlswj1p7bsmwdshlgizdwz8nja 2440417 2440416 2026-07-23T15:02:49Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet 2440417 wikitext text/x-wiki 00g0ozfvvgecp9rulazz31pf4ermccm 2440418 2440417 2026-07-23T15:02:50Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet 2440418 wikitext text/x-wiki spxcixvctp9khuvswvgvfm3au7djdyn 2440419 2440418 2026-07-23T15:06:22Z Stashbot 7414 cmooney@cumin1003: conftool action : set/pooled=yes; selector: name=dns1004.wikimedia.org 2440419 wikitext text/x-wiki 8tc5hk010obwifxwh8dz3p9k5zxvhwj 2440421 2440419 2026-07-23T15:08:02Z Stashbot 7414 brouberol@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2440421 wikitext text/x-wiki lirzhhmxvpoyel5860ejuuszdsbrr8z 2440422 2440421 2026-07-23T15:08:06Z Stashbot 7414 brouberol@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2440422 wikitext text/x-wiki 2oae3ttk3jdms6c66mkrdv2un0v76jx 2440423 2440422 2026-07-23T15:08:34Z Stashbot 7414 brouberol@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2440423 wikitext text/x-wiki f9gg3vfs1tqr0m6e8j1dpqdysuqdn87 2440424 2440423 2026-07-23T15:08:41Z Stashbot 7414 brouberol@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2440424 wikitext text/x-wiki 80l15ukkin440923rv4t6ulgormuuwr 2440425 2440424 2026-07-23T15:10:43Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2249.codfw.wmnet with reason: Maintenance 2440425 wikitext text/x-wiki b434izfa8ctmlkht8nluatxixqvykx4 2440426 2440425 2026-07-23T15:10:56Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host wdqs2008.codfw.wmnet with OS bookworm 2440426 wikitext text/x-wiki cok1hvkc2h80j22epqd884k81lw4opi 2440427 2440426 2026-07-23T15:11:47Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host wikikube-worker1071.eqiad.wmnet with OS trixie 2440427 wikitext text/x-wiki gtohjsi1dkh4mkk3b7t7ih03ion9292 2440428 2440427 2026-07-23T15:11:49Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1071.eqiad.wmnet 2440428 wikitext text/x-wiki 7lu3xa753qa7zfkz1hp0s4m0uykkk84 2440429 2440428 2026-07-23T15:12:21Z Stashbot 7414 klausman@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host ml-serve1001.eqiad.wmnet with OS trixie 2440429 wikitext text/x-wiki 95yobzz83tw4zt8atyhj0rh8gcv5hg1 2440430 2440429 2026-07-23T15:12:49Z Stashbot 7414 brouberol@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2440430 wikitext text/x-wiki f48ub74ph9b7jbwvyq7sgq36hpmflsm 2440431 2440430 2026-07-23T15:12:54Z Stashbot 7414 brouberol@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2440431 wikitext text/x-wiki g5e43h0rfi9n4ygw1vscme9tnw0vcd9 2440432 2440431 2026-07-23T15:14:25Z Stashbot 7414 cmooney@cumin1003: conftool action : set/pooled=yes; selector: name=dns2004.wikimedia.org 2440432 wikitext text/x-wiki of5yuntcerqpzfzoy5pn88s07hli1rd 2440433 2440432 2026-07-23T15:17:36Z Stashbot 7414 cmooney@dns2004: START - running authdns-update 2440433 wikitext text/x-wiki sv0h43bfvimkrlwi0gbn89yde4agye6 2440434 2440433 2026-07-23T15:19:36Z Stashbot 7414 cmooney@dns2004: END - running authdns-update 2440434 wikitext text/x-wiki 4n15n88805le6esnub48w2efpptj4os 2440435 2440434 2026-07-23T15:20:20Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2440435 wikitext text/x-wiki gz85r69z6k82vwdwgqyfp7t64ymejh2 2440436 2440435 2026-07-23T15:21:03Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2440436 wikitext text/x-wiki qxgv00j6zgmfs5kln10obvgffi9ih8g 2440437 2440436 2026-07-23T15:21:09Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2440437 wikitext text/x-wiki gntf92i5mltgkqsdo9cuksf1g148swl 2440438 2440437 2026-07-23T15:21:14Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440438 wikitext text/x-wiki p3uqbnzjmdf20qvl8yrvz5s04ftm14o 2440439 2440438 2026-07-23T15:21:26Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2440439 wikitext text/x-wiki ewljdb5pz26mw5ylf1u0gz0hux52u7a 2440440 2440439 2026-07-23T15:26:40Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2440440 wikitext text/x-wiki t5a7qyges5hpq2kqpforkf7s23s2anp 2440441 2440440 2026-07-23T15:29:05Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2440441 wikitext text/x-wiki kno0gjj9jttaux1ww1cz7i8ml4crqwz 2440442 2440441 2026-07-23T15:29:38Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2440442 wikitext text/x-wiki qnzkvegnudjir8p2ossfc2rzjmnis2d 2440443 2440442 2026-07-23T15:30:09Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply 2440443 wikitext text/x-wiki adaf0i9txcijzdd1gkfbsvmy9d63a7q 2440444 2440443 2026-07-23T15:30:11Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2440444 wikitext text/x-wiki rtkgvvj0zsrzwuw0cc97qbhmxa0m4zi 2440445 2440444 2026-07-23T15:30:17Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply 2440445 wikitext text/x-wiki d9ip4wf7j2zp2ja7fnjic069slw6lit 2440446 2440445 2026-07-23T15:30:54Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2440446 wikitext text/x-wiki pb9d10u49shngxes67fpzk2dba20czt 2440448 2440446 2026-07-23T15:33:16Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2249: Maintenance 2440448 wikitext text/x-wiki ila93rqt6vucwhfwf10maxqoz559uzl 2440449 2440448 2026-07-23T15:34:04Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: push any outstanding updates - cmooney@cumin1003" 2440449 wikitext text/x-wiki gm7yuuhsm5xyxp9wby2wd5m0ewhydaa 2440450 2440449 2026-07-23T15:34:09Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: push any outstanding updates - cmooney@cumin1003" 2440450 wikitext text/x-wiki 7wox4vlolq0emvimigi32miakag23g2 2440451 2440450 2026-07-23T15:34:11Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440451 wikitext text/x-wiki 2bijcps4am2721z3v0d9ep8jdahy5l8 2440452 2440451 2026-07-23T15:35:34Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1072.eqiad.wmnet 2440452 wikitext text/x-wiki c8u5l5sq1hulovd7qibzemtawobbjqp 2440453 2440452 2026-07-23T15:35:38Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet 2440453 wikitext text/x-wiki qc6dvgwffgvg8ym1i3puogpqgxs777h 2440454 2440453 2026-07-23T15:36:10Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet 2440454 wikitext text/x-wiki 9e95k01fh5wtwnf16tabrpuaf1e1395 2440455 2440454 2026-07-23T15:41:26Z Stashbot 7414 musikanimal@deploy1003: Started scap sync-world: Backport for [[gerrit:1314861|Util: Cast numeric config strings to their declared types (T432910)]] 2440455 wikitext text/x-wiki pn0rdjpkorc6h901oyet3y65p7w7cbr 2440456 2440455 2026-07-23T15:43:20Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Backport for [[gerrit:1314861|Util: Cast numeric config strings to their declared types (T432910)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2440456 wikitext text/x-wiki lcehe0tc8tw9dgl9f8ubtn3ggvdk3f7 2440457 2440456 2026-07-23T15:43:49Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Continuing with deployment 2440457 wikitext text/x-wiki 7axwmv9gs1j75676sax13w2dh9sjz0v 2440458 2440457 2026-07-23T15:45:15Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2440458 wikitext text/x-wiki l2xo2ls8g72ay5qg3bbgeqau0k0o2vs 2440459 2440458 2026-07-23T15:48:00Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: codfw rack B8 re-pool after maintenance 2440459 wikitext text/x-wiki npqiu84mq8xp3fg9x9qu8tgkxfylfsq 2440460 2440459 2026-07-23T15:48:04Z Stashbot 7414 musikanimal@deploy1003: Finished scap sync-world: Backport for [[gerrit:1314861|Util: Cast numeric config strings to their declared types (T432910)]] (duration: 06m 37s) 2440460 wikitext text/x-wiki h5mqk8prlyczrdlnetq5bkz1bmttk1e 2440461 2440460 2026-07-23T15:48:06Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.pool pool db2164: codfw rack B8 re-pool after maintenance 2440461 wikitext text/x-wiki a48xnib2k98xl9383457qdb7y1z9hn6 2440462 2440461 2026-07-23T15:49:46Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1072.eqiad.wmnet with OS trixie 2440462 wikitext text/x-wiki gk1du2wfdnnan18vwdfmxe5zcaawir9 2440463 2440462 2026-07-23T15:50:06Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1072 2440463 wikitext text/x-wiki 26e4954zyizq7yf4z1iy3ysi9xnfslm 2440464 2440463 2026-07-23T15:50:07Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1072 2440464 wikitext text/x-wiki 7044n9akk1vv0vge5eeo14v6n5bpol5 2440465 2440464 2026-07-23T16:06:37Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1072.eqiad.wmnet with reason: host reimage 2440465 wikitext text/x-wiki igt4thfivg82xsjvn9fykckusmrftl8 2440466 2440465 2026-07-23T16:06:41Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1072.eqiad.wmnet with reason: host reimage 2440466 wikitext text/x-wiki rj9h5qi6vbi7gzcgd2eohew7x4wgtu4 2440467 2440466 2026-07-23T16:18:44Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2249: Maintenance 2440467 wikitext text/x-wiki khf337oxgtvjsskjjx16xnjl6pdt3km 2440468 2440467 2026-07-23T16:27:59Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host wikikube-worker1072.eqiad.wmnet with OS trixie 2440468 wikitext text/x-wiki 3la071v5ntdk9axey4sbq424aw2ni9b 2440469 2440468 2026-07-23T16:28:01Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1072.eqiad.wmnet 2440469 wikitext text/x-wiki 32u1s8oxy93iac5nmgemza0jp32qkky 2440470 2440469 2026-07-23T16:33:21Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: codfw rack B8 re-pool after maintenance 2440470 wikitext text/x-wiki 5svfhjpr6dishzsw7yosrikcby98qwc 2440471 2440470 2026-07-23T16:33:25Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.pool pool db2189: codfw rack B8 re-pool after maintenance 2440471 wikitext text/x-wiki tr3kvtrnkhw8wn3fuap4s4bftao89px 2440472 2440471 2026-07-23T16:35:11Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 138128 2440472 wikitext text/x-wiki l0naqx24a2v11ylkm7fdwlfhmf75xmm 2440473 2440472 2026-07-23T16:36:27Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 138128 2440473 wikitext text/x-wiki 16djyq3of5da98w3lyx265j82acm0kx 2440479 2440473 2026-07-23T16:58:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 T432986 2440479 wikitext text/x-wiki tta3iosl4q7ygcobolkjycjk43bem6g 2440480 2440479 2026-07-23T16:58:35Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2191 with weight 0 T432986', diff saved to https://phabricator.wikimedia.org/P95123 and previous config saved to /var/cache/conftool/dbconfig/20260723-165831-cwilliams.json 2440480 wikitext text/x-wiki 9e5z7ucyaa61bv7yyynw98ngrg8t8qm 2440482 2440480 2026-07-23T17:05:07Z Stashbot 7414 cezmunsta: Starting x1 codfw failover from db2215 to db2191 - T432986 2440482 wikitext text/x-wiki 35u7s62r39nmustu09xx48ul9ypmf81 2440483 2440482 2026-07-23T17:06:15Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2191 to x1 primary T432986', diff saved to https://phabricator.wikimedia.org/P95125 and previous config saved to /var/cache/conftool/dbconfig/20260723-170612-cwilliams.json 2440483 wikitext text/x-wiki msfop88lsfnxom2wvs1yg9wszug3rnf 2440484 2440483 2026-07-23T17:09:06Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2215 T432986', diff saved to https://phabricator.wikimedia.org/P95126 and previous config saved to /var/cache/conftool/dbconfig/20260723-170903-cwilliams.json 2440484 wikitext text/x-wiki f8ty5s8yzhzsyafniyahwpqbez188ch 2440486 2440484 2026-07-23T17:17:56Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2440486 wikitext text/x-wiki cmite0bb0c96pmk5llla2fmanwwmji1 2440487 2440486 2026-07-23T17:18:27Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2215.codfw.wmnet with reason: Maintenance 2440487 wikitext text/x-wiki kl90ggt877ja7hize90kjpk55gebk27 2440488 2440487 2026-07-23T17:18:42Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: codfw rack B8 re-pool after maintenance 2440488 wikitext text/x-wiki 2knpd01kh3t5lla8ipcmuyg6hbz1otz 2440489 2440488 2026-07-23T17:18:46Z Stashbot 7414 cmooney@cumin2003: START - Cookbook sre.mysql.pool pool es2052: codfw rack B8 re-pool after maintenance 2440489 wikitext text/x-wiki ow7e3le8t7kci6u0vmiyukt16rhuama 2440490 2440489 2026-07-23T17:21:03Z Stashbot 7414 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply 2440490 wikitext text/x-wiki 05xvab7fuudr5wpsov9z1czke5q625c 2440491 2440490 2026-07-23T17:21:21Z Stashbot 7414 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply 2440491 wikitext text/x-wiki b78p17679nqyvj5rniq1g2qybd82m2x 2440492 2440491 2026-07-23T17:21:26Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply 2440492 wikitext text/x-wiki 4gb62wylkt7m3swaqgzma1cqb6daz12 2440493 2440492 2026-07-23T17:21:44Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply 2440493 wikitext text/x-wiki pa9uw1lyvx063gq1v6o2hxftielk5ts 2440494 2440493 2026-07-23T17:21:56Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply 2440494 wikitext text/x-wiki sp2tfsluuyue6o0wx7e1ushr1p1gkfn 2440495 2440494 2026-07-23T17:22:24Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply 2440495 wikitext text/x-wiki scm6coq0s583k8niheni5d66ns4fpov 2440496 2440495 2026-07-23T17:23:07Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new entries for row a b spines - cmooney@cumin1003" 2440496 wikitext text/x-wiki pko95a1tkkvtc8p5sg1sc1crlx322rr 2440498 2440496 2026-07-23T17:27:05Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2440498 wikitext text/x-wiki 7h5eqzg70ms3dmwjhvpxafjonpmvcis 2440499 2440498 2026-07-23T17:29:12Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2440499 wikitext text/x-wiki que59s0f73d8kodvnv1fvmz90helwkr 2440500 2440499 2026-07-23T17:32:22Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2215: Maintenance 2440500 wikitext text/x-wiki ij3wmx4s7ixx1h1dfpvculgh1kvi0ep 2440503 2440500 2026-07-23T17:54:40Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new entries for row a b spines - cmooney@cumin1003" 2440503 wikitext text/x-wiki hhkmolnf09haec7014wbr8hxt6nrixf 2440504 2440503 2026-07-23T17:54:41Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440504 wikitext text/x-wiki 6a3ks0p6bxaggtwy868h8zd2agaoi9x 2440505 2440504 2026-07-23T17:54:48Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2440505 wikitext text/x-wiki kvfk3m4r8q3is2zind7ke9zs7y29s2o 2440506 2440505 2026-07-23T18:04:04Z Stashbot 7414 cmooney@cumin2003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2052: codfw rack B8 re-pool after maintenance 2440506 wikitext text/x-wiki jda3mc2vyfdmdq6oew7txk29u3mwsqn 2440507 2440506 2026-07-23T18:05:46Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new entries for row a b spines - cmooney@cumin1003" 2440507 wikitext text/x-wiki r6t52v9d7ug52u4ztt0ozql0w5z0qvh 2440508 2440507 2026-07-23T18:06:08Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new entries for row a b spines - cmooney@cumin1003" 2440508 wikitext text/x-wiki otk056eej54hvi7o1x8jwv8q3j04sm0 2440509 2440508 2026-07-23T18:06:09Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440509 wikitext text/x-wiki 2coghc09gy299glja8q14elsuceko0b 2440510 2440509 2026-07-23T18:15:24Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2440510 wikitext text/x-wiki bbcy3f6pze8ykbjgczflzvhvl16j1j9 2440511 2440510 2026-07-23T18:15:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2440511 wikitext text/x-wiki 7eo3eamc9cz8gs6749od87t516sehfv 2440512 2440511 2026-07-23T18:18:56Z Stashbot 7414 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.12 refs T430831 2440512 wikitext text/x-wiki 4p82owenwib0zqly9a9l2j4nrqd3lr5 2440514 2440512 2026-07-23T18:21:19Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2215: Maintenance 2440514 wikitext text/x-wiki rml30ga8m1wykrcdowb0ntu70iescgm 2440515 2440514 2026-07-23T18:22:00Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1014 - bking@cumin2003" 2440515 wikitext text/x-wiki 19kb2e5haiwwg79gjelojs9zdj1j19x 2440516 2440515 2026-07-23T18:22:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1014 - bking@cumin2003" 2440516 wikitext text/x-wiki n4ag0cll53wmqgb1j9ja3hficdreqvc 2440517 2440516 2026-07-23T18:22:06Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440517 wikitext text/x-wiki a9z3tpelucpskcqeptssszw0klwtv00 2440518 2440517 2026-07-23T18:22:07Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1014.eqiad.wmnet 188.32.64.10.in-addr.arpa 8.8.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440518 wikitext text/x-wiki gmietcccj5807jeism8gwsdc1evoec7 2440519 2440518 2026-07-23T18:22:09Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1014.eqiad.wmnet 188.32.64.10.in-addr.arpa 8.8.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2440519 wikitext text/x-wiki esldgwqzjjm37zja9mrvrtfdjrkoy8s 2440520 2440519 2026-07-23T18:22:10Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1014 2440520 wikitext text/x-wiki d95j34548x2nkp9556ltm9iagm0jw7x 2440521 2440520 2026-07-23T18:22:42Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1014 2440521 wikitext text/x-wiki hzsxsc1rrsfbdnwda3nvd4v3povp1kn 2440522 2440521 2026-07-23T18:22:44Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1014 2440522 wikitext text/x-wiki 8crc8qadja628ueh2dds4w7ajxyfgak 2440523 2440522 2026-07-23T18:28:49Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2440523 wikitext text/x-wiki 8qyf8wxajwyzb0nrbry9l5yvix5zh1w 2440524 2440523 2026-07-23T18:28:58Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 19s) 2440524 wikitext text/x-wiki h3t0b4sjhq3k7fbq3gidbz7roil9eao 2440525 2440524 2026-07-23T18:30:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling source-only afterwards 2440525 wikitext text/x-wiki fbpwf9qv6znvsdakpcrhuxw86moariu 2440526 2440525 2026-07-23T18:38:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1014.eqiad.wmnet with reason: host reimage 2440526 wikitext text/x-wiki 6ogq2bp6r3npet2itoh28zcsy7s360u 2440529 2440526 2026-07-23T18:44:22Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1014.eqiad.wmnet with reason: host reimage 2440529 wikitext text/x-wiki beouqjpg8fe1wdawf9j61w1gi9jmonn 2440532 2440529 2026-07-23T19:04:03Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.noop 2440532 wikitext text/x-wiki a65w556y29v0qvnygdenlnm9z5tj1x9 2440533 2440532 2026-07-23T19:04:13Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.noop (exit_code=99) 2440533 wikitext text/x-wiki 26xzd4lnn4abelmaxmc0nb8ea2e998z 2440534 2440533 2026-07-23T19:05:55Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host wdqs1014.eqiad.wmnet with OS bookworm 2440534 wikitext text/x-wiki n7bovyztijpm0tu67um9nrbj2d33av8 2440536 2440534 2026-07-23T19:14:39Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2440536 wikitext text/x-wiki e56gq7xce3rcwu8gcm5kew4dhuz2qww 2440537 2440536 2026-07-23T19:24:24Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating the rest of the ipv6 dns records. - jhancock@cumin2002" 2440537 wikitext text/x-wiki 7fnrf6zz38c5tplnofv9k4w45jvgeh1 2440538 2440537 2026-07-23T19:24:29Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating the rest of the ipv6 dns records. - jhancock@cumin2002" 2440538 wikitext text/x-wiki bwe0e7ldfidu4jmh80qfcs508xzcepl 2440539 2440538 2026-07-23T19:24:31Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2440539 wikitext text/x-wiki mv1eqodm0lu58bfd45z543xkpddkiqu 2440542 2440539 2026-07-23T20:05:19Z Stashbot 7414 dani@deploy1003: Started scap sync-world: Backport for [[gerrit:1314951|Undeploy Referring Experiences survey on enwiki (T432289)]] 2440542 wikitext text/x-wiki tqnyv2t9d52ijtbf7094tz4h4fthvf6 2440544 2440542 2026-07-23T20:07:06Z Stashbot 7414 dani@deploy1003: dani: Backport for [[gerrit:1314951|Undeploy Referring Experiences survey on enwiki (T432289)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2440544 wikitext text/x-wiki jb7w8owpktkc9krc4etdff43ixf74z8 2440545 2440544 2026-07-23T20:13:01Z Stashbot 7414 dani@deploy1003: dani: Continuing with deployment 2440545 wikitext text/x-wiki 9ix54h02r7gd7sy6x3e6kt9ne9kwv2q 2440546 2440545 2026-07-23T20:17:16Z Stashbot 7414 dani@deploy1003: Finished scap sync-world: Backport for [[gerrit:1314951|Undeploy Referring Experiences survey on enwiki (T432289)]] (duration: 11m 57s) 2440546 wikitext text/x-wiki 3db2lv4s8tdxmcoy9x0my2naapudz3k 2440547 2440546 2026-07-23T21:00:09Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:cassandra-dev: Setting storage compatibility to UPGRADING - eevans@cumin1003 2440547 wikitext text/x-wiki sesa6n83yajn8erurlplimidylx78v6 2440548 2440547 2026-07-23T21:19:39Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:cassandra-dev: Setting storage compatibility to UPGRADING - eevans@cumin1003 2440548 wikitext text/x-wiki k9nxkr0xeihbs1t2oq0mk1gq3v7bsnm 2440549 2440548 2026-07-23T21:28:18Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2440549 wikitext text/x-wiki qbpo0gkblfrgidwe547x1iu6ot27es4 2440550 2440549 2026-07-23T21:28:25Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 46s) 2440550 wikitext text/x-wiki bdf541t0h7a9d41gsg4o5q1mpxn60wu 2440551 2440550 2026-07-23T21:29:33Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1011.eqiad.wmnet -> wdqs1014.eqiad.wmnet, repooling source-only afterwards 2440551 wikitext text/x-wiki 7xqk9dqyiowoenziaw1fcky21h7wzjp 2440552 2440551 2026-07-23T22:01:02Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:cassandra-dev: Setting storage compatibility to NONE - eevans@cumin1003 2440552 wikitext text/x-wiki j0timmwbk630863hruii121d7fssx57 2440553 2440552 2026-07-23T22:21:05Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:cassandra-dev: Setting storage compatibility to NONE - eevans@cumin1003 2440553 wikitext text/x-wiki se0aevg6l8296axh4duib613mjf3gjt 2440554 2440553 2026-07-23T23:27:15Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mr1-eqsin.oob,mr1-eqsin.oob IPv6 with reason: switch refresh 2440554 wikitext text/x-wiki 7yaus845jg4jd6iozv6eishqrt20lx3 2440557 2440554 2026-07-24T02:01:33Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2440557 wikitext text/x-wiki n5cozqxo016xb8vbw9zev5tql3nfs3r 2440558 2440557 2026-07-24T02:08:01Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 27s) 2440558 wikitext text/x-wiki hppsz513n2zfvza3twvttizt6akp80z 2440559 2440558 2026-07-24T02:38:21Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cr[2-3]-eqsin.mgmt,ps1-[603-604]-eqsin with reason: connection issue 2440559 wikitext text/x-wiki smrbr38t8fwrpawgsm7co1nmx8fehab 2440560 2440559 2026-07-24T02:49:43Z Stashbot 7414 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mr1-eqsin,mr1-eqsin IPv6 with reason: connection issue 2440560 wikitext text/x-wiki d4bnc3gqoltcfafu9wnjzg1mgyyxpdi 2440561 2440560 2026-07-24T03:36:49Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2440561 wikitext text/x-wiki 8kmuz1breo1t1cv8b2cc5j4zanf464o 2440562 2440561 2026-07-24T03:37:22Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2440562 wikitext text/x-wiki cbepv0l941xc3xqev4ya0q3y706abnd 2440563 2440562 2026-07-24T03:37:24Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2440563 wikitext text/x-wiki fkvezik5uw3b43pqk0xiuzt5udvbv9q 2440564 2440563 2026-07-24T03:38:00Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2440564 wikitext text/x-wiki efu5lonp0lu0yhpqb1tmu9yalifps6c 2440565 2440564 2026-07-24T03:40:33Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2440565 wikitext text/x-wiki f8umgrzh9lukxjti4ynkb0b43hw3w73 2440566 2440565 2026-07-24T03:40:35Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2440566 wikitext text/x-wiki h6bqg9v5cyu1caki06xrrhtjwd7alzo 2440567 2440566 2026-07-24T03:40:36Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2440567 wikitext text/x-wiki hg9u8djcmq3ogyje9r93ro5axd3snew 2440568 2440567 2026-07-24T03:40:38Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2440568 wikitext text/x-wiki c4tnuf5edg304dzjb707ntu6rz6gqvn 2440570 2440568 2026-07-24T06:44:36Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet,service=s4 2440570 wikitext text/x-wiki tofq5skwhlomawm78471ru4iaqt9ph5 2440571 2440570 2026-07-24T06:44:40Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet,service=s6 2440571 wikitext text/x-wiki kynk8w3yeqrwkcxtba3axe252w5veod 2440572 2440571 2026-07-24T06:45:59Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1025.eqiad.wmnet,service=s6 2440572 wikitext text/x-wiki ft5e8kz60c1we0jiccq33x15jfsjh0u 2440573 2440572 2026-07-24T06:46:03Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1025.eqiad.wmnet,service=s4 2440573 wikitext text/x-wiki qts1e6y5zhgqlin8c028d6wz9iktw0b 2440574 2440573 2026-07-24T06:46:31Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1025.eqiad.wmnet with reason: Cloning 2440574 wikitext text/x-wiki pw8ho15sf48gqlhkf73hfapfv5ruohi 2440577 2440574 2026-07-24T07:31:20Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2440577 wikitext text/x-wiki 8jjegrt23r2m91am3n4asykycnlr3su 2440578 2440577 2026-07-24T07:31:32Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2440578 wikitext text/x-wiki mzlufyk86qx3xdajoqf151rnc6tpspy 2440579 2440578 2026-07-24T07:32:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2440579 wikitext text/x-wiki evwocgvnsq5051jbj3c6ncbbqkbeff9 2440580 2440579 2026-07-24T07:32:32Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2440580 wikitext text/x-wiki 5grj2lq3nfy7lxyma3t2asq325zfalp 2440581 2440580 2026-07-24T07:57:26Z Stashbot 7414 brouberol@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2440581 wikitext text/x-wiki spmj4z81ejgytxmjr0z1fd5ktkvdbvx 2440582 2440581 2026-07-24T07:57:40Z Stashbot 7414 brouberol@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2440582 wikitext text/x-wiki tf4lfnv4km4vgv9pl6cgikcqhw1h2p5 2440583 2440582 2026-07-24T07:59:58Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2440583 wikitext text/x-wiki 1eqtoi7psy9toetw9u5rh2jq0yb09bl 2440584 2440583 2026-07-24T08:01:12Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2440584 wikitext text/x-wiki joxelnfsr9vecad7piesd7du91tgrtv 2440585 2440584 2026-07-24T08:02:02Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2440585 wikitext text/x-wiki fig860fkxekbdvwbrpcvszso84jwm1s 2440586 2440585 2026-07-24T08:07:41Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2440586 wikitext text/x-wiki 5htbtkmb5oyvs9gy8bf4fg8e13shg5p 2440587 2440586 2026-07-24T08:07:43Z Stashbot 7414 brouberol@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2440587 wikitext text/x-wiki ldeyrd7ihqdpdjd7biz1jv9z66ajwdd 2440588 2440587 2026-07-24T08:07:47Z Stashbot 7414 brouberol@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2440588 wikitext text/x-wiki fhdbp6y5lgw9x117sr23xzp7h6quchb 2440589 2440588 2026-07-24T08:16:25Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2440589 wikitext text/x-wiki dq0qkxjxlw7czhysv87zg642zs2zd6y 2440590 2440589 2026-07-24T08:16:33Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2440590 wikitext text/x-wiki lkzqvbj6kav3in4tsvmnyvzd1i03toa 2440591 2440590 2026-07-24T08:24:19Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2440591 wikitext text/x-wiki 2y4hmttmbjig5yp3kg4bq4sypkhyss8 2440593 2440591 2026-07-24T08:32:20Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2440593 wikitext text/x-wiki oqp3tmxcempt262sjnev5bgvlq6hjs0 2440594 2440593 2026-07-24T09:07:30Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1001.eqiad.wmnet 2440594 wikitext text/x-wiki i8q0m5v3juv243s5evvt65u97le9mkh 2440595 2440594 2026-07-24T09:11:21Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1001.eqiad.wmnet 2440595 wikitext text/x-wiki lrk0snp8zse28toqb385m5phs9uc685 2440596 2440595 2026-07-24T09:11:31Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1002.eqiad.wmnet 2440596 wikitext text/x-wiki 2bllqxgk4hfa3dk8aedoqrq99y4ki5o 2440597 2440596 2026-07-24T09:13:49Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1002.eqiad.wmnet 2440597 wikitext text/x-wiki eprqg8v69k9oj1vnjwp9eldynhn4h3b 2440598 2440597 2026-07-24T09:13:57Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1003.eqiad.wmnet 2440598 wikitext text/x-wiki 7hbdpqogcwyad96et2kw58kjtdf3f3s 2440599 2440598 2026-07-24T09:15:27Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply 2440599 wikitext text/x-wiki 6jpk08gx0ln7r6j5inbcu8wjn8alm49 2440600 2440599 2026-07-24T09:15:53Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply 2440600 wikitext text/x-wiki l54trv4xw15lq14y0xksjwa5zxk0mve 2440601 2440600 2026-07-24T09:16:19Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1003.eqiad.wmnet 2440601 wikitext text/x-wiki ap6g8a4swjugd8svtr4tfxiqyzgvp1i 2440602 2440601 2026-07-24T09:16:30Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2003.codfw.wmnet 2440602 wikitext text/x-wiki 7q3pmkpxalgbq4gcd3smqdvb8iu05ha 2440603 2440602 2026-07-24T09:19:36Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for 16 hosts 2440603 wikitext text/x-wiki 0k8oqkmvz11netp2pylljf8m9ax84bk 2440604 2440603 2026-07-24T09:20:28Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2003.codfw.wmnet 2440604 wikitext text/x-wiki 2o9atymr6neroxxznd7o35blmannpn9 2440605 2440604 2026-07-24T09:21:38Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 57 hosts 2440605 wikitext text/x-wiki fgg85p3oxdlh9nqjzpi0eihmp9wub43 2440606 2440605 2026-07-24T09:22:06Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2002.codfw.wmnet 2440606 wikitext text/x-wiki pm6ukynfg0vzjcuntvuzob554i5ly0v 2440607 2440606 2026-07-24T09:24:31Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2002.codfw.wmnet 2440607 wikitext text/x-wiki psgp0raxejiy2e55l8uxqatc0pw2q8r 2440608 2440607 2026-07-24T09:24:39Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2001.codfw.wmnet 2440608 wikitext text/x-wiki b7a0h56hr7sdpc06cbqaz3s9bpc07ba 2440609 2440608 2026-07-24T09:25:19Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 57 hosts 2440609 wikitext text/x-wiki mugg6pxh01m5o23hhmavqte6b9v9vbg 2440610 2440609 2026-07-24T09:26:03Z Stashbot 7414 brouberol@dns1004: START - running authdns-update 2440610 wikitext text/x-wiki aj2r5rzf6fkvbcosahd8ixig7u2ef8i 2440611 2440610 2026-07-24T09:26:47Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 9 hosts 2440611 wikitext text/x-wiki 94e9dhjzytrtpnae7qwx3omd59w4di8 2440612 2440611 2026-07-24T09:27:05Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2001.codfw.wmnet 2440612 wikitext text/x-wiki sba57dywbvgo5r3vn71vpw8uwigeea3 2440613 2440612 2026-07-24T09:27:16Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2001.codfw.wmnet 2440613 wikitext text/x-wiki cxmmg5l3ue1p8zdqax05msldfz10qp1 2440614 2440613 2026-07-24T09:27:32Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 9 hosts 2440614 wikitext text/x-wiki 9wp29xl2npjrja9pwrr5uzaszbeiz5k 2440615 2440614 2026-07-24T09:29:41Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2001.codfw.wmnet 2440615 wikitext text/x-wiki pome0vlowztm41bcv2z1wlfs4sw69t9 2440616 2440615 2026-07-24T09:29:50Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2002.codfw.wmnet 2440616 wikitext text/x-wiki r7ywmywi134farqv895mqo7icmagrgj 2440617 2440616 2026-07-24T09:30:00Z Stashbot 7414 brouberol@dns1004: END - running authdns-update 2440617 wikitext text/x-wiki ht936p5b4l7m1hux6hxhwt6hrxmk0j2 2440620 2440617 2026-07-24T09:32:22Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2002.codfw.wmnet 2440620 wikitext text/x-wiki aozmvf4hhx3swpntrog1rpcnyatjmn7 2440621 2440620 2026-07-24T09:32:30Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2003.codfw.wmnet 2440621 wikitext text/x-wiki crg437a0100sa3nm9zg4e7djk4a2b5b 2440622 2440621 2026-07-24T09:34:55Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2003.codfw.wmnet 2440622 wikitext text/x-wiki km6ff7yfnnwfylu323ss0a9yk2yc5uz 2440624 2440622 2026-07-24T09:44:27Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1001.eqiad.wmnet 2440624 wikitext text/x-wiki fbim2q3cik0ujbp3azy1tbmfbl840mx 2440625 2440624 2026-07-24T09:47:36Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1001.eqiad.wmnet 2440625 wikitext text/x-wiki mgmstnf32el4i8oaommhqhfk89rij2k 2440626 2440625 2026-07-24T09:47:44Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1002.eqiad.wmnet 2440626 wikitext text/x-wiki sbcqfw3l5c4zpnnhbttlzx3vckmq8gn 2440627 2440626 2026-07-24T09:51:07Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1002.eqiad.wmnet 2440627 wikitext text/x-wiki pmwzs68ieef0zk88nzbgdxu4ps4xz1x 2440628 2440627 2026-07-24T09:51:29Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2002.codfw.wmnet 2440628 wikitext text/x-wiki 8sywrxg1ecevviupr51l9s6hfbt7dru 2440630 2440628 2026-07-24T09:56:27Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2002.codfw.wmnet 2440630 wikitext text/x-wiki k82st7901akrf750wv7rdxc6bf3jl5b 2440631 2440630 2026-07-24T09:57:08Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2002.codfw.wmnet 2440631 wikitext text/x-wiki edgswjtpufp8e5dxxl4zta3gts983u0 2440632 2440631 2026-07-24T10:02:06Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2002.codfw.wmnet 2440632 wikitext text/x-wiki 3b1jb70vf7oxp4rm93io4eqytef9mgd 2440633 2440632 2026-07-24T10:02:22Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2001.codfw.wmnet 2440633 wikitext text/x-wiki t31yy238f5ewx9tw1l2xcvkin0agrq3 2440634 2440633 2026-07-24T10:12:01Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2001.codfw.wmnet 2440634 wikitext text/x-wiki fry08gb44neb0idlu6p1p2eipks63b9 2440635 2440634 2026-07-24T10:13:01Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet 2440635 wikitext text/x-wiki dcs3f3407kbbw4n3gzjt6z6ft6onl17 2440636 2440635 2026-07-24T10:17:59Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2001.codfw.wmnet 2440636 wikitext text/x-wiki cqqo6j9secukl61gdzxusn9eozqrg9t 2440637 2440636 2026-07-24T10:18:50Z Stashbot 7414 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for 16 hosts 2440637 wikitext text/x-wiki 4nzdn71yd0aix4gluo0kqtej4x2ais5 2440638 2440637 2026-07-24T10:19:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2440638 wikitext text/x-wiki 93luv9o3ci4strgr8knuy1rvi0h3s5k 2440639 2440638 2026-07-24T10:24:07Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie 2440639 wikitext text/x-wiki mc0hjxj3szk79bqne81o97f6yc9dwlx 2440640 2440639 2026-07-24T10:30:33Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for 8 hosts 2440640 wikitext text/x-wiki pxj2m5svbezp0azak9rnz5wa7p42uft 2440641 2440640 2026-07-24T10:31:19Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2440641 wikitext text/x-wiki k7fyxo1gi42fqgakgnn7qzsc7j3ieys 2440642 2440641 2026-07-24T10:31:32Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2440642 wikitext text/x-wiki 9itswej7krz71ypp7uys8blyrfrdmiy 2440644 2440642 2026-07-24T10:32:23Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2440644 wikitext text/x-wiki l4t77u5wows6o24jqobi9g9drthwb4u 2440645 2440644 2026-07-24T10:32:37Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2440645 wikitext text/x-wiki h3qtoo6oo05deh47f4nndp85ou6topj 2440646 2440645 2026-07-24T10:33:55Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2440646 wikitext text/x-wiki 6h0otcvv41hq7iz1npz62oq7suvdw7x 2440647 2440646 2026-07-24T10:34:09Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2440647 wikitext text/x-wiki 38p8c950fmip4630ak9ryoo68b73wex 2440648 2440647 2026-07-24T10:35:42Z Stashbot 7414 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for 8 hosts 2440648 wikitext text/x-wiki t8k9irxx47rf0uhey649t15owptlyns 2440649 2440648 2026-07-24T10:41:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2440649 wikitext text/x-wiki qo26pwut2dajiowgvejs03f5dsiz64n 2440650 2440649 2026-07-24T10:41:39Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2440650 wikitext text/x-wiki sn2o38m345dmp5frobcqqn7gbuv3llz 2440651 2440650 2026-07-24T10:44:04Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2440651 wikitext text/x-wiki t70kqpgdr31ordwxsa8npca7usadyjw 2440652 2440651 2026-07-24T10:44:18Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2440652 wikitext text/x-wiki hosltcasly9cmlrezpjdb69jn14g59l 2440653 2440652 2026-07-24T10:47:59Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for 7 hosts 2440653 wikitext text/x-wiki 58ln14a5xnsuy3wzsravpv4nwo13nwe 2440657 2440653 2026-07-24T10:56:59Z Stashbot 7414 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for 7 hosts 2440657 wikitext text/x-wiki 02tbcksq3e1efk3if7vvylm9nbkuq0z 2440661 2440657 2026-07-24T11:11:39Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for 7 hosts 2440661 wikitext text/x-wiki 581helu9846ikfgqzmawys39l63mpwj 2440662 2440661 2026-07-24T11:33:47Z Stashbot 7414 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for 7 hosts 2440662 wikitext text/x-wiki e42s1if7c7kx4o9r5s7rje5b80wajx2 0 17290 2440420 2440112 2026-07-23T15:06:33Z Stashbot 7414 dancy: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/1314124 (T295351) 2440420 wikitext text/x-wiki 3nxf7sew0igve2a8c2ppwzxcjq5jg7u 498 30942 2440331 2440062 2026-07-23T13:44:21Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate 2440331 wikitext text/x-wiki iqbzveblci9mtfvcwt5w7aacsmjc7tp 2440334 2440331 2026-07-23T13:44:44Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate 2440334 wikitext text/x-wiki fnitpqhfpob7x53nbky9z5pyw348x83 2440335 2440334 2026-07-23T13:45:00Z Stashbot 7414 andrewbogott: restarting designate services in eqiad1; seeing many miscellaneous designate-sink errors in logs 2440335 wikitext text/x-wiki qp8fdct69r07iq0b5omvyn3g07od6j2 0 421011 2440474 2382235 2026-07-23T16:44:53Z Paramuch 57956 Add The Planet Thinks — live 3D globe of geo-tagged edits built on the recentchange EventStreams (open source, MIT). Disclosure: I am the author (User:Paramuch). 2440474 wikitext text/x-wiki qkebrchldjoimxtdnv5xy98crs0tf0g 3 443249 2440501 1812159 2026-07-23T17:44:23Z ArielGlenn 11 thanks for the code cleanup 2440501 wikitext text/x-wiki j0lui4aqki3h858mn812diacnjqhasu 498 444707 2440543 2430403 2026-07-23T20:06:34Z Stashbot 7414 mutante: puppetmaster-1003: /srv/git/operations/puppet# sudo -u gitpuppet git pull origin 2440543 wikitext text/x-wiki 23rl5oifbtu2usir291fh1vj4l6rw57 498 446585 2440308 2134795 2026-07-23T12:51:37Z Stashbot 7414 godog: delete tool -- now unused 2440308 wikitext text/x-wiki 7lmnu198oqdvwxwf9izjy2eqxv1cngj 498 446686 2440286 2440244 2026-07-23T12:02:57Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud 2440286 wikitext text/x-wiki r4pg5ur5eaq8ivae2io68rc21i7pejp 2440287 2440286 2026-07-23T12:03:11Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud 2440287 wikitext text/x-wiki sni1ma2owkhn4974a0szdn0fnptqhhq 2440288 2440287 2026-07-23T12:04:06Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud 2440288 wikitext text/x-wiki i2bz8okfl05kr3e9bfy6pa5mjjdd6w3 2440289 2440288 2026-07-23T12:05:36Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud 2440289 wikitext text/x-wiki 58o59y7axpqgpk6uh4itcmn1d98ha5m 2440290 2440289 2026-07-23T12:09:18Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-prometheus-3 2440290 wikitext text/x-wiki 213d1p9xlf2tpa48e5ys0h1kc84ij7p 2440291 2440290 2026-07-23T12:10:03Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-prometheus-3 2440291 wikitext text/x-wiki gnbqx7lgkqv4lyglypu8i7q2tzb3bnd 0 448033 2440497 2431734 2026-07-23T17:25:16Z Dreamyshade 53333 Link to page at [[meta:Machine learning models]] with related documentation 2440497 wikitext text/x-wiki kx6ws6fznggilbf7eyhl8yxd98vz7gj 0 449160 2440481 2440230 2026-07-23T16:59:21Z Dexbot 30554 Bot: Updating the report 2440481 wikitext text/x-wiki csokciq6r3tcahuym7en1fvnxdyjsep 2440556 2440481 2026-07-24T00:00:17Z Dexbot 30554 Bot: Updating the report 2440556 wikitext text/x-wiki jyrvn0iks0rshr3gy856g66o6qt91nu 0 451481 2440643 2432305 2026-07-24T10:31:39Z Effie Mouzeli (WMF) 12880 /* Puppet: remove kubeconfig files */ 2440643 wikitext text/x-wiki 9u9dteb9a9nofk2fz54vl43ufcsdat1 2440658 2440643 2026-07-24T11:00:25Z Effie Mouzeli (WMF) 12880 /* 4. Remove the kubernetes service definitions */ 2440658 wikitext text/x-wiki tccxwnlb89d77gh2l0cjd8nmcfzim0c 2440659 2440658 2026-07-24T11:00:44Z Effie Mouzeli (WMF) 12880 /* 4. Remove the kubernetes namespace service definitions */ 2440659 wikitext text/x-wiki igj4shrwszucgnm48k4izqap87i77sh 2440660 2440659 2026-07-24T11:01:12Z BJensen-WMF 48942 /* Preparing the commit */ fix numbering 2440660 wikitext text/x-wiki 68ewqrqysnyyb49wm1bg3szs2fho5tx 12 452600 2440535 2433505 2026-07-23T19:14:17Z Slashme 3558 No EOF when not making a here document 2440535 wikitext text/x-wiki 2njiqfgud0oqixosaxmt4ih0s86naxb 0 453382 2440629 2428503 2026-07-24T09:52:12Z AYounsi (WMF) 10040 remote_instance:gnmi_bgp_neighbor_prefixes_received_pre_policy{} 2440629 wikitext text/x-wiki lgmoropr32i0f35t1h3mqc38r7dlz9l 0 453763 2440654 2426284 2026-07-24T10:48:36Z Phuedx (WMF) 47850 Phuedx (WMF) moved page [[Test Kitchen/Deploy client libraries]] to [[Test Kitchen/Archive/Deploy client libraries]] 2426284 wikitext text/x-wiki guqmyhnff86kmnunpk8osh7d152khah 116 453906 2440302 2440191 2026-07-23T12:30:26Z Gitlabaccountapprovalbot 37332 @fermiboson was approved. 2440302 wikitext text/x-wiki 74ky8ks3blahodcrnebzclfdvd6fv9g 2440623 2440302 2026-07-24T09:39:24Z Gitlabaccountapprovalbot 37332 @pankaj199 was approved. 2440623 wikitext text/x-wiki 9k8y28bhkbsz3syn70ggjik8lujlzrt 2440656 2440623 2026-07-24T10:54:27Z Gitlabaccountapprovalbot 37332 @ysogo was approved. 2440656 wikitext text/x-wiki 796v8j4issgq5vrnsrxr8768b8ck38e 0 459044 2440447 2440111 2026-07-23T15:33:06Z RKemper (WMF) 17058 /* Alerts */ Add CirrusSearchTitleSuggestIndexTooOld partially stolen from https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#CirrusSearch_titlesuggest_index_is_too_old 2440447 wikitext text/x-wiki jd2k6ikxmf8eymuofy6s7iviusllaf2 2440513 2440447 2026-07-23T18:20:33Z BKing (WMF) 29017 /* Alerts */ 2440513 wikitext text/x-wiki rigtkrh2e1wcda5sekiu6u2i84xapbr 0 459699 2440555 2440125 2026-07-23T23:50:21Z Stashbot 7414 larssandergreen: civicrm upgraded from bf93abac to 98ef4abd 2440555 wikitext text/x-wiki hk9cclfu35abaj0oc54vws6sfjku1l8 2 460039 2440618 2436335 2026-07-24T09:31:06Z Atsuko-WMF 54082 buildx 2440618 wikitext text/x-wiki e6xtgek5it82dmymu8grf52z5gte8n6 2440619 2440618 2026-07-24T09:31:43Z Atsuko-WMF 54082 /* Tools/tricks */ 2440619 wikitext text/x-wiki tfxd3zpox5g3ced0nk8v4eseq11j2uk 1 460290 2440502 2423930 2026-07-23T17:46:34Z Dreamyshade 53333 /* Is this the right place for discussion? */ Reply 2440502 wikitext text/x-wiki iwzch9pd875rj324q6jdj3ky3sd61u6 3 460478 2440297 2026-07-23T12:17:39Z StrikerBot 8475 Welcome to Toolforge! 2440297 wikitext text/x-wiki jhzfhvn46oat46g4y22ai4zw0e9sksc 3 460479 2440318 2026-07-23T13:33:03Z StrikerBot 8475 Welcome to Toolforge! 2440318 wikitext text/x-wiki 85xip4rtvhjptyyfywa201ph6qrdfn3 0 460480 2440527 2026-07-23T18:42:20Z ADancy (WMF) 17694 Instructions on how to enable the MOTD 2440527 wikitext text/x-wiki 6geijuifcbs6k85sruwgrgz75p5ckdh 2440530 2440527 2026-07-23T19:00:01Z ADancy (WMF) 17694 First real version 2440530 wikitext text/x-wiki kwqtyayegw4fu5kuwiit3jsndtnyr5x 2440531 2440530 2026-07-23T19:03:29Z ADancy (WMF) 17694 add note about checking the branch 2440531 wikitext text/x-wiki 5yh4cowzktti7j9vi6vo5dfkg1my562 2440541 2440531 2026-07-23T19:42:41Z Mutante 20 add {{gerrit nav}} to hopefully add a link on the main page 2440541 wikitext text/x-wiki qfuxs9l2yady9j0l271yx2co8dhpde6 3 460481 2440569 2026-07-24T06:41:57Z StrikerBot 8475 Welcome to Toolforge! 2440569 wikitext text/x-wiki 2z76o0d81rc8o3j7h2lz711x7bvx3cl 3 460482 2440575 2026-07-24T06:47:23Z StrikerBot 8475 Welcome to Toolforge! 2440575 wikitext text/x-wiki kdzfe5n7kr5wjlfyf16mgym55quvfo7 3 460483 2440576 2026-07-24T06:53:39Z StrikerBot 8475 Welcome to Toolforge! 2440576 wikitext text/x-wiki s8ixymb3we8b2ga0dcpgm5bp5y8h9cy 2 460484 2440592 2026-07-24T08:29:57Z Empirepetroleumservices 57972 Empire Petroleum Services stands out among gas station repair companies by offering fast, reliable solutions that keep fueling operations running safely and efficiently. Our skilled technicians diagnose equipment issues, perform expert repairs, and provide preventative maintenance to reduce unexpected breakdowns. We are committed to helping fuel station owners protect their investment and deliver a better customer experience. 2440592 wikitext text/x-wiki ljzvg44wde05b5jb6hf92g6qg3rgk0h 0 460485 2440655 2026-07-24T10:48:36Z Phuedx (WMF) 47850 Phuedx (WMF) moved page [[Test Kitchen/Deploy client libraries]] to [[Test Kitchen/Archive/Deploy client libraries]] 2440655 wikitext text/x-wiki kg0nwnqrujqctwkce0a887gvirdtiux