Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.20 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2457627 2457348 2026-09-15T18:09:13Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2457627 wikitext text/x-wiki 5f677k0wo618jbb9e7amwmh8jvxj27c 0 3863 2457846 2453295 2026-09-16T00:04:10Z MSK 47079 /* Wholesome Fun */ [[module:scapsay]] 2457846 wikitext text/x-wiki 9p8eex6b2ebgn33jobasxqlofpn490x 2457859 2457846 2026-09-16T03:02:24Z MSK 47079 scap: now in scribunto, pt 2 2457859 wikitext text/x-wiki 41mm6xvf8bcrkojtwfvtzadlff6p6lg 0 4108 2457444 2457301 2026-09-15T12:27:10Z ScheduleDeploymentBot 37566 Add [[gerrit:1341861]] to Tuesday, September 15 UTC afternoon backport window 2457444 wikitext text/x-wiki bh0vsvgjgd9m8bs59chonvpl9gjy1of 2457447 2457444 2026-09-15T12:41:17Z ScheduleDeploymentBot 37566 Add [[gerrit:1340558]] to Tuesday, September 15 UTC afternoon backport window 2457447 wikitext text/x-wiki n8kf0szqumeklkg313sbdpn1dco61cz 2457453 2457447 2026-09-15T12:54:59Z A826 34037 /* {{Deployment_day|date=2026-09-15}} */ removed already merged 2457453 wikitext text/x-wiki meuc27hv43m4aeq4ev9ojyxx0a10m5f 2457481 2457453 2026-09-15T13:22:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1341292]] to Tuesday, September 15 UTC afternoon backport window 2457481 wikitext text/x-wiki pmf245x207vgj8glyul7407lzyw2wz0 2457511 2457481 2026-09-15T14:42:37Z ScheduleDeploymentBot 37566 Add [[gerrit:1339741]] to Tuesday, September 15 UTC late backport window 2457511 wikitext text/x-wiki b3x5klkr0r6shu6w6onuif0cprc7shz 2457646 2457511 2026-09-15T18:29:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1341271]] to Tuesday, September 15 UTC late backport window 2457646 wikitext text/x-wiki 6e1wqf6mdoko7crwx9mmbrkgyb3a5jt 2457704 2457646 2026-09-15T19:46:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1342025]] to Tuesday, September 15 UTC late backport window 2457704 wikitext text/x-wiki qi90839465xqks04dj1toz6kvhy9cwa 2457802 2457704 2026-09-15T22:39:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1331830]] to Wednesday, September 16 UTC morning backport window 2457802 wikitext text/x-wiki rbk9248ve7ip0kpppm7ana7h8491h7j 2457803 2457802 2026-09-15T22:41:05Z Cscott 41391 Undid revision [[Special:Diff/2457802|2457802]] by [[Special:Contributions/ScheduleDeploymentBot|ScheduleDeploymentBot]] ([[User talk:ScheduleDeploymentBot|talk]]) Did not mean to schedule a deploy at 3am! 2457803 wikitext text/x-wiki qi90839465xqks04dj1toz6kvhy9cwa 2457804 2457803 2026-09-15T22:41:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1331830]] to Wednesday, September 16 UTC late backport window 2457804 wikitext text/x-wiki s3ieykedklsbwnz6723k0thivnpg5ak 2457805 2457804 2026-09-15T22:42:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1340216]] to Wednesday, September 16 UTC late backport window 2457805 wikitext text/x-wiki 87p200faf8u8gq63o9fu9d0a4nysxj0 2457806 2457805 2026-09-15T22:42:44Z ScheduleDeploymentBot 37566 Add [[gerrit:1341949]] to Wednesday, September 16 UTC late backport window 2457806 wikitext text/x-wiki 10t5gtphgynopcuov91k37zaj14m5wg 2457876 2457806 2026-09-16T06:36:11Z ScheduleDeploymentBot 37566 Add [[gerrit:1342112]] to Wednesday, September 16 UTC morning backport window 2457876 wikitext text/x-wiki 6nzq8j1yuy2n48znzysglz9642roeoq 2457877 2457876 2026-09-16T06:36:21Z ScheduleDeploymentBot 37566 Add [[gerrit:1342113]] to Wednesday, September 16 UTC morning backport window 2457877 wikitext text/x-wiki e7zqhltz1hmqub0da8hc1xofk4q6dd8 2457880 2457877 2026-09-16T06:50:09Z ScheduleDeploymentBot 37566 Add [[gerrit:1342117]] to Wednesday, September 16 UTC morning backport window 2457880 wikitext text/x-wiki 76n806hijswoq982ofd8i4y6tdziqk8 2457881 2457880 2026-09-16T06:50:16Z ScheduleDeploymentBot 37566 Add [[gerrit:1342118]] to Wednesday, September 16 UTC morning backport window 2457881 wikitext text/x-wiki cdfcktnushvkbpq97kwhhfjvtefzq78 498 6726 2457621 2457346 2026-09-15T17:47:30Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2457621 wikitext text/x-wiki 1y9un8z93clrnugrnwe7cejzq4kslkc 0 7195 2457505 2443276 2026-09-15T14:19:31Z BTullis (WMF) 25295 /* Hostname prefixes */ Add ceph-admin hostname prefix 2457505 wikitext text/x-wiki kj7s015meret22lgrzl188zqcsin7qq 0 7919 2457417 2457416 2026-09-15T12:00:54Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7001.magru.wmnet} and A:liberica 2457417 wikitext text/x-wiki rqobs7su5n8o3dz7ea8ed7iekd878m4 2457418 2457417 2026-09-15T12:01:13Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7001.magru.wmnet} and A:liberica 2457418 wikitext text/x-wiki 9ofn3ppjmg49glbboagmdkrlwas3wbf 2457419 2457418 2026-09-15T12:02:00Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2457419 wikitext text/x-wiki b8njpmr82k11uklamwldf6gpfwxerqf 2457420 2457419 2026-09-15T12:03:25Z Stashbot 7414 XioNoX: push pfw policies - T437627 2457420 wikitext text/x-wiki 8ejnohl0f1qp52z6yx6t2zn5p4ct760 2457421 2457420 2026-09-15T12:03:51Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2457421 wikitext text/x-wiki 0qn34abf88p3vsrrxnht2vvyv0wu0e2 2457422 2457421 2026-09-15T12:06:02Z Stashbot 7414 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply 2457422 wikitext text/x-wiki qlk5y7opunokpddx1hbhwt0bomq1d2j 2457423 2457422 2026-09-15T12:06:49Z Stashbot 7414 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply 2457423 wikitext text/x-wiki icqy48vzlz6bg4makmup0gm947epe6z 2457424 2457423 2026-09-15T12:07:07Z Stashbot 7414 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply 2457424 wikitext text/x-wiki 3vqc4z0nvq76ekrfwu9jvkj16yny5m6 2457425 2457424 2026-09-15T12:07:39Z Stashbot 7414 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply 2457425 wikitext text/x-wiki m7ekl2o8dy0f7tuolh1evdfhlo4frhr 2457426 2457425 2026-09-15T12:07:47Z Stashbot 7414 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply 2457426 wikitext text/x-wiki nlyjq9o035q3cnshm7tcngbc9vkd4jg 2457427 2457426 2026-09-15T12:08:16Z Stashbot 7414 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply 2457427 wikitext text/x-wiki 89jdgv14m1vk5685lp5cdihqn9mk541 2457428 2457427 2026-09-15T12:09:43Z Stashbot 7414 jmm@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on install7002.wikimedia.org with reason: switch reboot 2457428 wikitext text/x-wiki dgmr0nfbw3i6qgsww46unrmsz47ykvy 2457429 2457428 2026-09-15T12:11:21Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: switch reboot, T437984] 2457429 wikitext text/x-wiki gkw68ybgml4dhc6tr9utmpctj9ww524 2457430 2457429 2026-09-15T12:11:24Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: switch reboot, T437984] 2457430 wikitext text/x-wiki hqakdae0x0klj9f5q57kjfe718iz4v0 2457431 2457430 2026-09-15T12:11:51Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2457431 wikitext text/x-wiki 3004i5hqvy7seyulrb6bc47g9t9notr 2457433 2457431 2026-09-15T12:12:37Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b4-magru,asw1-b4-magru IPv6,asw1-b4-magru.mgmt with reason: Switch maintenance 2457433 wikitext text/x-wiki 96vmbcz4ojao3mqso69jcx1p0z9j6l3 2457434 2457433 2026-09-15T12:13:19Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 12 hosts with reason: Switch maintenance 2457434 wikitext text/x-wiki df8ae07qihf5i4de1c1s5mzzmkxpofz 2457435 2457434 2026-09-15T12:13:59Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2457435 wikitext text/x-wiki gos419dtkd2iseej4ayvh3o8az8li5e 2457436 2457435 2026-09-15T12:19:06Z Stashbot 7414 slyngshede@puppetserver1001: conftool action : set/weight=1; selector: name=cp7010.magru.wmnet 2457436 wikitext text/x-wiki kx5btc1hqmy17zdmqv6wb22i5ifx8vv 2457437 2457436 2026-09-15T12:22:11Z Stashbot 7414 moritzm: installing shadow security updates 2457437 wikitext text/x-wiki jc0rt48m58bv70gmnrq9tdjej4md0yw 2457438 2457437 2026-09-15T12:23:28Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7001.magru.wmnet} and A:liberica 2457438 wikitext text/x-wiki peakwgxxdxrmfy8ghtuvtbrhdpzp3o9 2457439 2457438 2026-09-15T12:23:43Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin depooling P{lvs7001.magru.wmnet} and A:liberica 2457439 wikitext text/x-wiki 1cub3zp25dwmu2tk4qdlfbe4z7nxb4i 2457440 2457439 2026-09-15T12:23:55Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7001.magru.wmnet} and A:liberica 2457440 wikitext text/x-wiki rvmhnp498wdqujkmqknec6adxrkpdp0 2457441 2457440 2026-09-15T12:24:08Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin pooling P{lvs7001.magru.wmnet} and A:liberica 2457441 wikitext text/x-wiki 7hrqhpcmuqyfu5bh7c7kjmb0tjw6vs7 2457442 2457441 2026-09-15T12:24:30Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7001.magru.wmnet} and A:liberica 2457442 wikitext text/x-wiki oe24eddkvi66eg1xj7zc9u2hlvq1s31 2457443 2457442 2026-09-15T12:24:32Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7001.magru.wmnet} and A:liberica 2457443 wikitext text/x-wiki 31l0r7i6k3o9jqyock68uqg5yqmd0or 2457445 2457443 2026-09-15T12:29:30Z Stashbot 7414 XioNoX: asw1-b4-magru> request system reboot - T437984 2457445 wikitext text/x-wiki lmbli5e675kinyyunzq5xctd03t6z25 2457446 2457445 2026-09-15T12:40:55Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: network maintenance finished, T437984] 2457446 wikitext text/x-wiki sleyw6pa4yf7l10uh4er3nun976mn7b 2457448 2457446 2026-09-15T12:42:37Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: network maintenance finished, T437984] 2457448 wikitext text/x-wiki pwjgzfwj98dw7h9i60to9aybvvt4wa8 2457449 2457448 2026-09-15T12:45:30Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on P{cephosd2001.codfw.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad) 2457449 wikitext text/x-wiki k6vaqivi2bkiivo66wywbswo340hmhm 2457450 2457449 2026-09-15T12:46:42Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.rotate-osd-keys (exit_code=0) rolling rotate_keys on P{cephosd2001.codfw.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad) 2457450 wikitext text/x-wiki apbb00fu7mklacl9jkc243aeobrwm3l 2457454 2457450 2026-09-15T12:59:44Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on P{cephosd2001.codfw.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad) 2457454 wikitext text/x-wiki sfbrbq3u2h36eva1ns2yb9kjhh7l8ih 2457455 2457454 2026-09-15T13:00:56Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.rotate-osd-keys (exit_code=0) rolling rotate_keys on P{cephosd2001.codfw.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad) 2457455 wikitext text/x-wiki 2doau4ztw5jlvjmljzhth3ipja2cv2c 2457456 2457455 2026-09-15T13:03:49Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on P{cephosd2001.codfw.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad) 2457456 wikitext text/x-wiki 15k9ik4sudnv2ivxi13rxmunqqg1n5o 2457458 2457456 2026-09-15T13:07:28Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica 2457458 wikitext text/x-wiki cm8cwz8s8ws0x9hffkwu373kwt52na7 2457459 2457458 2026-09-15T13:07:32Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica 2457459 wikitext text/x-wiki ovtwicvacsirxsgxtajda054yh7i9pf 2457460 2457459 2026-09-15T13:07:43Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica 2457460 wikitext text/x-wiki t8vfvn71iu6d3a9987mngbqzbkzi88q 2457461 2457460 2026-09-15T13:07:56Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica 2457461 wikitext text/x-wiki mh7tjibbdgckk5z6cgsmxcmf8c9id73 2457462 2457461 2026-09-15T13:08:18Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica 2457462 wikitext text/x-wiki nzxnk3ug80x6psunbtemdsnlakd9c5a 2457463 2457462 2026-09-15T13:08:20Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica 2457463 wikitext text/x-wiki s5tcv6fwsxgmwnogh2jkv7kx9aq62go 2457464 2457463 2026-09-15T13:10:14Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7003.magru.wmnet} and A:liberica 2457464 wikitext text/x-wiki ljf6e7yv3kz1gf2tnsjvm1lozugw9pb 2457465 2457464 2026-09-15T13:10:22Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1334946|ArticleGuidance: Remove the experiment configuration keys (T434487)]] 2457465 wikitext text/x-wiki ryuh98tlttdqbyudcrtn56508kgxfmu 2457466 2457465 2026-09-15T13:10:33Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7003.magru.wmnet} and A:liberica 2457466 wikitext text/x-wiki rvrhvpg8vfr067dj82xrbwrwy9wxrse 2457469 2457466 2026-09-15T13:14:41Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1334946|ArticleGuidance: Remove the experiment configuration keys (T434487)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457469 wikitext text/x-wiki o9169px4w7tqllk32fp9v2uv6kpp95j 2457470 2457469 2026-09-15T13:15:09Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2457470 wikitext text/x-wiki p826pajkapal76y6469h04mqgfbj1xk 2457472 2457470 2026-09-15T13:17:00Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.rotate-osd-keys (exit_code=0) rolling rotate_keys on P{cephosd2001.codfw.wmnet} and (A:cephosd-codfw or A:cephosd-eqiad) 2457472 wikitext text/x-wiki 85b67rum2an8scfzifwqrmrtnueym47 2457478 2457472 2026-09-15T13:19:41Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1334946|ArticleGuidance: Remove the experiment configuration keys (T434487)]] (duration: 09m 19s) 2457478 wikitext text/x-wiki pcbwj1kltwuounb82onbl15agateag4 2457479 2457478 2026-09-15T13:20:36Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply 2457479 wikitext text/x-wiki 2rv29elcspyvtqdef63n57u5t887tj4 2457480 2457479 2026-09-15T13:21:04Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply 2457480 wikitext text/x-wiki oe5yruwu49v4xirylk1ozq4e6kc7rtj 2457482 2457480 2026-09-15T13:26:09Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1341861|SuggestedInvestigations: Update "sockpuppet" queue view defaults (T438018)]] 2457482 wikitext text/x-wiki 7o5w5cz7n4e4vpzlbl9cjvx4whk4m5d 2457484 2457482 2026-09-15T13:30:25Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1341861|SuggestedInvestigations: Update "sockpuppet" queue view defaults (T438018)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457484 wikitext text/x-wiki 9n8mtcxyahisc3gp0yvok7f4udl7ckp 2457485 2457484 2026-09-15T13:32:03Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2457485 wikitext text/x-wiki 9bcunmp0d69m1c25n3cbxtwiyx9jqpd 2457487 2457485 2026-09-15T13:36:32Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341861|SuggestedInvestigations: Update "sockpuppet" queue view defaults (T438018)]] (duration: 10m 23s) 2457487 wikitext text/x-wiki 2mcn2kcm8jjwy8c43f9wg8xothao2nv 2457492 2457487 2026-09-15T13:43:56Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.ceph.rotate-osd-keys rolling rotate_keys on A:cephosd-codfw 2457492 wikitext text/x-wiki md21untr5fz7k4t823cw6k60sskx3oi 2457497 2457492 2026-09-15T14:05:13Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1341895|ReportIncidentController: Instance cache expensive methods (T437588)]] 2457497 wikitext text/x-wiki kw9hhuwumqzd7gu4dcevuaxisq0vhk3 2457498 2457497 2026-09-15T14:09:34Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1341895|ReportIncidentController: Instance cache expensive methods (T437588)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457498 wikitext text/x-wiki quygu774zyp4yt2pjz7i0cq1lzsqvp0 2457500 2457498 2026-09-15T14:12:39Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2457500 wikitext text/x-wiki 960o10gwrsxkxuwcplgit0h8nxa6lh7 2457501 2457500 2026-09-15T14:12:44Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2457501 wikitext text/x-wiki axr8p7jlookogm92rijm5yc6jn3l6b4 2457502 2457501 2026-09-15T14:12:57Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2457502 wikitext text/x-wiki saaiyctbxy0al1np9j8ptuhzejk96ds 2457503 2457502 2026-09-15T14:16:49Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.ceph.rotate-osd-keys (exit_code=0) rolling rotate_keys on A:cephosd-codfw 2457503 wikitext text/x-wiki otpfkh2hg2nbqnory9s3myhvdm1fncv 2457504 2457503 2026-09-15T14:17:09Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341895|ReportIncidentController: Instance cache expensive methods (T437588)]] (duration: 11m 56s) 2457504 wikitext text/x-wiki 0f8j4l8enasjh9d9s9qokex1zqinqo9 2457508 2457504 2026-09-15T14:34:56Z Stashbot 7414 tgr@deploy1003: Started scap sync-world: Backport for [[gerrit:1341292|CommonSettings: Use a restrictive CSP for auth.wikimedia.org (T419684)]] 2457508 wikitext text/x-wiki 5lccoz0x2vr399lhr12s5ogqcgoju8k 2457509 2457508 2026-09-15T14:39:14Z Stashbot 7414 tgr@deploy1003: tgr, arendpieter: Backport for [[gerrit:1341292|CommonSettings: Use a restrictive CSP for auth.wikimedia.org (T419684)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457509 wikitext text/x-wiki 5rvke6uy8qi1hu72qlinrz66b6e42zx 2457510 2457509 2026-09-15T14:42:03Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2457510 wikitext text/x-wiki 2966x3bsc9zm1l0mxseahxwuhpfn58i 2457512 2457510 2026-09-15T14:42:48Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2457512 wikitext text/x-wiki d9chj6i3w2gum7fqu39e7ma3jbacjdk 2457513 2457512 2026-09-15T14:44:24Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply 2457513 wikitext text/x-wiki 1504qmffxp11lh2i4h8s0zfjqqrq1j6 2457514 2457513 2026-09-15T14:45:02Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply 2457514 wikitext text/x-wiki ov1bjhugqjhhv86anzrmf7b6yy9r7dm 2457515 2457514 2026-09-15T14:45:20Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2457515 wikitext text/x-wiki 62lzrve5g5ymfocahk3tthcbjy9jvok 2457516 2457515 2026-09-15T14:45:43Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2457516 wikitext text/x-wiki 0n4bid7e74qaze3t25qmumcg3x1mx2j 2457518 2457516 2026-09-15T14:47:24Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2457518 wikitext text/x-wiki oz6icybn0m2gi0rrdk83kib2pxke3pp 2457519 2457518 2026-09-15T14:47:58Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2457519 wikitext text/x-wiki sczzvwgtisj70fiwfke8a1h3u8lgyt1 2457520 2457519 2026-09-15T14:48:06Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2457520 wikitext text/x-wiki 1butad0a4pfibnbopyxmqd1a45gahhx 2457521 2457520 2026-09-15T14:48:38Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2457521 wikitext text/x-wiki 62vxroteds6stvehq0ayy5ropk050li 2457522 2457521 2026-09-15T14:48:53Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply 2457522 wikitext text/x-wiki insgmxbadokvniuoaxg22g0773lzrcn 2457523 2457522 2026-09-15T14:49:25Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply 2457523 wikitext text/x-wiki c9km1ttpc5an0mr945v7qo63t39qla2 2457524 2457523 2026-09-15T14:52:37Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply 2457524 wikitext text/x-wiki tcun98pih8dwrorbgkg3cvz3prg0p5y 2457525 2457524 2026-09-15T14:53:09Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply 2457525 wikitext text/x-wiki suc0t3jz3pn2qnheoxc40t3jszmkusg 2457526 2457525 2026-09-15T14:53:52Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2457526 wikitext text/x-wiki o0553efr536ut0gcpz6jvip4gpgrnp9 2457527 2457526 2026-09-15T14:54:27Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2457527 wikitext text/x-wiki 00ozfpaeydjhfd1olbkycmt1gqg2ngk 2457528 2457527 2026-09-15T14:54:35Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply 2457528 wikitext text/x-wiki ilfhh1wumr79f91kczfu3x1gh9out3i 2457529 2457528 2026-09-15T14:54:58Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply 2457529 wikitext text/x-wiki hukqnt3ete3y6ry73wotm3p0638hwtf 2457530 2457529 2026-09-15T14:55:07Z Stashbot 7414 tgr@deploy1003: tgr, arendpieter: Continuing with deployment 2457530 wikitext text/x-wiki 98l00yz693goblo9qaorwidfgi92kvk 2457531 2457530 2026-09-15T15:00:08Z Stashbot 7414 tgr@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341292|CommonSettings: Use a restrictive CSP for auth.wikimedia.org (T419684)]] (duration: 25m 11s) 2457531 wikitext text/x-wiki sazqkc6ys316z0lzbmda10uvlo4y9uk 2457533 2457531 2026-09-15T15:11:45Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. 2457533 wikitext text/x-wiki ondihobazi0j4cnnbu0wock1w99ycfq 2457534 2457533 2026-09-15T15:11:50Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. 2457534 wikitext text/x-wiki 1tlecawokc0h2yetxjm8orvzkk530md 2457535 2457534 2026-09-15T15:12:20Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2457535 wikitext text/x-wiki d75gohr97cvt8cf193mlify08dag767 2457536 2457535 2026-09-15T15:12:23Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2457536 wikitext text/x-wiki mb5pwh6sd7co1f84ns9xnrtjos067cu 2457543 2457536 2026-09-15T15:26:29Z Stashbot 7414 moritzm: pruned obsolete Bullseye image amd-gpu-tester from the docker registry T416452 2457543 wikitext text/x-wiki 21n8uloonr9vci90640u375qem0mxga 2457544 2457543 2026-09-15T15:26:42Z Stashbot 7414 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet,phab[1005-1006].eqiad.wmnet with reason: Phabricator deploy 2457544 wikitext text/x-wiki o2ou0dz5dsibxvg22s418i4c47y02uy 2457545 2457544 2026-09-15T15:31:38Z Stashbot 7414 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2457545 wikitext text/x-wiki fclih4hgrd25hbh0t6nacc0b4hyv8yp 2457546 2457545 2026-09-15T15:31:44Z Stashbot 7414 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2457546 wikitext text/x-wiki 7knncfnif94qebez085a3sqo1japt4v 2457547 2457546 2026-09-15T15:31:56Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@c386249]: deploy phab2003 for T437930 2457547 wikitext text/x-wiki 7trkblkv5hbbzpmzltgbszevc4w9myh 2457548 2457547 2026-09-15T15:32:20Z Stashbot 7414 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. 2457548 wikitext text/x-wiki iux2d91uw80vs1xbrjt28rkg3d08cas 2457549 2457548 2026-09-15T15:32:25Z Stashbot 7414 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. 2457549 wikitext text/x-wiki rm7ao9e7712789ovfgslb2lj66o30j2 2457550 2457549 2026-09-15T15:32:49Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@c386249]: deploy phab2003 for T437930 (duration: 00m 52s) 2457550 wikitext text/x-wiki fsd5g5akmzcid0wxcd89tus5zehiqan 2457551 2457550 2026-09-15T15:32:50Z Stashbot 7414 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. 2457551 wikitext text/x-wiki qroe35agfmwxwnvd8oo2hdoj6cy0nbi 2457552 2457551 2026-09-15T15:32:51Z Stashbot 7414 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2457552 wikitext text/x-wiki 4wceas2di0z8le9c891ewt0jxmiirjj 2457553 2457552 2026-09-15T15:33:12Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@c386249]: deploy phab1005 for T437930 2457553 wikitext text/x-wiki 7f2hdwpcqi162pci6ihtypzyodbd99e 2457554 2457553 2026-09-15T15:33:16Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1098.eqiad.wmnet with OS bullseye 2457554 wikitext text/x-wiki eo6m055wvac1k37bpm9zkndzvtsjknh 2457555 2457554 2026-09-15T15:33:53Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@c386249]: deploy phab1005 for T437930 (duration: 00m 39s) 2457555 wikitext text/x-wiki fz1rqzq4ds7jyq4zgv1gz8b4fm6i3g5 2457556 2457555 2026-09-15T15:34:04Z Stashbot 7414 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. 2457556 wikitext text/x-wiki aadvr2pfksdhtm245fu2dmz9okolufc 2457557 2457556 2026-09-15T15:34:07Z Stashbot 7414 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. 2457557 wikitext text/x-wiki t55ncewvmccn3lorxedvodb46ih9ql4 2457558 2457557 2026-09-15T15:40:16Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1341890|Restore table borders for client-side MathJax (T435274)]], [[gerrit:1340558|lift IP cap for edit-a-thon /workshop (T437609 T437594 T437470)]] 2457558 wikitext text/x-wiki aemu4l38acfjgw8g5kqaef08r9jmntb 2457560 2457558 2026-09-15T15:44:50Z Stashbot 7414 krinkle@deploy1003: anzx, krinkle: Backport for [[gerrit:1341890|Restore table borders for client-side MathJax (T435274)]], [[gerrit:1340558|lift IP cap for edit-a-thon /workshop (T437609 T437594 T437470)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457560 wikitext text/x-wiki 6ufjc02wnqi0xw35gq7dxwwld6noet3 2457564 2457560 2026-09-15T15:59:43Z Stashbot 7414 krinkle@deploy1003: anzx, krinkle: Continuing with deployment 2457564 wikitext text/x-wiki orlx5ltigdabul4p5z4oo0qvbq3pw53 2457567 2457564 2026-09-15T16:04:34Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341890|Restore table borders for client-side MathJax (T435274)]], [[gerrit:1340558|lift IP cap for edit-a-thon /workshop (T437609 T437594 T437470)]] (duration: 24m 19s) 2457567 wikitext text/x-wiki kjdqcraw9vhtwil7mjk97xgxry1xo4u 2457569 2457567 2026-09-15T16:21:19Z Stashbot 7414 mutante: temp disabling puppet on C:zookeeper (32 hosts) - safe deploy of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327569 2457569 wikitext text/x-wiki 489s4fd6dgv91w0aarmklny610tiipb 2457573 2457569 2026-09-15T16:33:51Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1098.eqiad.wmnet with OS bullseye 2457573 wikitext text/x-wiki t8sjrj5h27tjzeauuhuhm4j8pbhf1lz 2457576 2457573 2026-09-15T16:49:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2457576 wikitext text/x-wiki 9mi96gqd5ebvbf0072t9j1kt72it4mz 2457577 2457576 2026-09-15T16:53:54Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [ms-be1099] - vriley@cumin1003" 2457577 wikitext text/x-wiki n26wz8w3d93l252pnbom4uh05psvnkv 2457578 2457577 2026-09-15T16:53:58Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [ms-be1099] - vriley@cumin1003" 2457578 wikitext text/x-wiki er6xezrempsetu0gfwjm4cbwvxoaj3e 2457579 2457578 2026-09-15T16:53:59Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2457579 wikitext text/x-wiki emzx6zjib3siltobzvgnjfsw04sw4sa 2457580 2457579 2026-09-15T16:54:39Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1098.eqiad.wmnet with OS bullseye 2457580 wikitext text/x-wiki ihubac62xtjkv00i9evgfw7z7krvn9u 2457581 2457580 2026-09-15T16:54:59Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1099 2457581 wikitext text/x-wiki bse31ahx0icrlfwu6vtidftzh7gjlqm 2457582 2457581 2026-09-15T16:55:01Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ms-be1099 2457582 wikitext text/x-wiki 0wxq52e5cqjkqarzc5b8jxh02p9027g 2457583 2457582 2026-09-15T16:55:22Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1099 2457583 wikitext text/x-wiki chcuczph7de22amsh3ofuqq18dx27jv 2457584 2457583 2026-09-15T16:55:24Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ms-be1099 2457584 wikitext text/x-wiki 5un8zujyh91dahk4ub6bizal15vrzg6 2457585 2457584 2026-09-15T16:55:55Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2457585 wikitext text/x-wiki jsm0yrlneuprbf215c7jbrtq7m8ahlv 2457586 2457585 2026-09-15T16:55:56Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2457586 wikitext text/x-wiki 8zyxvfw0z6rkod4kvzsxxsnciyshubi 2457587 2457586 2026-09-15T16:56:47Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2457587 wikitext text/x-wiki p3z453e1q4zk2jglkvdlrb61eaxddet 2457588 2457587 2026-09-15T16:59:19Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2457588 wikitext text/x-wiki r668juopnwe3e7qmjgffcuu1wc7gkx7 2457589 2457588 2026-09-15T16:59:44Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1099 2457589 wikitext text/x-wiki fx699yznrc9mdsnfwcqqqa5kbh6jvtg 2457590 2457589 2026-09-15T16:59:45Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ms-be1099 2457590 wikitext text/x-wiki kszbo2w4slaajii0p1i4w2omghld88a 2457591 2457590 2026-09-15T16:59:59Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2457591 wikitext text/x-wiki 32v7eoiz5nn3a0il0vvfpmpo76qnttx 2457592 2457591 2026-09-15T17:00:00Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2457592 wikitext text/x-wiki byvh4d8bj8apyk0e1fach6c9y11u8f1 2457593 2457592 2026-09-15T17:01:38Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2457593 wikitext text/x-wiki e1bko67k44iztt1evdm0ngtev11kv3l 2457594 2457593 2026-09-15T17:01:39Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2457594 wikitext text/x-wiki 40b8uxzxeaantgxogm1xsdxsx0bb937 2457595 2457594 2026-09-15T17:02:57Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7009.* 2457595 wikitext text/x-wiki 9i6zdbivu9ittvg0qmr4p05vbksmria 2457596 2457595 2026-09-15T17:21:02Z Stashbot 7414 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org 2457596 wikitext text/x-wiki aq1fbgfob3syab0w5bsjavqy24n6wfz 2457599 2457596 2026-09-15T17:22:39Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2457599 wikitext text/x-wiki 4ad4cj3zfalusjjs01z1at7gcuylqui 2457600 2457599 2026-09-15T17:22:40Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2457600 wikitext text/x-wiki c5d4pa3o7an417y3h39rjhxda5diun6 2457602 2457600 2026-09-15T17:23:09Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp1101.eqiad.wmnet 2457602 wikitext text/x-wiki ajuc65tpto1rg4gr8pqmj9pl7ritgyu 2457605 2457602 2026-09-15T17:24:51Z Stashbot 7414 cdobbins@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7009.magru.wmnet 2457605 wikitext text/x-wiki 522glxmsgj3exw40f2tna86cg91up1g 2457607 2457605 2026-09-15T17:25:35Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp1101.eqiad.wmnet with OS trixie 2457607 wikitext text/x-wiki s8mt40zov25kcqs7zp2534uit4q6tt9 2457610 2457607 2026-09-15T17:26:14Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp7009.magru.wmnet with OS trixie 2457610 wikitext text/x-wiki bgk86qb8uiutmtug4tgweex9oozcwfg 2457611 2457610 2026-09-15T17:28:51Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4045.* 2457611 wikitext text/x-wiki 116ad74e6ulhqem55as33irv47qn2ir 2457612 2457611 2026-09-15T17:28:54Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4046.* 2457612 wikitext text/x-wiki 1g2f16d75dcmgk1g7l3fqb6uyu4ihxn 2457613 2457612 2026-09-15T17:33:38Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp3074.esams.wmnet 2457613 wikitext text/x-wiki kayzb3cok3bfjt19xftn3d0zolwl0bv 2457614 2457613 2026-09-15T17:34:14Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp4045.ulsfo.wmnet with OS trixie 2457614 wikitext text/x-wiki jfswjr0dn2p3crvmre85exzieu1wfj3 2457615 2457614 2026-09-15T17:34:21Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp4046.ulsfo.wmnet with OS trixie 2457615 wikitext text/x-wiki nnxism46cybo1i3vofx7e5nul4gzgv0 2457617 2457615 2026-09-15T17:36:50Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp3074.esams.wmnet with OS trixie 2457617 wikitext text/x-wiki 69j721kicvlcesjk2o5bcz9rg36lf6a 2457619 2457617 2026-09-15T17:43:44Z Stashbot 7414 cdobbins@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2044.codfw.wmnet 2457619 wikitext text/x-wiki bwibgyjjikge3cqqnwur8dcpm3lfv5y 2457622 2457619 2026-09-15T17:48:50Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp2044.codfw.wmnet with OS trixie 2457622 wikitext text/x-wiki l8d0tdifaa11rvgz0f6ewh425ox3q9f 2457623 2457622 2026-09-15T17:52:55Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7009.magru.wmnet with reason: host reimage 2457623 wikitext text/x-wiki 9w99wibvo19npj2zxhuubgasyuir2su 2457624 2457623 2026-09-15T18:07:26Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3074.esams.wmnet with reason: host reimage 2457624 wikitext text/x-wiki e9yvcihzymg3pkpnvszue23bnhha6fz 2457626 2457624 2026-09-15T18:08:54Z Stashbot 7414 eevans@cumin1004: START - Cookbook sre.hosts.reboot-single for host sessionstore1006.eqiad.wmnet 2457626 wikitext text/x-wiki npcbapq3nf57d5m9k5xrmlfoapm6gl6 2457628 2457626 2026-09-15T18:09:27Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp4045.ulsfo.wmnet with OS trixie 2457628 wikitext text/x-wiki ilsj80xloq57b1y49oi1ypw46ujej3y 2457629 2457628 2026-09-15T18:09:43Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2044.codfw.wmnet with reason: host reimage 2457629 wikitext text/x-wiki bm635lt9nc5zib98fvpmte4w7ji1s0r 2457630 2457629 2026-09-15T18:10:14Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp1101.eqiad.wmnet with OS trixie 2457630 wikitext text/x-wiki f9dnftj61gq9vwz9i1tvp5xatgus7qb 2457635 2457630 2026-09-15T18:19:27Z Stashbot 7414 eevans@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1006.eqiad.wmnet 2457635 wikitext text/x-wiki 3ixthxb3qopt3lvwl89j7ccimexiztm 2457636 2457635 2026-09-15T18:19:28Z Stashbot 7414 eevans@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts sessionstore1006.eqiad.wmnet 2457636 wikitext text/x-wiki qnzfvalbh0zkmutvgcv5ien9bk2cfus 2457637 2457636 2026-09-15T18:19:44Z Stashbot 7414 eevans@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457637 wikitext text/x-wiki 3uwgl0sch0w8q7d71fsa05fsobm0fiq 2457638 2457637 2026-09-15T18:20:23Z Stashbot 7414 eevans@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457638 wikitext text/x-wiki t04hat4taejmz90dl8cgoqfgcf8hxke 2457639 2457638 2026-09-15T18:20:44Z Stashbot 7414 eevans@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1006.eqiad.wmnet with OS bookworm 2457639 wikitext text/x-wiki 0i85z2keb7uior7rw0hxoiuv5r0scvn 2457640 2457639 2026-09-15T18:22:16Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7009.magru.wmnet with OS trixie 2457640 wikitext text/x-wiki tdfay688xvpvkstxwz2i68b6ihgwt4x 2457644 2457640 2026-09-15T18:26:17Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1101.eqiad.wmnet with reason: host reimage 2457644 wikitext text/x-wiki euvslqzs00rn805bqgfcug35nob4nll 2457647 2457644 2026-09-15T18:29:26Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.provision for host cp4045.mgmt.ulsfo.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457647 wikitext text/x-wiki 825uyzinxt8wgvvyxrtikj47c0aajz8 2457649 2457647 2026-09-15T18:30:20Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1101.eqiad.wmnet with reason: host reimage 2457649 wikitext text/x-wiki jvo52f4bwhf3xt0ss1jszgv6hkw7lar 2457650 2457649 2026-09-15T18:31:34Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3074.esams.wmnet with OS trixie 2457650 wikitext text/x-wiki 66fdnf3h05q1np3yx9qkmrp48ehspp0 2457652 2457650 2026-09-15T18:32:18Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2044.codfw.wmnet with OS trixie 2457652 wikitext text/x-wiki ngol229n2nzbj8wtfojy79amqqu5kfg 2457653 2457652 2026-09-15T18:33:21Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=1; selector: name=cp3074.esams.wmnet 2457653 wikitext text/x-wiki 559t2dshtts0p6s4cypq5l3ghc491a7 2457655 2457653 2026-09-15T18:36:40Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3008.esams.wmnet} and A:liberica 2457655 wikitext text/x-wiki l6v2n9lrzgvmqv7b4to6gqqzum644y2 2457656 2457655 2026-09-15T18:36:59Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3008.esams.wmnet} and A:liberica 2457656 wikitext text/x-wiki oegxxmn9gmco6tynfz879w8mpqyxn9r 2457657 2457656 2026-09-15T18:39:26Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp3074.esams.wmnet 2457657 wikitext text/x-wiki 4mbm9yczqk10cuzz9cu2xykzmcrxo2d 2457658 2457657 2026-09-15T18:40:23Z Stashbot 7414 eevans@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage 2457658 wikitext text/x-wiki 8yz0pi5khoxvcixs3x8jrone8h2l040 2457659 2457658 2026-09-15T18:40:25Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4045.mgmt.ulsfo.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457659 wikitext text/x-wiki is74gz04p1shbj403360m684h2pkiw4 2457661 2457659 2026-09-15T18:44:25Z Stashbot 7414 eevans@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage 2457661 wikitext text/x-wiki nwk0du9x96k7sdy4z1tzd7kipp1pcay 2457662 2457661 2026-09-15T18:44:31Z Stashbot 7414 cdobbins@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2044.codfw.wmnet 2457662 wikitext text/x-wiki b781u9prcalgprvo1jgzofq01eyw9pb 2457663 2457662 2026-09-15T18:45:15Z Stashbot 7414 cdobbins@puppetserver1001: conftool action : set/weight=1; selector: name=cp2044.codfw.wmnet 2457663 wikitext text/x-wiki pknyovx324icgdaxra6eaai3008g3kz 2457664 2457663 2026-09-15T18:52:50Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1101.eqiad.wmnet with OS trixie 2457664 wikitext text/x-wiki 11aomd9kb5af0n16zctp1jzfwx4g33t 2457665 2457664 2026-09-15T18:52:55Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.provision for host cp4046.mgmt.ulsfo.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457665 wikitext text/x-wiki fpgm9fwzetjmpqz4dmy3405f0b7okfs 2457666 2457665 2026-09-15T18:55:11Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp4045.ulsfo.wmnet with OS trixie 2457666 wikitext text/x-wiki nwt39mfrdj246imh951cwgizlz65f38 2457667 2457666 2026-09-15T18:55:37Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=1; selector: name=cp1101.eqiad.wmnet 2457667 wikitext text/x-wiki jlkrb4igseg07cw93ctr6hkhprv5rcg 2457668 2457667 2026-09-15T18:55:41Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1101.eqiad.wmnet 2457668 wikitext text/x-wiki s8rx63fxm532fon29lete25udb624wb 2457669 2457668 2026-09-15T18:57:59Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp3075.esams.wmnet 2457669 wikitext text/x-wiki cnewpce1hv1zyivln1evezx51m1f4qk 2457670 2457669 2026-09-15T18:58:16Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.provision for host cp7009.mgmt.magru.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457670 wikitext text/x-wiki 4gs63estyv9tdmz2fqy26ahmj5m6hr5 2457671 2457670 2026-09-15T19:00:08Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.provision for host cp3075.mgmt.esams.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457671 wikitext text/x-wiki dd0htu7tp2kk53mjj810gj5egcwunfd 2457672 2457671 2026-09-15T19:01:17Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1103.eqiad.wmnet 2457672 wikitext text/x-wiki pfa72c8taaanfo5czu2v3e5tgceh3tj 2457673 2457672 2026-09-15T19:01:19Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp1103.eqiad.wmnet 2457673 wikitext text/x-wiki en31astq26ekrunyaal5q049hrqnclc 2457674 2457673 2026-09-15T19:02:25Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4046.mgmt.ulsfo.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457674 wikitext text/x-wiki hf0l55fb6jvxu2n127ze5hnkiamlwto 2457675 2457674 2026-09-15T19:04:01Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp3075.* 2457675 wikitext text/x-wiki 67kq8nbk3oxu145otbcysbmx1e3odsg 2457676 2457675 2026-09-15T19:05:03Z Stashbot 7414 eevans@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1006.eqiad.wmnet with OS bookworm 2457676 wikitext text/x-wiki rzz9mov5ah07phtlhckfhx6rkli7j6c 2457677 2457676 2026-09-15T19:05:56Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp4046.ulsfo.wmnet with OS trixie 2457677 wikitext text/x-wiki sevtlwrtm35xd999c9bgdp91cc5tpc3 2457678 2457677 2026-09-15T19:08:14Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.provision for host cp1103.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457678 wikitext text/x-wiki rj0akho5rm8au2s7p18ab48pvjl74zr 2457679 2457678 2026-09-15T19:08:46Z Stashbot 7414 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp3075.esams.wmnet with reason: reimaging 2457679 wikitext text/x-wiki dqp0dte9g135xn0qgwqorcw98hnjmlq 2457682 2457679 2026-09-15T19:09:49Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7009.mgmt.magru.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457682 wikitext text/x-wiki 2707mde53cff0w1jfynlr59by9o90yw 2457683 2457682 2026-09-15T19:10:06Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp7009.magru.wmnet with OS trixie 2457683 wikitext text/x-wiki o03wgml3jl9xvgoldq4g53gxsxd3wg6 2457684 2457683 2026-09-15T19:11:04Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp3075.mgmt.esams.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457684 wikitext text/x-wiki f7mww7ess934l4i07vakwnax4nz4jee 2457685 2457684 2026-09-15T19:12:13Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2457685 wikitext text/x-wiki bzl6b9878i82tbqk6ikw81mak5osdp1 2457686 2457685 2026-09-15T19:12:15Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4045.ulsfo.wmnet with reason: host reimage 2457686 wikitext text/x-wiki 80ecr4lip3f2ivgxntbhby95oia9of9 2457687 2457686 2026-09-15T19:12:34Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp3075.esams.wmnet with OS trixie 2457687 wikitext text/x-wiki fvk0txuodq82xk0rpbnt5phbatluuit 2457688 2457687 2026-09-15T19:12:44Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2457688 wikitext text/x-wiki 0c6giom877ax9q4t8vno1evpqzoymyt 2457689 2457688 2026-09-15T19:12:45Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2457689 wikitext text/x-wiki mrc2vu9f1lrxp6nvraab8yqv65qttqj 2457690 2457689 2026-09-15T19:13:16Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2457690 wikitext text/x-wiki fv52pcuhcjb3m1cz449gbamqac855bq 2457691 2457690 2026-09-15T19:18:53Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4045.ulsfo.wmnet with reason: host reimage 2457691 wikitext text/x-wiki oyi0xun4p9t2fhgun3ll7wsd8a30pgk 2457692 2457691 2026-09-15T19:19:02Z Stashbot 7414 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp1103.eqiad.wmnet with reason: reimage 2457692 wikitext text/x-wiki 0jlxifvq3ftjkumr5bha44qk8mj35ek 2457693 2457692 2026-09-15T19:19:37Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp1103.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457693 wikitext text/x-wiki ezj7prkyn6d5gmil21kizqn3jau4lhb 2457694 2457693 2026-09-15T19:21:36Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp1103.eqiad.wmnet with OS trixie 2457694 wikitext text/x-wiki 7o57fyjam1twqrgrcl5nm1b9dr03jh0 2457695 2457694 2026-09-15T19:23:55Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4046.ulsfo.wmnet with reason: host reimage 2457695 wikitext text/x-wiki pf5pagpjiq3riktzbqfgc5hb8cbal37 2457696 2457695 2026-09-15T19:27:48Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4046.ulsfo.wmnet with reason: host reimage 2457696 wikitext text/x-wiki 17qps0rg4a0f9psuoarghx94op4x1hj 2457697 2457696 2026-09-15T19:32:08Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7009.magru.wmnet with reason: host reimage 2457697 wikitext text/x-wiki agla8q3fjnhgrwqtun2c0ha3l5lw6vh 2457698 2457697 2026-09-15T19:33:36Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3075.esams.wmnet with reason: host reimage 2457698 wikitext text/x-wiki blaq66s3jhmk78095lfpgaxd1exqxcz 2457699 2457698 2026-09-15T19:37:40Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7009.magru.wmnet with reason: host reimage 2457699 wikitext text/x-wiki jlemdy0mow9i337slyxo27on0077mx3 2457700 2457699 2026-09-15T19:39:05Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1103.eqiad.wmnet with reason: host reimage 2457700 wikitext text/x-wiki d6kzf3d4yqr84o801gx2qa2c3aa1bh2 2457701 2457700 2026-09-15T19:41:52Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3075.esams.wmnet with reason: host reimage 2457701 wikitext text/x-wiki h0lr70mxz5pff2wwwh2fk9w09mlzcbc 2457702 2457701 2026-09-15T19:43:31Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4045.ulsfo.wmnet with OS trixie 2457702 wikitext text/x-wiki hz9f98x70dsjxk4sktw5mebx7l0jcju 2457703 2457702 2026-09-15T19:45:11Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1103.eqiad.wmnet with reason: host reimage 2457703 wikitext text/x-wiki kij1xbmfdxyznt031ekl3qop124eebb 2457705 2457703 2026-09-15T19:49:58Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4045.* 2457705 wikitext text/x-wiki m2eouha75czrvx7cqsr3y6o5yuvg76h 2457706 2457705 2026-09-15T19:51:30Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4046.ulsfo.wmnet with OS trixie 2457706 wikitext text/x-wiki lqhy9gwlmaegv8xiqroezydz35m7045 2457707 2457706 2026-09-15T19:51:49Z Stashbot 7414 cdobbins@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2046.codfw.wmnet 2457707 wikitext text/x-wiki 11hp8uqts912iyzysgk9yk0xpq0d8t4 2457708 2457707 2026-09-15T19:52:02Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4046.* 2457708 wikitext text/x-wiki n5n49wj3krk0ionnd19ly5h249wwyhg 2457710 2457708 2026-09-15T19:53:33Z Stashbot 7414 brett@puppetserver1001: conftool action : set/weight=1; selector: name=cp4046.* 2457710 wikitext text/x-wiki 6rnyp4rijv25cr8hlh2brxwpgwl10hf 2457711 2457710 2026-09-15T19:53:45Z Stashbot 7414 brett@puppetserver1001: conftool action : set/weight=1; selector: name=cp4045.* 2457711 wikitext text/x-wiki 71h1qnfwbvc2wir804ojib8q3w0skmw 2457713 2457711 2026-09-15T20:01:13Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7009.magru.wmnet with OS trixie 2457713 wikitext text/x-wiki 9h72djjz7ocb3e8djhxacx41dr8rax8 2457715 2457713 2026-09-15T20:02:12Z Stashbot 7414 brett@puppetserver1001: conftool action : set/weight=1; selector: name=cp7009.* 2457715 wikitext text/x-wiki f1yu1ifv2ui7emx7647ow0kwo38wkxp 2457716 2457715 2026-09-15T20:02:17Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7009.* 2457716 wikitext text/x-wiki k7rckdm2bskgm32m0lnqpev5vwamtp6 2457717 2457716 2026-09-15T20:02:25Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7009.* 2457717 wikitext text/x-wiki o1awky9ky0z2096yl0ko9cfg0jicm6r 2457718 2457717 2026-09-15T20:04:43Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host cp2046.codfw.wmnet with OS trixie 2457718 wikitext text/x-wiki ovv8grn498ldocdnhyd6yerymslqoh6 2457719 2457718 2026-09-15T20:04:58Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7009.* 2457719 wikitext text/x-wiki lq6pglwxuqd71mi1x84bo92g4r5610f 2457720 2457719 2026-09-15T20:06:36Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3075.esams.wmnet with OS trixie 2457720 wikitext text/x-wiki c3bvez9c7ihbwucp50194jvuw9s73vk 2457721 2457720 2026-09-15T20:07:19Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1339741|Enable ReaderExperiments in eswiki, jawiki, and ptwiki (T438009)]] 2457721 wikitext text/x-wiki jkdasd4bvqclma8tuqv98zslr57l0gq 2457722 2457721 2026-09-15T20:07:31Z Stashbot 7414 inflatador: bking@ganeti1046 sudo gnt-instance modify -B memory=4g,vcpus=4 dse-k8s-etcd100[1-3].eqiad.wmnet T438084 2457722 wikitext text/x-wiki fsehl13dkgemj6o1ezobyocjieo50p5 2457723 2457722 2026-09-15T20:08:09Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1103.eqiad.wmnet with OS trixie 2457723 wikitext text/x-wiki cp42z4gvqf8l03s9p56cpix9a4omebk 2457724 2457723 2026-09-15T20:08:34Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading A:liberica-ulsfo (T436363) 2457724 wikitext text/x-wiki bhz9wwbkh5cobcnhvqaovzakkirneix 2457725 2457724 2026-09-15T20:10:35Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading A:liberica-ulsfo (T436363) 2457725 wikitext text/x-wiki 43uacc40kffvkpkjqrk2ic363hiwkgv 2457726 2457725 2026-09-15T20:11:40Z Stashbot 7414 arlolra@deploy1003: lwatson, arlolra: Backport for [[gerrit:1339741|Enable ReaderExperiments in eswiki, jawiki, and ptwiki (T438009)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457726 wikitext text/x-wiki b9ej1nw0925z1619qhtfwt9k5e34b0a 2457727 2457726 2026-09-15T20:16:34Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T436363) 2457727 wikitext text/x-wiki fecntrrmaazhtmtszpkmzic94zdmrt3 2457728 2457727 2026-09-15T20:16:39Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin depooling P{lvs4008.ulsfo.wmnet} and A:liberica 2457728 wikitext text/x-wiki lbnflbn3bezo9mq8dgup83sh0ohb0eu 2457729 2457728 2026-09-15T20:16:52Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs4008.ulsfo.wmnet} and A:liberica 2457729 wikitext text/x-wiki 3of48e2h8au8pr2hn5gxtqf07acd6mz 2457730 2457729 2026-09-15T20:16:58Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=1; selector: name=cp1103.eqiad.wmnet 2457730 wikitext text/x-wiki 75gs2p8b2rpjaa4w6l98p95bclgc04e 2457731 2457730 2026-09-15T20:17:01Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin pooling P{lvs4008.ulsfo.wmnet} and A:liberica 2457731 wikitext text/x-wiki 6aza773gdw5pqqayek3dn4tcerbl5pc 2457732 2457731 2026-09-15T20:17:04Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1103.eqiad.wmnet 2457732 wikitext text/x-wiki nr4mse1kbc5cejnxqrybz5engmes0aw 2457733 2457732 2026-09-15T20:17:13Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=1; selector: name=cp3075.esams.wmnet 2457733 wikitext text/x-wiki s7s7bajf8fovxlfevwu16k2jww36ict 2457734 2457733 2026-09-15T20:17:18Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp3075.esams.wmnet 2457734 wikitext text/x-wiki mkn8qaup88v05n1fmjhsemhbqv8kl0s 2457735 2457734 2026-09-15T20:17:24Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs4008.ulsfo.wmnet} and A:liberica 2457735 wikitext text/x-wiki sy29iy87ptgiv0pco9p2cs00r8nozx1 2457736 2457735 2026-09-15T20:17:44Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin depooling P{lvs4009.ulsfo.wmnet} and A:liberica 2457736 wikitext text/x-wiki sr7u65v2ojcdjizfyqsyq61t46o56kf 2457737 2457736 2026-09-15T20:17:56Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs4009.ulsfo.wmnet} and A:liberica 2457737 wikitext text/x-wiki 2wnau1fmkr99se99k8mzvz2cjw4jj9r 2457738 2457737 2026-09-15T20:18:19Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin pooling P{lvs4009.ulsfo.wmnet} and A:liberica 2457738 wikitext text/x-wiki e3b5ujtv7ufiopq092wzx1rrwmi3dvn 2457739 2457738 2026-09-15T20:18:45Z Stashbot 7414 arlolra@deploy1003: lwatson, arlolra: Continuing with deployment 2457739 wikitext text/x-wiki jr6ohnv879xbk5uqx86old1gl43xa3m 2457740 2457739 2026-09-15T20:18:46Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs4009.ulsfo.wmnet} and A:liberica 2457740 wikitext text/x-wiki anw25yovoqz13gxi2aysg89lhmzn2ei 2457741 2457740 2026-09-15T20:19:06Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin depooling P{lvs4010.ulsfo.wmnet} and A:liberica 2457741 wikitext text/x-wiki lmkflf5tqnydcro9j1vuc59dtjal42l 2457742 2457741 2026-09-15T20:19:18Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs4010.ulsfo.wmnet} and A:liberica 2457742 wikitext text/x-wiki qdzog7yd1tvsecxv8ryvl19hhws5lgw 2457743 2457742 2026-09-15T20:19:27Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin pooling P{lvs4010.ulsfo.wmnet} and A:liberica 2457743 wikitext text/x-wiki ijd4ru0rgb1kdrs14id209cef2f3824 2457744 2457743 2026-09-15T20:19:50Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs4010.ulsfo.wmnet} and A:liberica 2457744 wikitext text/x-wiki 3364ok70nf0i6gf5eyvboayqnbdhdlu 2457745 2457744 2026-09-15T20:19:55Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T436363) 2457745 wikitext text/x-wiki h1bfxy653smx3opedokft30g1mfh4vy 2457746 2457745 2026-09-15T20:20:58Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2046.codfw.wmnet with reason: host reimage 2457746 wikitext text/x-wiki nnjz42mk6ak3g4ckuz1m659o1uuo419 2457747 2457746 2026-09-15T20:23:17Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1339741|Enable ReaderExperiments in eswiki, jawiki, and ptwiki (T438009)]] (duration: 15m 58s) 2457747 wikitext text/x-wiki 0yz1810gqphe6sfzk8sgw2ydg3ffg5n 2457748 2457747 2026-09-15T20:24:07Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2046.codfw.wmnet with reason: host reimage 2457748 wikitext text/x-wiki n4undemh5q7x7bg75uson7jjs5wlpvq 2457749 2457748 2026-09-15T20:30:57Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6001.* 2457749 wikitext text/x-wiki opbwz35w4rhzseuhhc58v0s6jsvpg0b 2457750 2457749 2026-09-15T20:34:25Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.provision for host cp6001.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457750 wikitext text/x-wiki 82h38p3edfy1y0axpm4x6u8jnxqya18 2457751 2457750 2026-09-15T20:38:53Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1341271|Parsoid Read Views: Enable on all namespaces on wikitech (labswiki) (T437916)]] 2457751 wikitext text/x-wiki 8q3w3amgw2lyg120jt68004qqwbm1mm 2457753 2457751 2026-09-15T20:43:00Z Stashbot 7414 arlolra@deploy1003: ssastry, arlolra: Backport for [[gerrit:1341271|Parsoid Read Views: Enable on all namespaces on wikitech (labswiki) (T437916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457753 wikitext text/x-wiki g948cikuhpvo33rnk7cff9t06ks26jh 2457754 2457753 2026-09-15T20:43:33Z Stashbot 7414 arlolra@deploy1003: ssastry, arlolra: Continuing with deployment 2457754 wikitext text/x-wiki 0y0hwezl6696stlr5j8ugjdqb4d2g6p 2457756 2457754 2026-09-15T20:47:56Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2046.codfw.wmnet with OS trixie 2457756 wikitext text/x-wiki 6455d75hc6simbpowmsg5vogqag7wpg 2457757 2457756 2026-09-15T20:48:05Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341271|Parsoid Read Views: Enable on all namespaces on wikitech (labswiki) (T437916)]] (duration: 09m 11s) 2457757 wikitext text/x-wiki 5ardu0081mvorg3ol5qkrczz6c0dedw 2457758 2457757 2026-09-15T20:51:00Z Stashbot 7414 cdobbins@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2046.codfw.wmnet 2457758 wikitext text/x-wiki 0q4x5ggk7uxuv4fj8c3isiaqlw74irw 2457759 2457758 2026-09-15T20:51:14Z Stashbot 7414 cdobbins@puppetserver1001: conftool action : set/weight=1; selector: name=cp2046.codfw.wmnet 2457759 wikitext text/x-wiki ahxyydo5fmhyasdj1xv44k4ccgibx20 2457762 2457759 2026-09-15T21:08:33Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342049|MobileFrontend: Add app icons (T434258)]] 2457762 wikitext text/x-wiki sohf7j31347ayz4uc525zezdpe62j20 2457763 2457762 2026-09-15T21:12:39Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp6001.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457763 wikitext text/x-wiki 8awzg96naifanerl92xycb7p5ipazbu 2457764 2457763 2026-09-15T21:12:49Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson, cklimas: Backport for [[gerrit:1342049|MobileFrontend: Add app icons (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457764 wikitext text/x-wiki dxmfe9zcfvp7fnz419659vlwycabfsr 2457765 2457764 2026-09-15T21:13:09Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.reimage for host cp6001.drmrs.wmnet with OS trixie 2457765 wikitext text/x-wiki mlzdhwbo5u3xkgj8x5f80ncxlow82rt 2457768 2457765 2026-09-15T21:15:50Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson, cklimas: Continuing with deployment 2457768 wikitext text/x-wiki cbcuh3biebjfo2g2rdtq01s2n7zu36r 2457769 2457768 2026-09-15T21:20:20Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342049|MobileFrontend: Add app icons (T434258)]] (duration: 11m 47s) 2457769 wikitext text/x-wiki ay5cocjfzfmswbioeyu0ujmfoey3a9p 2457770 2457769 2026-09-15T21:30:05Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp6001.drmrs.wmnet with reason: host reimage 2457770 wikitext text/x-wiki q89ajd1lmchwaqff0jtf487zo9idgxn 2457771 2457770 2026-09-15T21:34:00Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6001.drmrs.wmnet with reason: host reimage 2457771 wikitext text/x-wiki fl025dsetwy3ruk43d3z12byqhxvac8 2457772 2457771 2026-09-15T21:48:26Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2457772 wikitext text/x-wiki 31uxrost589swzz80im9yxlskwws81b 2457773 2457772 2026-09-15T21:48:33Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2457773 wikitext text/x-wiki lyrmt3varxdfz05x6xixbnbuh6fuosg 2457774 2457773 2026-09-15T21:52:46Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2457774 wikitext text/x-wiki 9kvk8tn28mpj5ivwsnagwt904v2clut 2457775 2457774 2026-09-15T21:52:50Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2457775 wikitext text/x-wiki 5uqlcf1kkq4p8fhq7li6ecxtiuomb9j 2457776 2457775 2026-09-15T21:53:27Z Stashbot 7414 rzl@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2457776 wikitext text/x-wiki l4u2ilhqytt1t2o6696f0tlvl6shwp2 2457777 2457776 2026-09-15T21:53:32Z Stashbot 7414 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2457777 wikitext text/x-wiki mt62gltehgtrlgp2ru4h9w4glt7boia 2457778 2457777 2026-09-15T21:53:52Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2457778 wikitext text/x-wiki dc5e0ga13s17emx8dclailzq0yww5xn 2457779 2457778 2026-09-15T21:53:57Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2457779 wikitext text/x-wiki ngdnykye11ootq9f1tf5y6a9i3ajsk3 2457780 2457779 2026-09-15T21:55:27Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2457780 wikitext text/x-wiki s47460d9rcif8s0yh52733yis92ojz3 2457781 2457780 2026-09-15T21:55:31Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2457781 wikitext text/x-wiki ltr7xi50j6gw8ivqxatcbwwq6p5x5bl 2457782 2457781 2026-09-15T21:57:53Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6001.drmrs.wmnet with OS trixie 2457782 wikitext text/x-wiki 2c9g63potdot9ylbnrn5oo8xnojz4ee 2457784 2457782 2026-09-15T22:01:02Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS trixie 2457784 wikitext text/x-wiki qm4xbkzows4per9jooylfsxvi4su69g 2457786 2457784 2026-09-15T22:01:32Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.hosts.move-vlan for host mc-wf1002 2457786 wikitext text/x-wiki mxi02ruv5ewkiyjbjl06u3v5g3aw9i1 2457787 2457786 2026-09-15T22:02:07Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.dns.netbox 2457787 wikitext text/x-wiki c5m82qg9f60vvvqhjfb2uh4nvdt18fz 2457790 2457787 2026-09-15T22:07:13Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host mc-wf1002 - rzl@cumin2003" 2457790 wikitext text/x-wiki ku0zpw3252qctte71ab3uzxknd8h7gn 2457791 2457790 2026-09-15T22:07:17Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host mc-wf1002 - rzl@cumin2003" 2457791 wikitext text/x-wiki 6n5c4thiy39j6me954c8tdga4xb5uk6 2457792 2457791 2026-09-15T22:07:19Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2457792 wikitext text/x-wiki qy5gzphxjm7heusim1q6cfvv3keqyet 2457793 2457792 2026-09-15T22:07:20Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.dns.wipe-cache mc-wf1002.eqiad.wmnet 142.48.64.10.in-addr.arpa 2.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2457793 wikitext text/x-wiki iyign81ya2nuerqdhet5jl0bo6s67gv 2457794 2457793 2026-09-15T22:07:22Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mc-wf1002.eqiad.wmnet 142.48.64.10.in-addr.arpa 2.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2457794 wikitext text/x-wiki 1dtu420c4qq21lg3ylccmzol2x4djh4 2457795 2457794 2026-09-15T22:07:23Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc-wf1002 2457795 wikitext text/x-wiki s4fvy5joe6qyccyye8k75k2u3lw2a1a 2457796 2457795 2026-09-15T22:07:54Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-wf1002 2457796 wikitext text/x-wiki bspt0y43pmp1sx5y0c5zg3j1gc891mt 2457797 2457796 2026-09-15T22:07:55Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host mc-wf1002 2457797 wikitext text/x-wiki 64v4hr5yzdzcfbfi54i2q05703ravg4 2457798 2457797 2026-09-15T22:26:37Z Stashbot 7414 rzl@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage 2457798 wikitext text/x-wiki mmu6v5lrf1i8iyhtmb6f3a6z0xl4d1r 2457799 2457798 2026-09-15T22:32:53Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6001.* 2457799 wikitext text/x-wiki 7xqwvvqnb46rq0zifuox7zjltnqa27f 2457800 2457799 2026-09-15T22:33:20Z Stashbot 7414 brett@puppetserver1001: conftool action : set/weight=100; selector: name=cp6001.* 2457800 wikitext text/x-wiki smjrurouzjo3l4ooiwmqavm0qdnlt8o 2457801 2457800 2026-09-15T22:33:28Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage 2457801 wikitext text/x-wiki pn983sp1o5og6imjyv17atxghxardwm 2457807 2457801 2026-09-15T22:43:22Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-drmrs (T436363) 2457807 wikitext text/x-wiki 8ftfwzczqcegb41hw3oal3hu75ephfr 2457808 2457807 2026-09-15T22:43:37Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin depooling P{lvs6001.drmrs.wmnet} and A:liberica 2457808 wikitext text/x-wiki bk2ihv5mdpkdfsit36mlkutcv1bmvad 2457809 2457808 2026-09-15T22:43:53Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs6001.drmrs.wmnet} and A:liberica 2457809 wikitext text/x-wiki n1bernwhirb957xaqw86km4ksjtmw48 2457810 2457809 2026-09-15T22:44:08Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin pooling P{lvs6001.drmrs.wmnet} and A:liberica 2457810 wikitext text/x-wiki 2jhxv9s9i5993nqnw84l3e5ocndt3mk 2457811 2457810 2026-09-15T22:44:34Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs6001.drmrs.wmnet} and A:liberica 2457811 wikitext text/x-wiki cffwcd1s4mkzkg3idhau1l2km3m8sw7 2457812 2457811 2026-09-15T22:44:54Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin depooling P{lvs6002.drmrs.wmnet} and A:liberica 2457812 wikitext text/x-wiki oszpoepb079swi86s60ygt2469p6ted 2457813 2457812 2026-09-15T22:45:09Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs6002.drmrs.wmnet} and A:liberica 2457813 wikitext text/x-wiki 4swmqsud1yfm0ok6r1ddymkfk58go48 2457814 2457813 2026-09-15T22:45:25Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin pooling P{lvs6002.drmrs.wmnet} and A:liberica 2457814 wikitext text/x-wiki pmm0tsr9ggnw0v5tdgpwthatkp6pnxz 2457815 2457814 2026-09-15T22:45:50Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs6002.drmrs.wmnet} and A:liberica 2457815 wikitext text/x-wiki sgru2eihhb7jl5eugx0o05dynijl9eh 2457816 2457815 2026-09-15T22:46:10Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin depooling P{lvs6003.drmrs.wmnet} and A:liberica 2457816 wikitext text/x-wiki esvqr5gx0k4mlnkdmw5msdzrutcjqsq 2457817 2457816 2026-09-15T22:46:25Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs6003.drmrs.wmnet} and A:liberica 2457817 wikitext text/x-wiki 32f78buk5xft76hx1yo15xclllpoypr 2457818 2457817 2026-09-15T22:46:40Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.loadbalancer.admin pooling P{lvs6003.drmrs.wmnet} and A:liberica 2457818 wikitext text/x-wiki 5r7idtud9z9djksxbp6uqepi2tjb8c1 2457819 2457818 2026-09-15T22:46:55Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs6003.drmrs.wmnet} and A:liberica 2457819 wikitext text/x-wiki 0gtyntehxdrgqdntfdiihg0pa8eheez 2457820 2457819 2026-09-15T22:47:00Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-drmrs (T436363) 2457820 wikitext text/x-wiki d6qfhy1mepyuf0g9fvz8t50qjgbnozn 2457821 2457820 2026-09-15T22:50:22Z Stashbot 7414 rzl@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1002.eqiad.wmnet with OS trixie 2457821 wikitext text/x-wiki 6zp75q5urjv6rz6tt8vh9e9vghyq63m 2457822 2457821 2026-09-15T22:57:49Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=1; selector: name=cp6001.drmrs.wmnet,service=cdn 2457822 wikitext text/x-wiki 3rvmg2ppbr47yvc8thfp5nugaxmqmkc 2457823 2457822 2026-09-15T23:06:26Z Stashbot 7414 rzl@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2457823 wikitext text/x-wiki e7r2x58qsxsu77yjijo4591vy7toie6 2457824 2457823 2026-09-15T23:06:33Z Stashbot 7414 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2457824 wikitext text/x-wiki dan1xvy734sqao35jdxrqung6sqn6sw 2457825 2457824 2026-09-15T23:07:12Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2457825 wikitext text/x-wiki eya2u7fdodkfuaairh6nynxa7ywe15a 2457826 2457825 2026-09-15T23:07:19Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2457826 wikitext text/x-wiki 9ox53ec96gmlrbxwpj7btsjmdmj9ujn 2457827 2457826 2026-09-15T23:07:23Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2457827 wikitext text/x-wiki 2qacu7tp4ax67e28j9k2bldrn27e6uk 2457828 2457827 2026-09-15T23:07:28Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2457828 wikitext text/x-wiki fesdfpdsid19qjpwi1tnw0kyllrb925 2457829 2457828 2026-09-15T23:08:41Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2457829 wikitext text/x-wiki 63aaevlgpvbw477f7srlu5iz8vpuksi 2457830 2457829 2026-09-15T23:08:48Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2457830 wikitext text/x-wiki a6dzl5ioe6hc7q02vib6blukhfhb7ms 2457831 2457830 2026-09-15T23:08:56Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2457831 wikitext text/x-wiki im1bz17zi661cn5uk4qqzf4p6y8c5kp 2457832 2457831 2026-09-15T23:09:00Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2457832 wikitext text/x-wiki 5kdi7xwot00rsjsjsltwnvz3laqahsu 2457856 2457832 2026-09-16T02:01:06Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2457856 wikitext text/x-wiki kd4lfdjri4yx31r30pwnfltcurp34q7 2457857 2457856 2026-09-16T02:08:42Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) 2457857 wikitext text/x-wiki fs4qqtws6iqg27es7y265434agae4gr 2457860 2457857 2026-09-16T04:22:52Z Stashbot 7414 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2457860 wikitext text/x-wiki e94dzeei6keq2tezo5w8nkk3upjypka 2457861 2457860 2026-09-16T04:22:55Z Stashbot 7414 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2457861 wikitext text/x-wiki o8yy067o3aheeyipyepfdlmc8xe1dav 2457862 2457861 2026-09-16T04:24:35Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2457862 wikitext text/x-wiki j5h3x1ahrdju4wvo6xn3lxru382rw20 2457863 2457862 2026-09-16T04:24:37Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2457863 wikitext text/x-wiki qcpnq03by0j5yhyjwyj1ely80e8h6u8 2457864 2457863 2026-09-16T04:45:09Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply 2457864 wikitext text/x-wiki k41irvaka1gqnzyxzt2ws4stsr5nte6 2457865 2457864 2026-09-16T04:45:33Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply 2457865 wikitext text/x-wiki qhep2zz9qzge5ym68q66oyx3yiua6ep 2457866 2457865 2026-09-16T04:49:51Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply 2457866 wikitext text/x-wiki jjbaawtzn3dg3onpz1zq9tdmqw5kkig 2457867 2457866 2026-09-16T04:50:23Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply 2457867 wikitext text/x-wiki 03uzpb8ldg8qzudz534h5lp7s7szrx6 2457868 2457867 2026-09-16T04:54:21Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply 2457868 wikitext text/x-wiki akni38yb6y1qmqndxcd8escf3xqet0s 2457869 2457868 2026-09-16T04:54:55Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply 2457869 wikitext text/x-wiki rtftptp4ltbm60aniavy2kwf1ti6enl 2457870 2457869 2026-09-16T04:56:17Z Stashbot 7414 kart_: Updated Apertium to 2026-09-15-084320-production (T437213) 2457870 wikitext text/x-wiki p5988xduz2ossnlr7okkjuzreyl9x99 2457871 2457870 2026-09-16T05:12:32Z Stashbot 7414 moritzm: pruned obsolete Bullseye image buildkitd from the docker registry T416452 2457871 wikitext text/x-wiki erzzew0v3bpejpal42jyqqrl02e65je 2457878 2457871 2026-09-16T06:37:28Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2457878 wikitext text/x-wiki gxu9yxsszpv2wy3mbhzr9bw1pjowc6c 2457879 2457878 2026-09-16T06:47:56Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2457879 wikitext text/x-wiki pr5qdzv8ur6741w9tc9na3t5nuolkla 2457882 2457879 2026-09-16T06:50:26Z Stashbot 7414 moritzm: installing sudo security updates 2457882 wikitext text/x-wiki cgdufs9y7ydskvbhs27hmdmm4pqpzkt 2457885 2457882 2026-09-16T07:09:54Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1342112|Adds an instrument for pre-image-carousel-retest (T437076)]], [[gerrit:1342113|Adds an instrument for pre-image-carousel-retest (T437076)]], [[gerrit:1342117|Set up instrument for 5-arm test (T437076)]], [[gerrit:1342118|Set up instrument for 5-arm test (T437076)]] 2457885 wikitext text/x-wiki bubztvx0sq0ox1u8rnpn2xfhn57xhvg 2457886 2457885 2026-09-16T07:15:41Z Stashbot 7414 mlitn@deploy1003: mlitn: Backport for [[gerrit:1342112|Adds an instrument for pre-image-carousel-retest (T437076)]], [[gerrit:1342113|Adds an instrument for pre-image-carousel-retest (T437076)]], [[gerrit:1342117|Set up instrument for 5-arm test (T437076)]], [[gerrit:1342118|Set up instrument for 5-arm test (T437076)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be veri 2457886 wikitext text/x-wiki dwbafq7027nifkf5y0tcswysiggp00k 2457887 2457886 2026-09-16T07:16:12Z Stashbot 7414 mlitn@deploy1003: mlitn: Continuing with deployment 2457887 wikitext text/x-wiki t82898p1lnpmyjwvtw298sqzk35mm9t 2457888 2457887 2026-09-16T07:20:51Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342112|Adds an instrument for pre-image-carousel-retest (T437076)]], [[gerrit:1342113|Adds an instrument for pre-image-carousel-retest (T437076)]], [[gerrit:1342117|Set up instrument for 5-arm test (T437076)]], [[gerrit:1342118|Set up instrument for 5-arm test (T437076)]] (duration: 10m 56s) 2457888 wikitext text/x-wiki sfzm16oprop1d3g3fpjl1p4l3ybw9tw 2457890 2457888 2026-09-16T07:34:20Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2457890 wikitext text/x-wiki p54eabkol9th7miihux61x7spap4b03 2457891 2457890 2026-09-16T07:34:52Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2457891 wikitext text/x-wiki ra4k39qdsnv9b0f17ehxvz0ftds7ufa 2457893 2457891 2026-09-16T07:54:13Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw 2457893 wikitext text/x-wiki 15s4q3anvgojuvng450wo356i1o0tcc 2457894 2457893 2026-09-16T07:58:33Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2457894 wikitext text/x-wiki 12dr1iht04xjbji9qjigbhfj86mkcyl 2457895 2457894 2026-09-16T07:59:30Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2457895 wikitext text/x-wiki do8tkerpoxyynkic9om55y0gyxyc83t 2457896 2457895 2026-09-16T07:59:31Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw 2457896 wikitext text/x-wiki 6ckega5l30gwth0jaotby7c4nlap61k 2457897 2457896 2026-09-16T08:07:21Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad 2457897 wikitext text/x-wiki 9tmr9sqfo85ejxyc6k8ts6dehlxql5e 2457898 2457897 2026-09-16T08:10:37Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2457898 wikitext text/x-wiki g1g7jl2acx8guwp2wd903tn4v0pptx8 2457899 2457898 2026-09-16T08:11:29Z Stashbot 7414 jnuche@deploy1003: Rolling back deployment 2457899 wikitext text/x-wiki caqf4w77sba3u18rux6j6w32liv6jar 2457900 2457899 2026-09-16T08:11:44Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2457900 wikitext text/x-wiki nwp51hx79n8sldevmwyj0hq2jk90tjx 2457901 2457900 2026-09-16T08:11:45Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad 2457901 wikitext text/x-wiki 31p8fvsmneow8xkftsyf8nl97ix2epn 2457902 2457901 2026-09-16T08:14:06Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.20 refs T430839 2457902 wikitext text/x-wiki 14vdk5a1et5njyy6orpz9jpm7eb0qz7 2457903 2457902 2026-09-16T08:22:09Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2457903 wikitext text/x-wiki 0fhvkl90650vduoocirsf1r2hmqv5w3 2457904 2457903 2026-09-16T08:22:38Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2457904 wikitext text/x-wiki 3v5zjeyntgsgqg893yjrlz7hhmdjl16 2457905 2457904 2026-09-16T08:24:51Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.20 refs T430839 2457905 wikitext text/x-wiki mz185dg2q00gy4mg0d28zkjepah656t 2457906 2457905 2026-09-16T08:34:35Z Stashbot 7414 fabfur: start depooling eqsin (T438052) 2457906 wikitext text/x-wiki i5m41a8b949nsf3plhsji75zbvy4dyq 2457907 2457906 2026-09-16T08:35:03Z Stashbot 7414 slyngshede@cumin1003: conftool action : set/pooled=no; selector: cluster=dnsbox,dc=eqsin 2457907 wikitext text/x-wiki bfis7mtmwa89tlimegk11q4sp4iag3j 2457908 2457907 2026-09-16T08:35:14Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: no reason specified, no task ID specified] 2457908 wikitext text/x-wiki 7ituzd3nthui0662o2mdhyzj98jdy5x 2457909 2457908 2026-09-16T08:35:38Z Stashbot 7414 slyngshede@cumin1003: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool eqsin [reason: no reason specified, no task ID specified] 2457909 wikitext text/x-wiki top4o6i1bh9ey6onhagrbs1cqsbu8ed 2457910 2457909 2026-09-16T08:36:17Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: depooling for maintainance, T438052] 2457910 wikitext text/x-wiki jopmpb7y2v8h0lcn0mwa2127x62kjxc 2457911 2457910 2026-09-16T08:36:26Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: depooling for maintainance, T438052] 2457911 wikitext text/x-wiki 81r1g5qnk4ivt0xa2ivcj1jjzvb0i9l 2457912 2457911 2026-09-16T08:40:02Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2457912 wikitext text/x-wiki goa7jiid5bw97bhexjwtpisswf8ysz9 2457913 2457912 2026-09-16T08:40:07Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2457913 wikitext text/x-wiki kefrb2vx3egxc147mw64zfpt66ogwrb 2457914 2457913 2026-09-16T08:54:42Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2457914 wikitext text/x-wiki evv2rwuvuc4wh3cp0bg5hot1sqywy32 2457915 2457914 2026-09-16T08:55:14Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2457915 wikitext text/x-wiki sgiab05uhq667pcf1zmcbi8etwmri3j 2457917 2457915 2026-09-16T09:09:22Z Stashbot 7414 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. 2457917 wikitext text/x-wiki k1w4hj4h5iu70rc5hyh9cqe39xg0bzi 2457918 2457917 2026-09-16T09:09:24Z Stashbot 7414 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. 2457918 wikitext text/x-wiki sj80km0wz1wwxe6825sdy4lfffokxvf 2457919 2457918 2026-09-16T09:10:18Z Stashbot 7414 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. 2457919 wikitext text/x-wiki tq46d7edenho583vk00jds3caebegor 2457920 2457919 2026-09-16T09:10:22Z Stashbot 7414 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. 2457920 wikitext text/x-wiki 8avpq45sprzdjhdzgiqrcp0ge3gfnpg 2457921 2457920 2026-09-16T09:11:53Z Stashbot 7414 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. 2457921 wikitext text/x-wiki 2ivly6k6ckxivou04u9cvv4d3wcmg8e 2457922 2457921 2026-09-16T09:11:58Z Stashbot 7414 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. 2457922 wikitext text/x-wiki d26o6u1wtmkv4hvxwgss6k9y5kfo6m5 2457923 2457922 2026-09-16T09:12:47Z Stashbot 7414 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. 2457923 wikitext text/x-wiki s4ekcfaqjd5ryw1fs8si4fn8uzh5nmy 2457924 2457923 2026-09-16T09:12:51Z Stashbot 7414 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. 2457924 wikitext text/x-wiki e3w26o2xqhr9rkrrssr91xpj07rv9mp 2457925 2457924 2026-09-16T09:15:47Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2457925 wikitext text/x-wiki tui0rfwddgljlv5xisktabjx781hmfb 2457926 2457925 2026-09-16T09:18:15Z Stashbot 7414 slyngshede@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.esams.wmnet 2457926 wikitext text/x-wiki fh4fpqamjh0uor23s021thdunj69a76 2457927 2457926 2026-09-16T09:18:28Z Stashbot 7414 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.esams.wmnet 2457927 wikitext text/x-wiki f25h05si2lay13kwzdc4rq3kj714uiy 2457928 2457927 2026-09-16T09:19:37Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. 2457928 wikitext text/x-wiki jfdv205em8358a9begpeup9x34lbbkv 2457929 2457928 2026-09-16T09:19:41Z Stashbot 7414 slyngshede@cumin1003: conftool action : set/pooled=no; selector: name=cp5025.eqsin.wmnet 2457929 wikitext text/x-wiki itxmxs7j8hy2fw6qp4ax8gmn34cipi6 2457930 2457929 2026-09-16T09:19:54Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. 2457930 wikitext text/x-wiki 2cmp5y91iw7jg2nundzvymu8hobfvsj 2457932 2457930 2026-09-16T09:29:02Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.hosts.provision for host cp5025.mgmt.eqsin.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457932 wikitext text/x-wiki hhxy7e35qkqxnsk2xn0fg2iyo1cqboe 2457933 2457932 2026-09-16T09:30:40Z Stashbot 7414 slyngshede@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp5025.mgmt.eqsin.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457933 wikitext text/x-wiki 29b8lbqkba5rajzadzx10i0591rbgft 2457934 2457933 2026-09-16T09:38:17Z Stashbot 7414 moritzm: imported routinator 0.15.2-1trixie to thirdparty/routinator T438122 2457934 wikitext text/x-wiki m5ni3koavdbleua24wf4qqtca0w84o3 2457935 2457934 2026-09-16T09:41:33Z Stashbot 7414 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2457935 wikitext text/x-wiki ha3eddft5wjkr4mej2oadva3excim9a 2457936 2457935 2026-09-16T09:41:57Z Stashbot 7414 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2457936 wikitext text/x-wiki nk8qai70ld1k7kdtzgp1f35mw3j9l69 2457937 2457936 2026-09-16T09:48:21Z Stashbot 7414 slyngshede@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cp5025.eqsin.wmnet with reason: reimaging 2457937 wikitext text/x-wiki dfu25snsrow5clak5yvx6jcqr807235 2457938 2457937 2026-09-16T09:49:17Z Stashbot 7414 slyngshede@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cp5025.eqsin.wmnet with reason: reimaging 2457938 wikitext text/x-wiki j9q9bocif5ce29vtyihzzwji29v9ey6 2457939 2457938 2026-09-16T09:58:53Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2457939 wikitext text/x-wiki 33pdv2k5fo1tc44xvgfjy6n6lcgq58e 2457940 2457939 2026-09-16T10:02:18Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2457940 wikitext text/x-wiki hwzn6x8o4zpeqkru42un2k10vtusgm6 2457941 2457940 2026-09-16T10:02:38Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2457941 wikitext text/x-wiki kmv1wl5oyc0fongy23vvakr515lp5w1 2457942 2457941 2026-09-16T10:10:59Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.hosts.provision for host cp5025.mgmt.eqsin.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457942 wikitext text/x-wiki hmvnuqzme8rvuz3ln1ptu31llj1ysrn 2457943 2457942 2026-09-16T10:22:07Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5025.mgmt.eqsin.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2457943 wikitext text/x-wiki svxpfj0hdl3vld690udx0wmepgk6ow5 2457944 2457943 2026-09-16T10:37:37Z Stashbot 7414 jnuche@deploy1003: Started scap sync-world: Backport for [[gerrit:1342210|Revert "Tell VisualEditor about the app web edit tags" (T437736 T438125)]] 2457944 wikitext text/x-wiki nidt5rcn8z45y7kff12as00s8eirjs4 2457945 2457944 2026-09-16T10:57:29Z Stashbot 7414 jnuche@deploy1003: jnuche: Backport for [[gerrit:1342210|Revert "Tell VisualEditor about the app web edit tags" (T437736 T438125)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457945 wikitext text/x-wiki shkqldgg67esd54z8d4vfualklzukl1 2457946 2457945 2026-09-16T10:57:57Z Stashbot 7414 jnuche@deploy1003: jnuche: Continuing with deployment 2457946 wikitext text/x-wiki mechj5scv9idxsp5czksotbwd4yk27f 2457947 2457946 2026-09-16T11:01:01Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2457947 wikitext text/x-wiki cb36wdeqdyos254gjns9ndo6iv8m48i 2457948 2457947 2026-09-16T11:01:42Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2457948 wikitext text/x-wiki auyvpny27fnh82bqdwt655h1a5f0ysk 2457949 2457948 2026-09-16T11:03:57Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2457949 wikitext text/x-wiki nfkt8pzzrevfqtvzqr7i73207xpixm4 2457950 2457949 2026-09-16T11:04:30Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2457950 wikitext text/x-wiki qfzh3wdf4ile4borb00mnwuisvznd8o 2457951 2457950 2026-09-16T11:05:04Z Stashbot 7414 marostegui@cumin1004: dbctl commit (dc=all): 'Remove db1180 from dbctl T437222', diff saved to https://phabricator.wikimedia.org/P96459 and previous config saved to /var/cache/conftool/dbconfig/20260916-110502-marostegui.json 2457951 wikitext text/x-wiki j45igk6768chtsstcdk5w8hscu9jwgi 2457952 2457951 2026-09-16T11:10:31Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.hosts.reimage for host cp5025.eqsin.wmnet with OS trixie 2457952 wikitext text/x-wiki ptz9fi5pdcpnk8v9fwt99fxykvznfuh 2457953 2457952 2026-09-16T11:10:52Z Stashbot 7414 jnuche@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342210|Revert "Tell VisualEditor about the app web edit tags" (T437736 T438125)]] (duration: 33m 14s) 2457953 wikitext text/x-wiki fpl01nbbzxb80hk8xtmbr1qedpt44nc 2457954 2457953 2026-09-16T11:11:46Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply 2457954 wikitext text/x-wiki eqbk0eccs4vn4l9wkayqxyhm5xnte4u 2457955 2457954 2026-09-16T11:11:55Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply 2457955 wikitext text/x-wiki fxymhddz3i2nytuppf7wcv57643la62 2457956 2457955 2026-09-16T11:16:41Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply 2457956 wikitext text/x-wiki tdv44ymynxl9us53dg76mrb4hfwsrq0 2457957 2457956 2026-09-16T11:17:02Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply 2457957 wikitext text/x-wiki hce99g5nwbaazmrfdf1trelcm2esg63 2457958 2457957 2026-09-16T11:19:32Z Stashbot 7414 moritzm: kicked off a new run of production-images-weekly-rebuild.service on build2004 (previously some leftovers of buster in the config prevented a complete run) 2457958 wikitext text/x-wiki fwignm97rjngml56ihe90dgnyv8kx9s 2457959 2457958 2026-09-16T11:20:05Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply 2457959 wikitext text/x-wiki ipr6c76ahaxtqbw06mmemeuerw5tub2 2457960 2457959 2026-09-16T11:20:32Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply 2457960 wikitext text/x-wiki cdyjdr2ldse6nitxw9nwqqy3ylv9jzf 2457961 2457960 2026-09-16T11:21:07Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply 2457961 wikitext text/x-wiki 2cqujx27q3ihumryq9uj14nb03wzamu 2457962 2457961 2026-09-16T11:21:33Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply 2457962 wikitext text/x-wiki 32vqu5nnf913hgqmj7q6hl2gjklkibw 2457963 2457962 2026-09-16T11:24:59Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.20 refs T430839 2457963 wikitext text/x-wiki mxynhtw5pqtm44l5iby4f0sanlnwsk6 2457964 2457963 2026-09-16T11:25:51Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2457964 wikitext text/x-wiki of6nvxtyhkn8mh0kxhhu98nc09ud7vk 2457965 2457964 2026-09-16T11:26:57Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2457965 wikitext text/x-wiki o004yhy01z84d53lrh6y10hms2hwqst 2457966 2457965 2026-09-16T11:27:25Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2457966 wikitext text/x-wiki dbj116bawepkk6rkxk9g212wd9cd1a7 2457967 2457966 2026-09-16T11:27:48Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2457967 wikitext text/x-wiki pkbtlmmf1yprwwxic281h0zbpufcdp0 2457968 2457967 2026-09-16T11:31:28Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply 2457968 wikitext text/x-wiki p3kgze07d6a6thtbz4ddijd62xfzge2 2457969 2457968 2026-09-16T11:31:58Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply 2457969 wikitext text/x-wiki 5x0mw7yuszka39d80o5v849tbhcylxi 2457970 2457969 2026-09-16T11:33:50Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply 2457970 wikitext text/x-wiki 34gqlcap63rkbukb1eynjhj1s4xo5yh 2457971 2457970 2026-09-16T11:34:14Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply 2457971 wikitext text/x-wiki c2c61s1y0d8yragyee90zj0x9x8z1ig 2457972 2457971 2026-09-16T11:35:44Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5025.eqsin.wmnet with reason: host reimage 2457972 wikitext text/x-wiki 183lj6jlgd7xu8ed2dmlbk2lmysl9ik 2457973 2457972 2026-09-16T11:39:08Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5025.eqsin.wmnet with reason: host reimage 2457973 wikitext text/x-wiki ibc4dw4650m9shvrunbkvqvo1xocuqk 2457974 2457973 2026-09-16T11:39:56Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1341284|IS/IS-labs: Set wmgUseModeratorToolkit default false (T431000)]] 2457974 wikitext text/x-wiki bhgai922e3d2oncsm0mxmdu5lc9mows 2457975 2457974 2026-09-16T11:41:35Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet 2457975 wikitext text/x-wiki 877azvtsyuylwigff7q9vyrfdawnanj 2457976 2457975 2026-09-16T11:44:26Z Stashbot 7414 samtar@deploy1003: samtar: Backport for [[gerrit:1341284|IS/IS-labs: Set wmgUseModeratorToolkit default false (T431000)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2457976 wikitext text/x-wiki 6dt6ful72zwtom4yrn3udyq8uf7zliu 2457977 2457976 2026-09-16T11:45:57Z Stashbot 7414 samtar@deploy1003: samtar: Continuing with deployment 2457977 wikitext text/x-wiki myj46utd3fpbpl3srt7i3bcrzsma9du 2457978 2457977 2026-09-16T11:46:19Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet 2457978 wikitext text/x-wiki k4tg7fu6g9jzl95xemf0jdo23pc1sgq 2457980 2457978 2026-09-16T11:50:36Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341284|IS/IS-labs: Set wmgUseModeratorToolkit default false (T431000)]] (duration: 10m 32s) 2457980 wikitext text/x-wiki mr45tf31b94ynff11wrfesv3nj4gvd8 0 8703 2457783 2438599 2026-09-15T21:59:05Z Krinkle 44 2457783 wikitext text/x-wiki m5h64nbrix8cut828il1ztg0ah0f33w 498 9925 2457681 2457173 2026-09-15T19:09:16Z Stashbot 7414 Southparkfan: add deployment-cp-text09 and deployment-cp-upload09 to 'cache_hosts' hiera key - T436468 2457681 wikitext text/x-wiki 8j8zibw0n41419a5akr8xg1gk4gzdvf 0 17290 2457632 2457400 2026-09-15T18:16:30Z Stashbot 7414 James_F: Zuul: [mediawiki/services/wikispeech/*] Archive the services, for T344741 and T438076 2457632 wikitext text/x-wiki g5fe1965n07s7xl5f72qo8pbs589kil 2457634 2457632 2026-09-15T18:19:17Z Stashbot 7414 James_F: Zuul: [wikidata/query-builder, wikidata/query/gui] Archive repos, for T438076 2457634 wikitext text/x-wiki 61myzmwkh980jc351m1pfuz27bapks1 2457641 2457634 2026-09-15T18:22:22Z Stashbot 7414 James_F: Zuul: [mediawiki/services/recommendation-api] Archive service, for T429123 and T438076 2457641 wikitext text/x-wiki le9y8o0ctwm7p8bxd1ejfpoifncgqty 2457642 2457641 2026-09-15T18:24:23Z Stashbot 7414 James_F: Zuul: [operations/container/miscweb] Archive repository, for T438076 2457642 wikitext text/x-wiki tdpat21gky4wdd6g5jxwfxws16mhibs 2457643 2457642 2026-09-15T18:26:11Z Stashbot 7414 James_F: Zuul: [node-rdkafka-factory, node-rdkafka-statsd] Archive repos, for T366611 and T438076. 2457643 wikitext text/x-wiki 00q3f2xqvn7gj6thtp4n1nhcjxg6kgi 2457645 2457643 2026-09-15T18:28:09Z Stashbot 7414 James_F: Zuul: [cloud/toolforge/*] Archive two Toolforge repositories, for T438076 2457645 wikitext text/x-wiki rsohxrx8t8twog1ieprn7e4xrct5cid 2457648 2457645 2026-09-15T18:29:49Z Stashbot 7414 James_F: Zuul: [research/landing-page] Archive repository, for T438076 2457648 wikitext text/x-wiki 7xrhm71ytozp2nf45r1991zxfcn6d1k 2457651 2457648 2026-09-15T18:32:10Z Stashbot 7414 James_F: Zuul: [integration/gear] Archive repository, for T289512 and T438076 2457651 wikitext text/x-wiki iqvzhni0oetryz550m4q534eonjekf1 2457654 2457651 2026-09-15T18:33:54Z Stashbot 7414 James_F: Zuul: [labs/tools/ldap] Archive repository, for T438076 2457654 wikitext text/x-wiki l4v59jdkypwrqvk5do3s7bezwmpufr1 2457680 2457654 2026-09-15T19:09:14Z Stashbot 7414 Southparkfan: add deployment-cp-text09 and deployment-cp-upload09 to 'cache_hosts' hiera key - T436468 2457680 wikitext text/x-wiki 3qz3vfws1rk3ren1mdxl7nnoldu0c1j 2457752 2457680 2026-09-15T20:39:37Z Stashbot 7414 James_F: Zuul: […/OOJsUIAjaxLogin] Drop the JavaScript documentation job, for T391706 2457752 wikitext text/x-wiki m4rdf3az04s5lp0wqoq33s5qoioafwc 2457755 2457752 2026-09-15T20:46:45Z Stashbot 7414 James_F: Docker: [quibble-bookworm, node24-test, node26-test] Drop jsduck etc. for T363905 2457755 wikitext text/x-wiki p0mid31iw51yiwbxivmbzmk0jpmbfjd 2457875 2457755 2026-09-16T06:15:08Z Stashbot 7414 hashar: Updating node jobs from NodeJS 26.4.0 to 27.8.2 | https://gerrit.wikimedia.org/r/c/integration/config/+/1342056 2457875 wikitext text/x-wiki 41jmn4j9q4loruct0n78co4469w03ne 498 30942 2457597 2457128 2026-09-15T17:21:23Z Stashbot 7414 godog: depool clouddumps1002 from dumps-nfs to debug wedged mountpoints 2457597 wikitext text/x-wiki 2ag6okrehl2r1zddqbhf377n2j49u2t 2457625 2457597 2026-09-15T18:07:37Z Stashbot 7414 godog: pool clouddumps1002 as the only active server 2457625 wikitext text/x-wiki dwrre8p56kggh2gw67vz4zph7cchptt 498 46253 2457631 2450732 2026-09-15T18:11:40Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: bin/stashbot.sh restart 2457631 wikitext text/x-wiki 8oifdv30v1rcoqqicpr0na8244bgzbt 498 55577 2457620 2451435 2026-09-15T17:46:03Z Stashbot 7414 godog: reboot paws-127d-mq2566hee6zs-node-0.paws to debug T438062 2457620 wikitext text/x-wiki etqqv1zgs45tq1spcr35fha7z5ahq7w 498 443261 2457457 2457165 2026-09-15T13:03:52Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457457 wikitext text/x-wiki ee2wjow56fnwwstdxrxkaw2bq83xkh7 2457467 2457457 2026-09-15T13:12:21Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) 2457467 wikitext text/x-wiki 9hgblud169etsgjnfu79sz1kmotvlvr 2457473 2457467 2026-09-15T13:17:32Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.remove_cluster_node 2457473 wikitext text/x-wiki dym2dz6jfoz3qlyz2fza0aenay57l2o 2457474 2457473 2026-09-15T13:18:11Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.remove_cluster_node (exit_code=99) 2457474 wikitext text/x-wiki 33gy69mucd5ky1vg8ujikfu441d04un 2457475 2457474 2026-09-15T13:18:54Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-etcd-2 2457475 wikitext text/x-wiki qyjav0k71a0kh6pw4rm1s7ccv5d96k5 2457477 2457475 2026-09-15T13:19:16Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-etcd-2 2457477 wikitext text/x-wiki 7uof1oc8d9lul0a1l3c8ed8q2ccrg0s 2457483 2457477 2026-09-15T13:27:19Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457483 wikitext text/x-wiki 6zjkywn4h83wbeccqftgnzyhtuw97ng 2457486 2457483 2026-09-15T13:36:06Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) 2457486 wikitext text/x-wiki 8rqxw2lp21f4i8r796xqdb23r8kgs2v 2457488 2457486 2026-09-15T13:38:35Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457488 wikitext text/x-wiki rf45gan4vwbsx72girvmncuxi8ec4or 2457489 2457488 2026-09-15T13:39:00Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) 2457489 wikitext text/x-wiki 6wnmwx2zgbgbhpp4cqliycmk7brry1g 2457490 2457489 2026-09-15T13:40:42Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457490 wikitext text/x-wiki iicpet6siosv2vtbtke538zuzkdw41k 2457491 2457490 2026-09-15T13:41:08Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) 2457491 wikitext text/x-wiki nbda0c94fwrcidb9a0xmdk3zy78pvms 2457537 2457491 2026-09-15T15:20:42Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457537 wikitext text/x-wiki edxz1bkdh1mnbk1ca12oss3ofguhoww 2457538 2457537 2026-09-15T15:21:05Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) 2457538 wikitext text/x-wiki qfq6z3kgu6t4vpnns7blsbdjawds3pc 2457539 2457538 2026-09-15T15:22:44Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457539 wikitext text/x-wiki n8z16uaxsbpcw5hjia8yhelm5g1dcjz 2457540 2457539 2026-09-15T15:23:09Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) 2457540 wikitext text/x-wiki cs3c5vzax9lehhb12bhbxl7dltkyw2h 2457561 2457540 2026-09-15T15:50:33Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457561 wikitext text/x-wiki p8omk7fivfrnh5q03eishkfzonpedcg 2457562 2457561 2026-09-15T15:50:56Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) 2457562 wikitext text/x-wiki 354ib2rmthwnvvrclot9bq6d5fb5s1g 2457563 2457562 2026-09-15T15:54:32Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457563 wikitext text/x-wiki m4bknrpxcdnwlc2zlin2lgxfffkivfi 2457566 2457563 2026-09-15T16:03:42Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.etcd.add_cluster_node (exit_code=0) 2457566 wikitext text/x-wiki dzjqgyv0yzgmj8dewx3gw5vofsrmana 2457598 2457566 2026-09-15T17:22:14Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.remove_cluster_node 2457598 wikitext text/x-wiki 74fdxtbatzvu6dlo3pw5z5oq4l89uy8 2457601 2457598 2026-09-15T17:22:57Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.etcd.remove_cluster_node (exit_code=99) 2457601 wikitext text/x-wiki hv2kja9p5ip097tnnnqknwna2krtvgk 2457603 2457601 2026-09-15T17:24:28Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.remove_cluster_node 2457603 wikitext text/x-wiki r4vkjc5wnha4tjpsdk2tv15em3cjuk8 2457606 2457603 2026-09-15T17:25:30Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.etcd.remove_cluster_node (exit_code=0) 2457606 wikitext text/x-wiki j1nmk77w7yls1ro6evettvghrexoq1p 2457609 2457606 2026-09-15T17:26:05Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.etcd.add_cluster_node 2457609 wikitext text/x-wiki jxlqmsig61yu9tygfcusbuh7w7noja5 2457616 2457609 2026-09-15T17:35:14Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.etcd.add_cluster_node (exit_code=0) 2457616 wikitext text/x-wiki 41qwzsup6alysygytbevomn9pwavn5f 0 447310 2457517 2455766 2026-09-15T14:45:48Z KReid-WMF 49588 Add description of TestKitchen components to intro 2457517 wikitext text/x-wiki d5q599onnfwc34z74d55v8a7tqllmih 0 449160 2457845 2457239 2026-09-16T00:02:39Z Dexbot 30554 Bot: Updating the report 2457845 wikitext text/x-wiki 77q7apx2uo4ui0xe1kmt1jxgarblkzr 0 451792 2457789 2119982 2026-09-15T22:07:04Z Krinkle 44 replace tech blog (which now redirects to Diff firehose), to perf category 2457789 wikitext text/x-wiki sg4s4ww0t41xyfbvcadimt2w2929ox9 0 453813 2457709 2233824 2026-09-15T19:52:53Z PFischer-WMF 38527 /* Service Level Indicators (SLIs) */ Availability SLI: replace the TODO with the actual definition (backend_failures_total excluding type=rejected, over request_time_seconds_count), add the measured 30d value, document what the SLI does not cover and how to re-measure it, and replace the deprecated Graphite dashboard link 2457709 wikitext text/x-wiki fophrw1ch4l53iho26hep7pv4nw50l7 116 453906 2457712 2456840 2026-09-15T19:57:13Z Gitlabaccountapprovalbot 37332 modalya was rejected. 2457712 wikitext text/x-wiki igq7krhbln4hvfbhlmnlq0tdp2g7h56 0 454267 2457788 2228115 2026-09-15T22:05:02Z Krinkle 44 2457788 wikitext text/x-wiki hgisvdzq3wc6kam289rxvittqe2yir1 0 455855 2457660 2379295 2026-09-15T18:43:22Z BCornwall-WMF 31521 /* Reconfigure the server to boot via UEFI */ UEFI is default now, note the legacy flag 2457660 wikitext text/x-wiki qxb6pzf2k32iz1i72ow8y5k8l2jaljr 0 456356 2457506 2445215 2026-09-15T14:29:40Z KReid-WMF 49588 /* Test Kitchen UI */ Change to reflect that experiment configuration in TK UI is no longer supported 2457506 wikitext text/x-wiki rgbyss3e3f0tzw02n93x7kfan911vcw 2457507 2457506 2026-09-15T14:32:11Z KReid-WMF 49588 /* Test Kitchen UI */ Add notes about API usage 2457507 wikitext text/x-wiki avpj305tas5k6o36q05mbuxkopvk4gq 0 459265 2457493 2402453 2026-09-15T13:46:18Z Tiziano Fogli 37623 2457493 wikitext text/x-wiki hjrm05ju1mvhx7szxv3sfajyttx9nwf 2457495 2457493 2026-09-15T13:51:43Z Tiziano Fogli 37623 /* Why are some error queries forced to have a default value in the manifests? */ 2457495 wikitext text/x-wiki 3idxa4ykwkn0yosnvwus4nex26b8210 2457496 2457495 2026-09-15T13:51:56Z Tiziano Fogli 37623 /* SLOs manifests, CI and tests */ 2457496 wikitext text/x-wiki 32tjw8gt1kqnktfj49pkqhblrqtgrch 0 459699 2457541 2457256 2026-09-15T15:23:39Z Stashbot 7414 ejegg: disabled donations queue consumer for civi update 2457541 wikitext text/x-wiki c6au1jborgntda8nfmtnxjhsqkmwlz4 2457542 2457541 2026-09-15T15:24:05Z Stashbot 7414 ejegg: fundraising civicrm upgraded from ab17ae45 to ea392ac2 2457542 wikitext text/x-wiki 4m65o265dzk4eio0w7fnnnrcsxm9w3i 2457559 2457542 2026-09-15T15:40:32Z Stashbot 7414 ejegg: re-enabled donations queue consumer 2457559 wikitext text/x-wiki rjraaujvuretelsan5i6zpgtox10oml 2457883 2457559 2026-09-16T06:51:49Z Stashbot 7414 eileen: civicrm upgraded from ea392ac2 to fc41fd51 2457883 wikitext text/x-wiki jpi7fqwwg7drew765moc55cp0y1hmlw 0 460089 2457785 2434966 2026-09-15T22:01:05Z EEvans (WMF) 5035 began work on a provisioning section (more needed...) 2457785 wikitext text/x-wiki 148ekce7brq54wzieoj0p7djp2kzozr 2457833 2457785 2026-09-15T23:11:57Z EEvans (WMF) 5035 configuring & deploying 2457833 wikitext text/x-wiki pw21i55bkg29aaiob60ijy0uuynd6ov 0 460424 2457565 2457385 2026-09-15T15:59:54Z RCoccioli (WMF) 7416 /* Observability */ Updated SLO items 2457565 wikitext text/x-wiki 3oke7bvjsw17lid6kh7jaqhiy2zymnd 2457571 2457565 2026-09-15T16:24:08Z RCoccioli (WMF) 7416 /* Observability */ Compiled 4.7 2457571 wikitext text/x-wiki oij6t7b339julziaw0cgapko13l7ru7 2 460586 2457873 2457303 2026-09-16T05:34:55Z SSapaty (WMF) 28955 Clean up 2457873 wikitext text/x-wiki s3ys7fkexwojyaltpyk03dnxcfpdr9p 0 460608 2457494 2456780 2026-09-15T13:51:31Z Ottomata 88 added appendix about import timestamps 2457494 wikitext text/x-wiki is27650kqkhggpz1thhxo7e10tczw2c 2457499 2457494 2026-09-15T14:12:30Z Ottomata 88 /* Service Level Indicators (SLIs) */ 2457499 wikitext text/x-wiki 6iolumadiqp0l2rd85nty0bvfoh5hqf 2457892 2457499 2026-09-16T07:50:02Z APizzata-WMF 47941 2457892 wikitext text/x-wiki rnhccom44linl6albc1sq4pv1alcvti 1 460617 2457604 2457119 2026-09-15T17:24:34Z KHerron (WMF) 10463 /* Do not comment on a patch without casting a vote */ Reply 2457604 wikitext text/x-wiki 17tzx26czqm0kue42y5ost6ptuj4ckd 2457618 2457604 2026-09-15T17:42:30Z KHerron (WMF) 10463 /* Ping them actively to get their attention once you’re ready for their review, to ensure they are available */ new section 2457618 wikitext text/x-wiki 9x3lubiwiygkbn1d5g372pljcwtshmp 2457714 2457618 2026-09-15T20:01:33Z BCornwall-WMF 31521 /* Do not comment on a patch without casting a vote */ Reply 2457714 wikitext text/x-wiki 9ee16hf80ig52jpbs6vderkjx8w5fjt 0 460624 2457471 2454806 2026-09-15T13:15:21Z MMuhlenhoff (WMF) 5268 /* Service Summary */ wikitech done 2457471 wikitext text/x-wiki p7imj397azd6hv4wrxxhisukzh8ji18 2457476 2457471 2026-09-15T13:18:56Z MMuhlenhoff (WMF) 5268 /* Service Summary */ DB review done 2457476 wikitext text/x-wiki lmiadskt63lmqo4dh50fcxcg4ishso0 2457872 2457476 2026-09-16T05:17:52Z MMuhlenhoff (WMF) 5268 /* Operating Procedures */ sign off some n/a bits 2457872 wikitext text/x-wiki 5w3bss13dg80j6kb9lpf3pfsrucyci6 2457884 2457872 2026-09-16T06:51:53Z MMuhlenhoff (WMF) 5268 /* Observability */ sign off o11y changes 2457884 wikitext text/x-wiki 7rhp7qe6vt35h1def85o29vdx551z2d 2457889 2457884 2026-09-16T07:29:19Z MMuhlenhoff (WMF) 5268 /* Release Confidence */ sign off release confidence entries 2457889 wikitext text/x-wiki hcv0xy2rmprj60ak32ctkyxv70a1dm2 2457979 2457889 2026-09-16T11:49:25Z MMuhlenhoff (WMF) 5268 /* Operating Procedures */ sign offs for operating procedure 2457979 wikitext text/x-wiki fzmhqvgi0sks95hzxduh9xysr5vtlql 2 460651 2457633 2456573 2026-09-15T18:18:58Z Ashissarkar001 59471 /* */ 2457633 wikitext text/x-wiki 3ytblhni1i2f8l77xzw7z4pvzjvh986 0 460667 2457760 2457253 2026-09-15T21:05:53Z AWesterinen-WMF 54189 /* Manual mode retrieving skos:altLabel */ 2457760 wikitext text/x-wiki q0gapo9qr16n1uiohisadrhjdin5su5 2457761 2457760 2026-09-15T21:06:36Z AWesterinen-WMF 54189 /* Manual mode retrieving skos:altLabel */ 2457761 wikitext text/x-wiki dmvokzezexuu5ht8kr91o9q3q5u510p 2457766 2457761 2026-09-15T21:14:36Z AWesterinen-WMF 54189 /* Manual mode retrieving rdfs:label and skos:altLabel */ 2457766 wikitext text/x-wiki jmn3odsagcmleun9gdjufwscyuwew2o 2457767 2457766 2026-09-15T21:15:02Z AWesterinen-WMF 54189 /* Manual mode retrieving several labels and a schema:description */ 2457767 wikitext text/x-wiki 4so8xwqjt815yhbompxcxx7vvjta3qh 2457843 2457767 2026-09-15T23:54:27Z AWesterinen-WMF 54189 /* Manual mode retrieving several labels and a schema:description */ 2457843 wikitext text/x-wiki c5b6cdjkvmjmc0oxdme5oo5ker4yhl8 2457852 2457843 2026-09-16T00:37:43Z AWesterinen-WMF 54189 /* Manual mode retrieving rdfs:label and skos:altLabel */ 2457852 wikitext text/x-wiki gzphdhe7soeeh1jvyt8ue1l13ym3wkv 2457853 2457852 2026-09-16T00:42:14Z AWesterinen-WMF 54189 /* Manual mode retrieving rdfs:label */ 2457853 wikitext text/x-wiki g8cp1d0cwprdj5ec9vsupmxsw6783tt 2457854 2457853 2026-09-16T00:44:40Z AWesterinen-WMF 54189 /* Manual mode retrieving several labels and a schema:description */ 2457854 wikitext text/x-wiki 65ahu6lr1s4j28qk780kmr1oa1dn02k 2457855 2457854 2026-09-16T00:52:42Z AWesterinen-WMF 54189 /* Manual mode retrieving rdfs:label and skos:altLabel */ 2457855 wikitext text/x-wiki np463p2t8cw1vhadi5n2dfnol8l0ur1 12 460669 2457568 2457351 2026-09-15T16:18:01Z SSapaty (WMF) 28955 /* Connecting to the replicas */ reformat code using 'code sample' template 2457568 wikitext text/x-wiki 4lr45k1zz694izb820mfzv7ilpe7q28 2457570 2457568 2026-09-15T16:23:40Z SSapaty (WMF) 28955 /* Connecting to the replicas */ html formatting 2457570 wikitext text/x-wiki tet5pist6sb3g5p67ntewv8l2fkvc9z 2457572 2457570 2026-09-15T16:32:06Z SSapaty (WMF) 28955 /* How to create a basic Python Flask webservice */ replace syntaxhightlight tag with code sample template 2457572 wikitext text/x-wiki gzol8gs6yx4nabdg7626fjxjp4l8ive 2457574 2457572 2026-09-15T16:37:10Z SSapaty (WMF) 28955 Code sample template for syntaxhighlight tag 2457574 wikitext text/x-wiki gqqpjg26lzf0xd7zfersgwdowtnkw0g 2457575 2457574 2026-09-15T16:40:30Z SSapaty (WMF) 28955 /* Connecting to the replicas */ 2457575 wikitext text/x-wiki a0f5dvzdl4kwszpadaysf1acze6srng 2457608 2457575 2026-09-15T17:25:55Z SSapaty (WMF) 28955 /* How to create a basic Python Flask webservice */ code formatting 2457608 wikitext text/x-wiki nwdzxm75wqqo1e4gy8jii5c9g1omg5f 2 460671 2457451 2026-09-15T12:50:22Z CKlimas-WMF 59190 Created page with "(This is a work-in-progress doc and may be inaccurate.) == What is app remote config? == Wikimedia mobile apps read some config from a remote URL,https://en.wikipedia.org/api/rest_v1/feed/configuration. This config allows changes to app behavior that are out-of-band of the normal app release process. Deployments of this feed typically happen [[Deployments|at 12:00 UTC on Tuesdays]], but can be done on an ad-hoc basis. Ops and SRE staff are primarily on IRC, not Slack,..." 2457451 wikitext text/x-wiki 9ydvj1ildp64oq8nk5xsunfkqpwhow5 2457452 2457451 2026-09-15T12:51:28Z CKlimas-WMF 59190 2457452 wikitext text/x-wiki ry2upnj3tj3k4wl4sjc47kx3jlmdb8v 2457468 2457452 2026-09-15T13:13:52Z CKlimas-WMF 59190 2457468 wikitext text/x-wiki d77nv7ytkcqjnrshus8g9aykqsb2p9r 1 460672 2457532 2026-09-15T15:11:17Z Ottomata 88 /* Andrew & Antonio draft review slack thread */ new section 2457532 wikitext text/x-wiki ee4rzfg61xxorrtta3wmdreozf8bne3 828 460673 2457834 2026-09-15T23:22:33Z MSK 47079 scap: now in scribunto 2457834 Scribunto text/plain hzqls0rheayah6ckrka2hhvskjati26 2457835 2457834 2026-09-15T23:24:45Z MSK 47079 oops 2457835 Scribunto text/plain 23hr0lphicz7dm6l7dvqmp0ypzj8ydi 2457836 2457835 2026-09-15T23:27:47Z MSK 47079 testing... 2457836 Scribunto text/plain 92hxwl05txe4bhzt4meuxe44cpftdzs 2457838 2457836 2026-09-15T23:31:54Z MSK 47079 2457838 Scribunto text/plain my0phariykvfgdk1jh2nx2rgcym7x0t 2457839 2457838 2026-09-15T23:37:16Z MSK 47079 2457839 Scribunto text/plain mm5638ird8hzh45wvia5wh6pbsrs9jw 2457840 2457839 2026-09-15T23:37:40Z MSK 47079 that broke... everything 2457840 Scribunto text/plain my0phariykvfgdk1jh2nx2rgcym7x0t 2457841 2457840 2026-09-15T23:40:50Z MSK 47079 fixed 2457841 Scribunto text/plain r6umcuyrxdpdy8s9ucrw5x81g5ilda9 2457842 2457841 2026-09-15T23:50:09Z MSK 47079 pre and suf 2457842 Scribunto text/plain rbl3sh8xv2qeyuzws6kxy04c9iuar0p 2457847 2457842 2026-09-16T00:08:07Z MSK 47079 actually random now 2457847 Scribunto text/plain fx6o7rgd255hlcj3d2lz2bs2e9oossk 2457848 2457847 2026-09-16T00:10:16Z MSK 47079 fix? 2457848 Scribunto text/plain lfh147jesjhjmdr7uqjd5ttrzyjvoy2 2457850 2457848 2026-09-16T00:13:10Z MSK 47079 beautify 2457850 Scribunto text/plain orvfnii6qm5wnkfjb5dfatukxm1b7wc 2457851 2457850 2026-09-16T00:28:02Z MSK 47079 min width 2457851 Scribunto text/plain h0qu5ihdtuvup6y6tdjlt3h45qqzz8y 2457858 2457851 2026-09-16T03:01:48Z MSK 47079 fmt 2457858 Scribunto text/plain km7rygpluf6lge0qhgjn6yhalksalx2 828 460674 2457837 2026-09-15T23:29:46Z MSK 47079 Created page with "{{#invoke:scapsay|main|words=scap: now in scribunto}}" 2457837 wikitext text/x-wiki k5cuf3d7uot8fnb9ykrrux57yox1z5y 2457844 2457837 2026-09-15T23:59:52Z MSK 47079 actual docs 2457844 wikitext text/x-wiki ske03mhexsd70nn53vnhbesg2g57ofp 2457849 2457844 2026-09-16T00:11:10Z MSK 47079 probably shouldn't have words 2457849 wikitext text/x-wiki 2lizmgvrcourtrvmz1oq3v9ffhpbjqk 2 460675 2457874 2026-09-16T05:35:29Z SSapaty (WMF) 28955 Created page with "PTD2 *[[User:SSapaty_(WMF)/PTD/tutorial1 | Tutorial 1]] *[[User:SSapaty_(WMF)/PTD/my first python tool | My first Python tool]] *[[User:SSapaty_(WMF)/PTD/migrate_jobs_to_ptd | Migrate jobs to ptd]] *[[User:SSapaty_(WMF)/PTD/migrate build service job to ptd | Migrate build service jobs to ptd]]" 2457874 wikitext text/x-wiki 31v9a3vf0dyk98wgo1376udppkc2hsr 2 460677 2457931 2026-09-16T09:24:47Z JGiannelos (WMF) 17083 Created page with "= Handover docs - Sep 2026 = [[File:Handover-jgiannelos-high-level-overview.png|center|thumb|800px| High level overview, LLM generated]] <div style="clear:both"></div> == Kubernetes == === Image building === All of our services are using [https://doc.wikimedia.org/releng/blubber/ blubber] definitions to build images. Post merge, CI builds images and publishes them in the registry. Blubber config uses buildkit so devs can use docker to build images locally. ==== Local b..." 2457931 wikitext text/x-wiki sash2nwda2m3pk5tfghoim6gcitgbld