Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.17 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 12 1568 2451614 2447575 2026-08-26T07:18:18Z Filippo Giunchedi 2939 /* Making /public/dumps available in containers */ 2451614 wikitext text/x-wiki nk33gne856s923v81ubjqytv8edvaj0 2451615 2451614 2026-08-26T07:18:49Z Filippo Giunchedi 2939 /* Making /public/dumps available in self-managed containers */ 2451615 wikitext text/x-wiki 64cc3mh8mk79hggsaa0fy9zik7qmlwj 2451620 2451615 2026-08-26T07:23:05Z Filippo Giunchedi 2939 /* Making /public/dumps available in self-managed containers */ 2451620 wikitext text/x-wiki 6quino16lpjo186h1gku28kxm2ac8qs 498 2161 2451444 2451343 2026-08-25T14:06:52Z Stashbot 7414 taavi: update metricsinfra scrape names to not reference nginx specifically T435917 2451444 wikitext text/x-wiki qt0hprg3sbgsr793bb4wjn9f9ms5q1a 2451470 2451444 2026-08-25T14:50:23Z Stashbot 7414 taavi: update metricsinfra scrape names to not reference nginx specifically T429930 2451470 wikitext text/x-wiki 82wjtz0fbg966gw112mnoxck1cxjn6x 498 3086 2451659 2451279 2026-08-26T09:17:04Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2451659 wikitext text/x-wiki i8rkclje9qsstl5oche23njfh2are8d 2451665 2451659 2026-08-26T09:27:17Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2451665 wikitext text/x-wiki tn31wsxmy5tzs7kym3vzpnswqtcqebv 2451687 2451665 2026-08-26T09:50:33Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2451687 wikitext text/x-wiki af0pxj4etvt6fy0vh4zfg8isnh24g30 2451696 2451687 2026-08-26T10:00:26Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2451696 wikitext text/x-wiki ez08mu19j5kjp2f3i7506bvpfjscejh 0 3763 2451582 1860251 2026-08-25T21:03:57Z Greg (WMF) 968 Use language from metawiki, and some cleaner phrasing (imo) 2451582 wikitext text/x-wiki bfelfyk2nncf0c1xwqfqsy7adz3jq67 0 4108 2451394 2451378 2026-08-25T12:25:15Z APatro (WMF) 38793 Will backport later. 2451394 wikitext text/x-wiki pzcghrvetmplc977149wma5uztijzz1 2451528 2451394 2026-08-25T17:51:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1329339]] to Tuesday, August 25 UTC late backport window 2451528 wikitext text/x-wiki ezo309od31yb24vmxrnivti5lbtoiyd 2451556 2451528 2026-08-25T19:46:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1328528]] to Tuesday, August 25 UTC late backport window 2451556 wikitext text/x-wiki toreksxfwazlkdjc6jtitfnf67slval 2451561 2451556 2026-08-25T20:21:21Z ScheduleDeploymentBot 37566 Add [[gerrit:1329369]] to Tuesday, August 25 UTC late backport window 2451561 wikitext text/x-wiki j2mpq0smz0rha5whjpdl4r7bioxk3uo 2451564 2451561 2026-08-25T20:25:24Z ScheduleDeploymentBot 37566 Add [[gerrit:1329370]] to Tuesday, August 25 UTC late backport window 2451564 wikitext text/x-wiki k7hw2l1bg40boioorac6wdiyu0zdnpk 2451590 2451564 2026-08-25T21:34:17Z ScheduleDeploymentBot 37566 Add [[gerrit:1329386]] to Wednesday, August 26 UTC morning backport window 2451590 wikitext text/x-wiki 1dsc5ogv0bre0vgrxjyrjyfg3at4hxz 2451592 2451590 2026-08-25T22:13:11Z ScheduleDeploymentBot 37566 Add [[gerrit:1329358]] to Wednesday, August 26 UTC morning backport window 2451592 wikitext text/x-wiki 43pcuc9trbtwxk1d1lo1q4yi6bxanlc 2451628 2451592 2026-08-26T08:04:42Z ScheduleDeploymentBot 37566 Add [[gerrit:1323180]] to Wednesday, August 26 UTC afternoon backport window 2451628 wikitext text/x-wiki j050qv1ygv4kdszfg1tmuou6jk3eo12 2451629 2451628 2026-08-26T08:05:00Z ScheduleDeploymentBot 37566 Add [[gerrit:1323180]] to Wednesday, August 26 UTC afternoon backport window 2451629 wikitext text/x-wiki 5ei6m0uske450jt4s9hl3i5bn5412hj 2451700 2451629 2026-08-26T10:05:00Z ScheduleDeploymentBot 37566 Add [[gerrit:1324031]] to Wednesday, August 26 UTC afternoon backport window 2451700 wikitext text/x-wiki avpk9kdikilc38ap5q1hh9e2u7e57ii 2451709 2451700 2026-08-26T10:19:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1323180]] to Wednesday, August 26 UTC afternoon backport window 2451709 wikitext text/x-wiki 1kqv94evfc7is047p3oeqcn3sjkwmj2 2451753 2451709 2026-08-26T11:56:29Z KartikMistry 1115 Cleanup. 2451753 wikitext text/x-wiki rviwtsxjsk7ysv7jjob1nrbcwzaxpfk 0 4296 2451611 2401112 2026-08-26T07:13:10Z Filippo Giunchedi 2939 /* Maintenance/downtime */ remove outdated info 2451611 wikitext text/x-wiki tb05tsq3sl4hln9aqymli07ujwl9moz 498 6726 2451647 2451286 2026-08-26T08:59:31Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2451647 wikitext text/x-wiki 56o15zuarxichnkzbju8y8gp6dxvcbw 2451648 2451647 2026-08-26T09:00:45Z Stashbot 7414 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api 2451648 wikitext text/x-wiki mywrzq46ie34q7br3h8wx89nm1lj5nt 2451651 2451648 2026-08-26T09:05:11Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2451651 wikitext text/x-wiki emmsuh28prvz9a7n4s9tgmzybyp6ou4 2451658 2451651 2026-08-26T09:14:16Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2451658 wikitext text/x-wiki 9lpg8ef96qo8k0h6x7t8fsn1u3a5ndy 2451676 2451658 2026-08-26T09:39:22Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2451676 wikitext text/x-wiki 8gj3xvccy1x5hew27bb20ckey0c7ee4 2451686 2451676 2026-08-26T09:49:15Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2451686 wikitext text/x-wiki ffhawfflgtw3fxoogno5dkahq3wg35v 0 7919 2451390 2451388 2026-08-25T12:13:32Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2451390 wikitext text/x-wiki 2b7qyxa3wl0flgexwbwch49fxlrlens 2451391 2451390 2026-08-25T12:13:49Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2451391 wikitext text/x-wiki 8591o74seb0uh1brwybraxxc5z86z8b 2451392 2451391 2026-08-25T12:24:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2451392 wikitext text/x-wiki qdekr3pwnyxv7b72k2iup48c3nos8yi 2451393 2451392 2026-08-25T12:24:16Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db-test1002.eqiad.wmnet 2451393 wikitext text/x-wiki 21ftdrer804z3lfdieu2lcvqzyj26dh 2451395 2451393 2026-08-25T12:29:00Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2903.codfw.wmnet with OS trixie 2451395 wikitext text/x-wiki m16fzck4flo5jc6poyuag2y3p3csbz1 2451396 2451395 2026-08-25T12:29:05Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host db2903.codfw.wmnet 2451396 wikitext text/x-wiki mf0zwvkx3y2t2mjnp8wzmuiluiuj9ew 2451397 2451396 2026-08-25T12:29:11Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2451397 wikitext text/x-wiki caq9uhhqg17s0ddcmo7xzwwxhy372kl 2451398 2451397 2026-08-25T12:33:00Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe 2451398 wikitext text/x-wiki s40pi49he74cdbgr7eihd64lr7kpvwa 2451399 2451398 2026-08-25T12:33:16Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2451399 wikitext text/x-wiki 5rztaznr5541swnoahspcp7p1wb0phr 2451400 2451399 2026-08-25T12:34:26Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2451400 wikitext text/x-wiki mj350uztyzdi7s0r5rzh9a37yd7oscd 2451401 2451400 2026-08-25T12:34:39Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling reboot on A:thanos-fe 2451401 wikitext text/x-wiki rj8m0i3xjzcvf77dkceu5jhmgam672z 2451402 2451401 2026-08-25T12:34:48Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2451402 wikitext text/x-wiki 5swtump06dg9zr9x4v5w8jn9dtt4ctu 2451403 2451402 2026-08-25T12:34:51Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2451403 wikitext text/x-wiki q6ds4umzfacgtshtmwzv2dhek4sc453 2451404 2451403 2026-08-25T12:34:54Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db-test1002.eqiad.wmnet 2451404 wikitext text/x-wiki e8gi2bl3zzjjffpic3u8i46pdq1u8f9 2451405 2451404 2026-08-25T12:40:26Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Delete ipv6 addr - fceratto@cumin1003" 2451405 wikitext text/x-wiki 6unvet2c82sjri6vbhhc2w4fl1tpwym 2451406 2451405 2026-08-25T12:40:30Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Delete ipv6 addr - fceratto@cumin1003" 2451406 wikitext text/x-wiki ad67gxeel430nogmxztid3xqrvol6tr 2451407 2451406 2026-08-25T12:40:34Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2451407 wikitext text/x-wiki 480uw3f089luqg2nwmbbt55rz765rz7 2451408 2451407 2026-08-25T12:42:59Z Stashbot 7414 fceratto@cumin1003: Removing db-test1002 from zarcillo T435912 2451408 wikitext text/x-wiki tvywg2df0xj2cwonj6luy2v42cjbrzj 2451409 2451408 2026-08-25T12:43:04Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2451409 wikitext text/x-wiki pwezmuljwmmk7rxiscufrpm5pix00st 2451410 2451409 2026-08-25T12:43:06Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) 2451410 wikitext text/x-wiki 4vc3exyx1hic4cu3md9vsnmpylikmzu 2451411 2451410 2026-08-25T12:44:16Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2451411 wikitext text/x-wiki 1eof6zucjzda8540u2clsbpzrnme0mf 2451412 2451411 2026-08-25T12:45:28Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2451412 wikitext text/x-wiki akfb75m9ff6y01uc6tbip80wgm2jvdp 2451413 2451412 2026-08-25T12:45:38Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2451413 wikitext text/x-wiki kjvgqb3hbd1g25w0hrtqdx0bwj2ickj 2451414 2451413 2026-08-25T12:49:04Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db-test1003.eqiad.wmnet 2451414 wikitext text/x-wiki 0vbcir3s4oawr9f7cz38l2u41uus9gy 2451415 2451414 2026-08-25T12:51:50Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe 2451415 wikitext text/x-wiki f5q97lhugzby8jmsnpasgddlqa5567b 2451416 2451415 2026-08-25T12:54:00Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2451416 wikitext text/x-wiki snch8r41cj8ah754pbdn9vl5t4dcwpm 2451417 2451416 2026-08-25T12:58:13Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2451417 wikitext text/x-wiki 3co2mtbegxa2187qxmg5s01n3pgt9p5 2451418 2451417 2026-08-25T13:01:33Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . 2451418 wikitext text/x-wiki eshn348lk8s1ws5dzvgfnqwexecguvc 2451419 2451418 2026-08-25T13:05:59Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2451419 wikitext text/x-wiki 3ss0wk9v8d03gd5jzmi9pp8nb6v27bw 2451420 2451419 2026-08-25T13:06:02Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2451420 wikitext text/x-wiki 5pf9wey63yusgr281eg4cdnf3cmmix4 2451421 2451420 2026-08-25T13:06:04Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db-test1003.eqiad.wmnet 2451421 wikitext text/x-wiki hgboc5glxng0zwlvyiwhyzlp34kn9wg 2451424 2451421 2026-08-25T13:14:37Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . 2451424 wikitext text/x-wiki oyz40svdj949eghzdlms7bnejfulyal 2451425 2451424 2026-08-25T13:14:45Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . 2451425 wikitext text/x-wiki ac0ai32kzrb2qmjuf04hb4ndqn9roj4 2451427 2451425 2026-08-25T13:19:26Z Stashbot 7414 moritzm: installing python3.13 security updates 2451427 wikitext text/x-wiki n2tf5i33g3pv0wfvlzrred26cdccm7y 2451428 2451427 2026-08-25T13:20:49Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad 2451428 wikitext text/x-wiki jcbvlknipeqoft60shyyzchj2rjn9ty 2451430 2451428 2026-08-25T13:23:20Z Stashbot 7414 godog: failover dumps-nfs from clouddumps1002 to clouddumps1001 - T411248 2451430 wikitext text/x-wiki g7mzdi9t87ygbmsb6tas4frjfnng88t 2451431 2451430 2026-08-25T13:24:19Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2451431 wikitext text/x-wiki st5fx4d2e3nbrdeshes6q268q4gpox2 2451432 2451431 2026-08-25T13:25:24Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2451432 wikitext text/x-wiki ppbgy9l7yqvjcvk6u8q6b2ber515tma 2451433 2451432 2026-08-25T13:25:26Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad 2451433 wikitext text/x-wiki cvlc3aybn681xqpx2aywkckr58n84n9 2451434 2451433 2026-08-25T13:28:09Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: cluster=urldownloader 2451434 wikitext text/x-wiki a1jb4ydheqbc1gx5w8qxj33931qlvud 2451436 2451434 2026-08-25T13:31:02Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling reboot on A:thanos-fe 2451436 wikitext text/x-wiki 9jvzzmsg1r0cgcmsu99dnnicv8llsgh 2451437 2451436 2026-08-25T13:31:30Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling reboot on A:thanos-fe 2451437 wikitext text/x-wiki 0gbcwhrc0ujbanikjmh88c92e0l4n6a 2451438 2451437 2026-08-25T13:37:34Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=no; selector: name=dns2005.* [reason: trixie upgrade] 2451438 wikitext text/x-wiki r47tpv8hma6yote65ouwuliqwsdsj1i 2451439 2451438 2026-08-25T13:38:06Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns2005.wikimedia.org with OS trixie 2451439 wikitext text/x-wiki fbjte8fvrvngok532nh488sp0l4dfm6 2451441 2451439 2026-08-25T13:54:02Z Stashbot 7414 elukey: migrated the Docker Registry's /v2/repos/releng to the Releng's S3 bucket - T432829 2451441 wikitext text/x-wiki paws6kl9sy6ck05jw6sbbz3pn8n8o4h 2451442 2451441 2026-08-25T13:58:52Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns2005.wikimedia.org with reason: host reimage 2451442 wikitext text/x-wiki 2kpr2ou2u4c778qd3m9h0jm2vfny4t1 2451443 2451442 2026-08-25T14:02:37Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns2005.wikimedia.org with reason: host reimage 2451443 wikitext text/x-wiki slbh790qpi1s058dxyhxpcqkwojqt3m 2451445 2451443 2026-08-25T14:16:54Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . 2451445 wikitext text/x-wiki 9yqzzqmqgph0fyt2a0pf4n7akf45l4d 2451446 2451445 2026-08-25T14:20:13Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2903.codfw.wmnet with reason: Cloning 2451446 wikitext text/x-wiki 4rkxzfv9u1fbuxst7pd2ku6602yjcws 2451447 2451446 2026-08-25T14:22:57Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2451447 wikitext text/x-wiki gpdneu1f606dafbxak6sppeovt3qkdv 2451448 2451447 2026-08-25T14:23:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2451448 wikitext text/x-wiki 6qavujvhym7pkstxhoc130b6b417z76 2451449 2451448 2026-08-25T14:23:45Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling reboot on A:thanos-fe 2451449 wikitext text/x-wiki thdillxghtyvy7ms2kttwvty2bq3eo0 2451450 2451449 2026-08-25T14:25:15Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master 2451450 wikitext text/x-wiki age36vjj2tf3wllz1yuo5csrb6lhp28 2451451 2451450 2026-08-25T14:25:32Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors 2451451 wikitext text/x-wiki ecw2n6e0c38vqumrqt8e8wpqhht5hn6 2451452 2451451 2026-08-25T14:25:35Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors 2451452 wikitext text/x-wiki 95bbqcalty0jpqsu3j902ych05v0gnt 2451453 2451452 2026-08-25T14:27:05Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1329295|Disable CU on bswiktionary (T435954)]], [[gerrit:1329287|Close bswiktionary (T435954)]] 2451453 wikitext text/x-wiki 5vx0obiqcx2pgcae1nsq7p45y4dsbtw 2451454 2451453 2026-08-25T14:28:36Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . 2451454 wikitext text/x-wiki jlfiyd8eruxdkl1jyonjskmpsw3x701 2451455 2451454 2026-08-25T14:28:50Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . 2451455 wikitext text/x-wiki pv9tnaltn8ef6lakw5qaqdpt80adist 2451456 2451455 2026-08-25T14:29:20Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1329295|Disable CU on bswiktionary (T435954)]], [[gerrit:1329287|Close bswiktionary (T435954)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451456 wikitext text/x-wiki m6vs7ugmplyl3aevr1sx8nqux9gjv0b 2451457 2451456 2026-08-25T14:29:49Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with deployment 2451457 wikitext text/x-wiki 167862ee1zmqnk68iyq6xe05ndfh8fm 2451458 2451457 2026-08-25T14:29:54Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors 2451458 wikitext text/x-wiki n5r9jpa099n29ke2h47e90l10hjm0mv 2451459 2451458 2026-08-25T14:29:57Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors 2451459 wikitext text/x-wiki b7ntp42hq30i8r96xsli9489hac2zdw 2451460 2451459 2026-08-25T14:34:16Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1329295|Disable CU on bswiktionary (T435954)]], [[gerrit:1329287|Close bswiktionary (T435954)]] (duration: 07m 11s) 2451460 wikitext text/x-wiki 3s78eche4b7pz0smd6cpllf3g76n2bt 2451462 2451460 2026-08-25T14:36:29Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master 2451462 wikitext text/x-wiki bv21k15qislbfzqwc2pjedcae04kwbl 2451463 2451462 2026-08-25T14:41:26Z Stashbot 7414 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2451463 wikitext text/x-wiki kcaf9e7i7f05mswelq794whx1lt00lz 2451464 2451463 2026-08-25T14:42:13Z Stashbot 7414 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2451464 wikitext text/x-wiki tiya4iu2ct70mh1sg9rdmcyjniupg0g 2451466 2451464 2026-08-25T14:46:42Z Stashbot 7414 xcollazo@deploy1003: Started deploy [analytics/refinery@cd8542f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@cd8542f3] 2451466 wikitext text/x-wiki qv4rgnw53yqet31hk3yh9qiav529mps 2451467 2451466 2026-08-25T14:47:19Z Stashbot 7414 xcollazo@deploy1003: Finished deploy [analytics/refinery@cd8542f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@cd8542f3] (duration: 00m 37s) 2451467 wikitext text/x-wiki fsr1fq6ajxrdhh77e9dlxgwesg7qzj1 2451468 2451467 2026-08-25T14:47:39Z Stashbot 7414 xcollazo@deploy1003: Started deploy [analytics/refinery@cd8542f]: Regular analytics weekly train [analytics/refinery@cd8542f3] 2451468 wikitext text/x-wiki 2susobe3gxph735nigloh68m5zujdp3 2451469 2451468 2026-08-25T14:50:05Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns2005.wikimedia.org with OS trixie 2451469 wikitext text/x-wiki f9aqka6lujfqcy1xge02y21w45ph741 2451471 2451469 2026-08-25T14:50:36Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet 2451471 wikitext text/x-wiki 9nuvi9klqkt90iq8v2p3x4a4a4uk05m 2451472 2451471 2026-08-25T14:51:35Z Stashbot 7414 zabe@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.17 refs T430836 2451472 wikitext text/x-wiki 3rtrekengz6w2jwr368u1f3bt5vd7w1 2451473 2451472 2026-08-25T14:53:00Z Stashbot 7414 xcollazo@deploy1003: Finished deploy [analytics/refinery@cd8542f]: Regular analytics weekly train [analytics/refinery@cd8542f3] (duration: 05m 20s) 2451473 wikitext text/x-wiki b5tkye4fbeyukexqgdgkc4usbhmhk12 2451474 2451473 2026-08-25T14:53:15Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.284.0" for 3 host(s) 2451474 wikitext text/x-wiki 9l7z3yjfzd6nyri5vcc0cep001ixerv 2451475 2451474 2026-08-25T14:54:32Z Stashbot 7414 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Phabricator deploy 2451475 wikitext text/x-wiki 7m9dh82c0taw9xfwyizm53swm7an8r5 2451476 2451475 2026-08-25T14:54:50Z Stashbot 7414 xcollazo@deploy1003: Started deploy [analytics/refinery@cd8542f] (thin): Regular analytics weekly train THIN [analytics/refinery@cd8542f3] 2451476 wikitext text/x-wiki hlcy136hevzp4lndgxckuyj5f68721j 2451477 2451476 2026-08-25T14:54:55Z Stashbot 7414 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy 2451477 wikitext text/x-wiki lzu0lpuijb4il65d5eihy9day4hzwic 2451478 2451477 2026-08-25T14:55:13Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.284.0" completed for 3 hosts 2451478 wikitext text/x-wiki 48p9a4wd9wqhyxv03pou3ee5jhj3bu9 2451479 2451478 2026-08-25T14:55:29Z Stashbot 7414 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Phabricator deploy 2451479 wikitext text/x-wiki p7o79dz1x1wu4ell7feiafb3tmvj1k8 2451480 2451479 2026-08-25T14:57:06Z Stashbot 7414 xcollazo@deploy1003: Finished deploy [analytics/refinery@cd8542f] (thin): Regular analytics weekly train THIN [analytics/refinery@cd8542f3] (duration: 02m 16s) 2451480 wikitext text/x-wiki fn2gz3h8i7alc2s265eyp9619bs2qi5 2451482 2451480 2026-08-25T14:57:31Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet 2451482 wikitext text/x-wiki i1flh70hkq4yidlyleowdlu0m7ovolc 2451483 2451482 2026-08-25T14:58:23Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet 2451483 wikitext text/x-wiki acf8neuyvx404w11ngchckqphx69ioe 2451484 2451483 2026-08-25T15:00:09Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.remove-downtime for dns2005.wikimedia.org 2451484 wikitext text/x-wiki c9nn7367uy2c12vw2amehqc9le1xf6m 2451485 2451484 2026-08-25T15:00:12Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns2005.wikimedia.org 2451485 wikitext text/x-wiki swr2rq2zbj4j68pamq00p6ep6p62tnh 2451486 2451485 2026-08-25T15:01:29Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@7c592da]: deploy phab2003 for T435958 2451486 wikitext text/x-wiki m2qzg5x9wrme75evbgdmljo044dgevx 2451487 2451486 2026-08-25T15:02:20Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@7c592da]: deploy phab2003 for T435958 (duration: 00m 51s) 2451487 wikitext text/x-wiki 4wxwgkun1xwdjuht7du74eu31oya69j 2451488 2451487 2026-08-25T15:02:41Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@7c592da]: deploy phab1004 for T435958 2451488 wikitext text/x-wiki dffgyp3704wgsio27lp22rep1e6bofd 2451489 2451488 2026-08-25T15:02:46Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=yes; selector: name=dns2005.* [reason: trixie upgrade] 2451489 wikitext text/x-wiki 88l8cyfor7ff3lmo0m9jf59kdk9zaah 2451490 2451489 2026-08-25T15:03:14Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@7c592da]: deploy phab1004 for T435958 (duration: 00m 33s) 2451490 wikitext text/x-wiki 3tm58en4depjid11l2eioes32kivg4o 2451491 2451490 2026-08-25T15:03:29Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@7c592da]: deploy phab1005 for T435958 2451491 wikitext text/x-wiki os8mw2xht04yv7r24ng0f04olje0so0 2451492 2451491 2026-08-25T15:04:10Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@7c592da]: deploy phab1005 for T435958 (duration: 00m 41s) 2451492 wikitext text/x-wiki 1hasq2brdfd6nc8y7gve1stmwjrqzrj 2451493 2451492 2026-08-25T15:05:21Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet 2451493 wikitext text/x-wiki jddg6ett5jih3wpegrbcaaj0s5u9jnm 2451499 2451493 2026-08-25T15:16:55Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet 2451499 wikitext text/x-wiki jexeb2bee7jmsz2e7pnptgoklm8xd7k 2451500 2451499 2026-08-25T15:24:06Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet 2451500 wikitext text/x-wiki lfn995ygkazk42bnsm5qgg8tupn23vf 2451501 2451500 2026-08-25T15:26:08Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet 2451501 wikitext text/x-wiki nn1clfuhh1eixzx95d826dsk0aegfy0 2451502 2451501 2026-08-25T15:32:59Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet 2451502 wikitext text/x-wiki ntn5gtfnnufjs2k509n4mu43dtkzb44 2451504 2451502 2026-08-25T15:39:29Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2451504 wikitext text/x-wiki erh1ilze67z3z954axgfrcwie1oo7bl 2451505 2451504 2026-08-25T15:48:56Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2451505 wikitext text/x-wiki 8vkmdqymoovv7kvlgvox96wmf2i7oq8 2451510 2451505 2026-08-25T16:52:33Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.285.1" for 3 host(s) 2451510 wikitext text/x-wiki 8t631uwbbi8w7iboismxlzeqn6eqtjb 2451511 2451510 2026-08-25T16:54:27Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.285.1" completed for 3 hosts 2451511 wikitext text/x-wiki 4icj6zqiepkx3ae5l7wisw0hn4g7ste 2451512 2451511 2026-08-25T17:01:24Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=no; selector: name=dns2004.* [reason: trixie upgrade] 2451512 wikitext text/x-wiki 6haowfosx80x2ydk8vbyyk3abvkhvxv 2451513 2451512 2026-08-25T17:01:48Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns2004.wikimedia.org with OS trixie 2451513 wikitext text/x-wiki 2fxu17r8mjqj2mfjc783ulazg7b1ymu 2451514 2451513 2026-08-25T17:05:07Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. 2451514 wikitext text/x-wiki 830dri2sco3yrpbvct2shjbvhb7hp9j 2451515 2451514 2026-08-25T17:07:28Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. 2451515 wikitext text/x-wiki eg0fbuoq7moovekqbw557vjqgo9e3rh 2451516 2451515 2026-08-25T17:07:35Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-canary cluster: Roll restart of all Presto's jvm daemons. 2451516 wikitext text/x-wiki 6g9tqafj713gl9zgg0osern2gs1sjra 2451517 2451516 2026-08-25T17:09:42Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: testing https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 2451517 wikitext text/x-wiki b5ivhbf55xo6me49es92ut1p87jp0en 2451518 2451517 2026-08-25T17:10:08Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-canary cluster: Roll restart of all Presto's jvm daemons. 2451518 wikitext text/x-wiki sezetbrfwntho4de7rivdw8kkeaob94 2451519 2451518 2026-08-25T17:11:33Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. 2451519 wikitext text/x-wiki sr1zoexsf1l4j47ybarosn44gnrojgf 2451520 2451519 2026-08-25T17:13:43Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: testing https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (duration: 04m 01s) 2451520 wikitext text/x-wiki ldg0m4agcshkwzgwgus5mfoe4hflboj 2451521 2451520 2026-08-25T17:14:22Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: testing https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 2451521 wikitext text/x-wiki qr8oiwfp9cn2qldrsauzpj8h2a2dbp7 2451522 2451521 2026-08-25T17:20:03Z Stashbot 7414 dancy@deploy1003: Rolling back deployment 2451522 wikitext text/x-wiki 60lxx21hxxg6pwbft4du90511d2rfy8 2451523 2451522 2026-08-25T17:22:24Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: testing https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (duration: 08m 02s) 2451523 wikitext text/x-wiki 6yhv5qmtlnjns8to3ackhxjq2zes8wl 2451524 2451523 2026-08-25T17:22:52Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns2004.wikimedia.org with reason: host reimage 2451524 wikitext text/x-wiki ny9nbbohw6llwmpa4euz0e7to5kws4e 2451526 2451524 2026-08-25T17:28:28Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns2004.wikimedia.org with reason: host reimage 2451526 wikitext text/x-wiki 3c0yidqplq8mabwzcoaw9cic2n1qchx 2451527 2451526 2026-08-25T17:41:23Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. 2451527 wikitext text/x-wiki kgc1n3lhti1gbzvzforqelwprv0b9fg 2451530 2451527 2026-08-25T18:02:35Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet 2451530 wikitext text/x-wiki btk6yat9etgxrzvchxynhranqdbpzia 2451531 2451530 2026-08-25T18:04:31Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet 2451531 wikitext text/x-wiki b2466d3u1hn38g8p5xe4disnr8w0dy7 2451533 2451531 2026-08-25T18:11:58Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2451533 wikitext text/x-wiki fmuotlul4se3k77fjtejpxmspiunea8 2451534 2451533 2026-08-25T18:12:23Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2451534 wikitext text/x-wiki n77xhq3nejbraj2b24tc6b33q6eb8e7 2451536 2451534 2026-08-25T18:19:11Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2451536 wikitext text/x-wiki it3wvv0vzpc98y5u6uex3pvt6i60vxk 2451537 2451536 2026-08-25T18:19:22Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2451537 wikitext text/x-wiki fkrdoj2cnudgop2hhn98rf6vusb6wok 2451538 2451537 2026-08-25T18:23:37Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns2004.wikimedia.org with OS trixie 2451538 wikitext text/x-wiki 03vcn2kx5ppl18btbxgkaaqrpd141pg 2451539 2451538 2026-08-25T18:36:47Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1022: Pooling 2451539 wikitext text/x-wiki hb3afttjohvm457er6a2mb2wsxd3c59 2451540 2451539 2026-08-25T18:36:50Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2451540 wikitext text/x-wiki 143aez54t6eel3xrunijiztuppad4q9 2451541 2451540 2026-08-25T18:36:54Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2451541 wikitext text/x-wiki bfj48uv7c570v6oe2z2h2jpkyr7qfzq 2451542 2451541 2026-08-25T18:36:57Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1022: Pooling 2451542 wikitext text/x-wiki bloz8j4lu0jknib3zw5klet6y73fht2 2451543 2451542 2026-08-25T18:48:57Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc1022: depooling 2451543 wikitext text/x-wiki e2b1ymibn6sqh12m6ddphijogns3ju2 2451544 2451543 2026-08-25T18:49:01Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2451544 wikitext text/x-wiki ebynd0mjxh6lagc2q0t4fcaztciw99e 2451545 2451544 2026-08-25T18:49:05Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2451545 wikitext text/x-wiki dtcl0i1hvai08i9h7yi04mp2pzwgdin 2451546 2451545 2026-08-25T18:49:08Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1022: depooling 2451546 wikitext text/x-wiki bhqwznjs7zsifw59r3qrfk2umm4682m 2451547 2451546 2026-08-25T18:56:41Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=yes; selector: name=dns2004.* [reason: trixie upgrade] 2451547 wikitext text/x-wiki bx75oedj417t33uuk891993z7molj5k 2451548 2451547 2026-08-25T18:57:18Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.remove-downtime for dns2004.wikimedia.org 2451548 wikitext text/x-wiki ffy2zy0yynp3pdbtnuygde0hk3o8qvp 2451549 2451548 2026-08-25T18:57:21Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns2004.wikimedia.org 2451549 wikitext text/x-wiki n8ty04gk81hzm969hy9dwkbl0wdfvgz 2451552 2451549 2026-08-25T19:14:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. 2451552 wikitext text/x-wiki rfqzb2prln9vdgjlbwsuo1cas7io0bl 2451555 2451552 2026-08-25T19:44:36Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. 2451555 wikitext text/x-wiki 3sjy70r8oizeipixt2cwmt44wlgtx45 2451557 2451555 2026-08-25T19:46:59Z Stashbot 7414 inflatador: bking@apt1002 reprepro --component thirdparty/opensearch3 update trixie-wikimedia T420230 2451557 wikitext text/x-wiki sg60lorwevgzcoqxapomp6sz9ht720z 2451558 2451557 2026-08-25T20:06:55Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1259251|Remove reading list experiment instrumentation (T421939)]] 2451558 wikitext text/x-wiki bpjh7i8o3fp33w24ro6lvs1fwejuqer 2451559 2451558 2026-08-25T20:09:32Z Stashbot 7414 cjming@deploy1003: lmora, cjming: Backport for [[gerrit:1259251|Remove reading list experiment instrumentation (T421939)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451559 wikitext text/x-wiki pnz1o0andjfsckoh3gyq31gyga7l2ux 2451560 2451559 2026-08-25T20:18:38Z Stashbot 7414 cjming@deploy1003: lmora, cjming: Continuing with deployment 2451560 wikitext text/x-wiki 6i1c7e01derhh4x9o490bv304by2hq1 2451562 2451560 2026-08-25T20:22:59Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1259251|Remove reading list experiment instrumentation (T421939)]] (duration: 16m 03s) 2451562 wikitext text/x-wiki 70z3ixrosirgl0931tzffzgf9chrf3m 2451563 2451562 2026-08-25T20:24:38Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1328239|Enable readinglist for phase0 wikis (T435258)]] 2451563 wikitext text/x-wiki 07f5wx0noatj1sebhb03u1vsxw31c31 2451565 2451563 2026-08-25T20:29:38Z Stashbot 7414 cjming@deploy1003: cjming, bwang: Backport for [[gerrit:1328239|Enable readinglist for phase0 wikis (T435258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451565 wikitext text/x-wiki 719i658hmaz02k0baclxe1cneld325i 2451568 2451565 2026-08-25T20:34:13Z Stashbot 7414 cjming@deploy1003: cjming, bwang: Continuing with deployment 2451568 wikitext text/x-wiki ih4brxfabsexlnczyknxd85gm7rrudc 2451569 2451568 2026-08-25T20:38:42Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328239|Enable readinglist for phase0 wikis (T435258)]] (duration: 14m 03s) 2451569 wikitext text/x-wiki ck24yr7z8ociggogsg7y7mdkfnpyn6e 2451570 2451569 2026-08-25T20:41:45Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1329339|zhwiki: allow TAV remove themselves and grant abusefilter-revert to sysop and abusefilter (T435887)]] 2451570 wikitext text/x-wiki j8zz56lnkzexdnr5zhadhmfm7022r2h 2451571 2451570 2026-08-25T20:44:20Z Stashbot 7414 cjming@deploy1003: cjming, hamishz: Backport for [[gerrit:1329339|zhwiki: allow TAV remove themselves and grant abusefilter-revert to sysop and abusefilter (T435887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451571 wikitext text/x-wiki cptcc2exa0ne9jujcgvnxdn2jydmm9j 2451572 2451571 2026-08-25T20:45:23Z Stashbot 7414 cjming@deploy1003: cjming, hamishz: Continuing with deployment 2451572 wikitext text/x-wiki 6rwbwtc9pcph6gwilv44k400mg1cd4v 2451573 2451572 2026-08-25T20:49:41Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1329339|zhwiki: allow TAV remove themselves and grant abusefilter-revert to sysop and abusefilter (T435887)]] (duration: 07m 55s) 2451573 wikitext text/x-wiki j9jqa3fvjmwh2c6fwshqv1j4hp7ekvg 2451574 2451573 2026-08-25T20:53:22Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1328528|prv: Enable parsoid rendering for 9 wikisources (T435830)]], [[gerrit:1329369|Set a default for wgUseParsoidMessages in prod (T432760)]] 2451574 wikitext text/x-wiki f550nyj8i17y8pcxqjjc2ygf7ctdji2 2451575 2451574 2026-08-25T20:55:55Z Stashbot 7414 arlolra@deploy1003: jgiannelos, arlolra: Backport for [[gerrit:1328528|prv: Enable parsoid rendering for 9 wikisources (T435830)]], [[gerrit:1329369|Set a default for wgUseParsoidMessages in prod (T432760)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451575 wikitext text/x-wiki m79dy3wsxj0pcl2exm64t5x5lh9dzml 2451576 2451575 2026-08-25T20:56:31Z Stashbot 7414 arlolra@deploy1003: jgiannelos, arlolra: Continuing with deployment 2451576 wikitext text/x-wiki p3rmg8w6wndea56e5ihoxw1718pmhfy 2451577 2451576 2026-08-25T21:00:55Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328528|prv: Enable parsoid rendering for 9 wikisources (T435830)]], [[gerrit:1329369|Set a default for wgUseParsoidMessages in prod (T432760)]] (duration: 07m 33s) 2451577 wikitext text/x-wiki 9ltgzg61lvpnrxc1unqfxbhzm8mk30i 2451578 2451577 2026-08-25T21:03:20Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1329370|LR: Only unset deriv state for inner rendering (T434456)]] 2451578 wikitext text/x-wiki 6pboai1evjoxql4i2m7u692on45jxl2 2451583 2451578 2026-08-25T21:05:58Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1329370|LR: Only unset deriv state for inner rendering (T434456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451583 wikitext text/x-wiki etp6zdjkqglfazauqi5t6o928fxypfh 2451584 2451583 2026-08-25T21:08:44Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2451584 wikitext text/x-wiki 0kuipaygv8p9qrvk1pbptnzejorn9xj 2451585 2451584 2026-08-25T21:13:06Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1329370|LR: Only unset deriv state for inner rendering (T434456)]] (duration: 09m 45s) 2451585 wikitext text/x-wiki 01fpovsa10s1jevoyfzlb8goqr7zl7m 2451591 2451585 2026-08-25T21:40:28Z Stashbot 7414 sbassett: Removed security mitigation for T435455 2451591 wikitext text/x-wiki oka1ydq7ofj68it4gnuoyz6j4z0p8pc 2451596 2451591 2026-08-26T02:00:38Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2451596 wikitext text/x-wiki h7jgid2zzrqgtj8u91rwb771aw0bm69 2451597 2451596 2026-08-26T02:08:14Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) 2451597 wikitext text/x-wiki dbk8fgevcnw63u1fg0rjf8fevnxcb3y 2451600 2451597 2026-08-26T03:19:09Z Stashbot 7414 denisse: Upgrading Grafana on codfw - T435816 2451600 wikitext text/x-wiki kog59quhbb5zvufrdq7lo11aqa04ir1 2451601 2451600 2026-08-26T07:04:31Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: eqsin switch migration, T435406] 2451601 wikitext text/x-wiki opfndl7tttjq7lc6rvh68htjju8bshe 2451602 2451601 2026-08-26T07:04:41Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: eqsin switch migration, T435406] 2451602 wikitext text/x-wiki 90m4quvcmws7npxqljt675y0btsim1s 2451603 2451602 2026-08-26T07:04:52Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet 2451603 wikitext text/x-wiki hytu3b2t1q5jcb4q2obkuufdy5xpboe 2451604 2451603 2026-08-26T07:06:58Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2451604 wikitext text/x-wiki i8cha400c9zigu4p3es07tn40zlz4et 2451605 2451604 2026-08-26T07:07:11Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1329386|[MinMin] Do not display language count badge for 0 languages (T435249)]] 2451605 wikitext text/x-wiki kk4fwte1bvsttxz0u8b2u7yheuyi373 2451606 2451605 2026-08-26T07:09:50Z Stashbot 7414 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1329386|[MinMin] Do not display language count badge for 0 languages (T435249)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451606 wikitext text/x-wiki pl4hwyri8s5myxtby06wfhxkwi0poje 2451608 2451606 2026-08-26T07:11:30Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet 2451608 wikitext text/x-wiki 892qv935yy4xwhl5llv4v20kch3ba06 2451609 2451608 2026-08-26T07:11:35Z Stashbot 7414 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment 2451609 wikitext text/x-wiki 072p9rw3o67ingxtuy9nheswgrt2kdr 2451612 2451609 2026-08-26T07:16:00Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1329386|[MinMin] Do not display language count badge for 0 languages (T435249)]] (duration: 08m 49s) 2451612 wikitext text/x-wiki ttmz6w20ctbe0lej20pcslbdult5kcj 2451613 2451612 2026-08-26T07:17:40Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1329358|Add reader experiments exclusion to CNB (T435658)]] 2451613 wikitext text/x-wiki 0t60owcktf81rs4ckwt8t7j0v1b4ag5 2451616 2451613 2026-08-26T07:19:10Z Stashbot 7414 moritzm: installing Linux 6.12.105 on trixie hosts 2451616 wikitext text/x-wiki d0btgri2yna83l325t6861dlw0dipvv 2451617 2451616 2026-08-26T07:19:23Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet 2451617 wikitext text/x-wiki 1i76kl7kg7a6sxomiuzm1ykcl383lae 2451618 2451617 2026-08-26T07:20:17Z Stashbot 7414 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1329358|Add reader experiments exclusion to CNB (T435658)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451618 wikitext text/x-wiki 6mepms9418vr1oortm0fxc291a790vi 2451619 2451618 2026-08-26T07:22:35Z Stashbot 7414 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment 2451619 wikitext text/x-wiki iwi42okwxbnb7pmeomgwhnnrvsmvm45 2451621 2451619 2026-08-26T07:26:04Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet 2451621 wikitext text/x-wiki ttr1nhu6v86brh0q3wcxg4ibf6znrs4 2451622 2451621 2026-08-26T07:26:56Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1329358|Add reader experiments exclusion to CNB (T435658)]] (duration: 09m 16s) 2451622 wikitext text/x-wiki 00xa83mep10bgvblffm5zur0o3dlg1f 2451624 2451622 2026-08-26T07:48:13Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply 2451624 wikitext text/x-wiki 3awg8borjtnsc3q73hzn5iky8415ahn 2451625 2451624 2026-08-26T07:48:16Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply 2451625 wikitext text/x-wiki bc5samvy1738yzbwidkl1qb8nyl9ikn 2451626 2451625 2026-08-26T07:54:18Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet 2451626 wikitext text/x-wiki fzdjhhlm2cywxebnoc1jkkl0d4qfmhu 2451627 2451626 2026-08-26T08:00:45Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet 2451627 wikitext text/x-wiki si4xm2sqw2aut931y9we4g22qz80nfm 2451630 2451627 2026-08-26T08:12:16Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet 2451630 wikitext text/x-wiki g6ye7kelsi3ov93jfgnnb3eeinapd8o 2451631 2451630 2026-08-26T08:13:00Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.17 refs T430836 2451631 wikitext text/x-wiki m8f0h3bbm4ioaxnjpfe9k681e05ki7c 2451632 2451631 2026-08-26T08:14:10Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 12 hosts with reason: site migration to new Nokia switches 2451632 wikitext text/x-wiki 6essl2fbolwbq6wswm3df7oioicbo5f 2451633 2451632 2026-08-26T08:14:48Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 13 hosts with reason: site migration to new Nokia switches 2451633 wikitext text/x-wiki 3xncbji6zdd3fc7migvhcupm3847lew 2451634 2451633 2026-08-26T08:16:37Z Stashbot 7414 cmooney@cumin1003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 5:00:00 on 12 hosts with reason: site migration to new Nokia switches 2451634 wikitext text/x-wiki bdt2qrm56hai7zp01bfbzm83inr5o0c 2451635 2451634 2026-08-26T08:17:40Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 12 hosts with reason: site migration to new Nokia switches 2451635 wikitext text/x-wiki 21dpg6resupt1knye2hvwvu09fzyg5v 2451636 2451635 2026-08-26T08:19:48Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet 2451636 wikitext text/x-wiki jh382ozu4itbp4mjpox908kxcz6wyw0 2451637 2451636 2026-08-26T08:21:04Z Stashbot 7414 hashar: Rolling back group 1 from 1.47.0-wmf.17 to 1.47.0-wmf.16 due to Special:EntityUsage being broken on Wikibase client wikis # T436060 T430836 2451637 wikitext text/x-wiki gnztjptc9vkkw48qkgyrjhqeywpupf9 2451638 2451637 2026-08-26T08:28:32Z Stashbot 7414 fceratto@cumin1003: Removing db-test1003 from zarcillo T435912 2451638 wikitext text/x-wiki 7t4shqozmirplejthhoyojyos6q3r97 2451639 2451638 2026-08-26T08:28:36Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) 2451639 wikitext text/x-wiki ns6roqmvdfvo9jwmzsqr0louyz87tii 2451640 2451639 2026-08-26T08:30:09Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2451640 wikitext text/x-wiki rgzminkfi4s4wd6837i7ikj2rm8c5tb 2451641 2451640 2026-08-26T08:30:15Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.17 refs T430836 2451641 wikitext text/x-wiki fx9aj6oni5zxbvnwedxrjout0r1g7bc 2451642 2451641 2026-08-26T08:30:49Z Stashbot 7414 slyngs: Updating IDP/CAS-SSO to version 3.7.8.2 2451642 wikitext text/x-wiki edscrakiq3h0j3x0y51ihnuliza32d0 2451643 2451642 2026-08-26T08:32:29Z Stashbot 7414 slyngshede@dns1004: FAIL - running authdns-update 2451643 wikitext text/x-wiki 7rqj2nju0t05p9rk47huau6nwkm99v5 2451644 2451643 2026-08-26T08:42:10Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.metamonitoring.downtime Downtime for 4:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild 2451644 wikitext text/x-wiki dwffov9eeqyy8oqmn72midu7h8t3xgr 2451645 2451644 2026-08-26T08:42:15Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.metamonitoring.downtime (exit_code=0) Downtime for 4:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild 2451645 wikitext text/x-wiki a5crq1qjgueh22opbrik6oqf3vpcbri 2451646 2451645 2026-08-26T08:49:27Z Stashbot 7414 moritzm: installing openssl security updates on trixie 2451646 wikitext text/x-wiki s0k24ausggel7y1ha4enjke66trwz78 2451649 2451646 2026-08-26T09:01:52Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bookworm 2451649 wikitext text/x-wiki dsv4xzd2jnhzmxpq4g0kwj0vulrhga0 2451654 2451649 2026-08-26T09:07:09Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . 2451654 wikitext text/x-wiki nfi1xr0cu7sz0zxyee5tce82765rxui 2451655 2451654 2026-08-26T09:07:33Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . 2451655 wikitext text/x-wiki ttdju5mvhr5p19ubmowtr9v8a1nhk5f 2451656 2451655 2026-08-26T09:09:40Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . 2451656 wikitext text/x-wiki 6g2nx8uu7yqvhktxk81gnydybo3725t 2451660 2451656 2026-08-26T09:22:44Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . 2451660 wikitext text/x-wiki 63yee4wbcrkdgoqdneg0snv4m604psh 2451661 2451660 2026-08-26T09:24:25Z Stashbot 7414 hashar@deploy1003: Started scap sync-world: Backport for [[gerrit:1329511|Revert "Removes unused join and refactors helper method." (T436060)]], [[gerrit:1329512|Revert "This splits the joined `wbc_entity_usage` queries into individual queries because of X1 migration." (T436060)]] 2451661 wikitext text/x-wiki mjbzul1sxhcleougp5wlbnntpodcqw5 2451662 2451661 2026-08-26T09:26:46Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-presto1010.eqiad.wmnet with OS bookworm 2451662 wikitext text/x-wiki mhxu2xcuj7c7zlct0tzdu95a0hf0w4i 2451663 2451662 2026-08-26T09:27:01Z Stashbot 7414 hashar@deploy1003: awight, hashar: Backport for [[gerrit:1329511|Revert "Removes unused join and refactors helper method." (T436060)]], [[gerrit:1329512|Revert "This splits the joined `wbc_entity_usage` queries into individual queries because of X1 migration." (T436060)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451663 wikitext text/x-wiki pdvnxj6un5sc4d83juyv7qtmeiumwbx 2451664 2451663 2026-08-26T09:27:09Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bookworm 2451664 wikitext text/x-wiki bg4s437g01sum5927u0q15fhuznwjpo 2451666 2451664 2026-08-26T09:28:19Z Stashbot 7414 hashar@deploy1003: awight, hashar: Continuing with deployment 2451666 wikitext text/x-wiki 1rfan8thg0ccf3q14k9mg14q4cl12k6 2451667 2451666 2026-08-26T09:30:41Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2451667 wikitext text/x-wiki s858fyky3kyyzr5navw8453g5j0mgz1 2451668 2451667 2026-08-26T09:30:46Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) 2451668 wikitext text/x-wiki ph4jjouyfx8m3j1v5rwo4b6vv3rucw4 2451669 2451668 2026-08-26T09:31:21Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2451669 wikitext text/x-wiki o7gv0bel2ufvu061rsaqswj9064qblp 2451670 2451669 2026-08-26T09:31:27Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db-test2002.codfw.wmnet 2451670 wikitext text/x-wiki soagnxmu2lvqjnekqqmtnu7vwmxbak8 2451671 2451670 2026-08-26T09:32:39Z Stashbot 7414 hashar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1329511|Revert "Removes unused join and refactors helper method." (T436060)]], [[gerrit:1329512|Revert "This splits the joined `wbc_entity_usage` queries into individual queries because of X1 migration." (T436060)]] (duration: 08m 13s) 2451671 wikitext text/x-wiki fyiz6z0ueyl82v7jbnb9dtakxpk7y7x 2451677 2451671 2026-08-26T09:39:33Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts db-test2002.codfw.wmnet 2451677 wikitext text/x-wiki 2kacptrc4yjedhp32o21n33s0h16980 2451678 2451677 2026-08-26T09:39:36Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) 2451678 wikitext text/x-wiki b1s7ksrxmiaid1akmnvwfcfs5uh8fd3 2451679 2451678 2026-08-26T09:39:39Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2451679 wikitext text/x-wiki gok67vsi30mgt76zm1owv223rv6avtx 2451680 2451679 2026-08-26T09:39:44Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db-test2002.codfw.wmnet 2451680 wikitext text/x-wiki s9gmd4uyk78con2536pl885cqoh5geb 2451682 2451680 2026-08-26T09:41:48Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts db-test2002.codfw.wmnet 2451682 wikitext text/x-wiki pnk6gwr6477qe8ifeetrf3umovkfwq0 2451683 2451682 2026-08-26T09:41:51Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) 2451683 wikitext text/x-wiki ed5z8u75pj9pyd7wyo2yovisj2nvr3n 2451684 2451683 2026-08-26T09:42:18Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2022.codfw.wmnet,pc1022.eqiad.wmnet with reason: Working on pc2 truncation 2451684 wikitext text/x-wiki jjfbvwojf0nkiia6sbkvz0ndj1vcw2w 2451685 2451684 2026-08-26T09:47:32Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.17 refs T430836 2451685 wikitext text/x-wiki drywb8o4eh20mrb2j30v9zz0iip4g0n 2451688 2451685 2026-08-26T09:55:07Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1022: Pool back 2451688 wikitext text/x-wiki rn1dh7c7w0ijj036iuscuvef8v6ivsk 2451689 2451688 2026-08-26T09:55:09Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool pc1022: Pool back 2451689 wikitext text/x-wiki nwlpgm8hkla6t7q7hprtvom9hfhbpfv 2451690 2451689 2026-08-26T09:55:13Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1022: Pool back pc2 2451690 wikitext text/x-wiki t5bmhgswiyo9d5hw2v0xzvy5xbfxvim 2451691 2451690 2026-08-26T09:55:16Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool pc1022: Pool back pc2 2451691 wikitext text/x-wiki 8oljk2az1btow01sfq054ozwhguterx 2451692 2451691 2026-08-26T09:55:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1022: Pool back pc2 2451692 wikitext text/x-wiki geptm1zcud4h7riugx1vbh4r534qzm5 2451693 2451692 2026-08-26T09:55:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2451693 wikitext text/x-wiki 335o3s99ehyg8nrkq0zu5qmdpvqr5mv 2451694 2451693 2026-08-26T09:55:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2451694 wikitext text/x-wiki ao8toa3yl4ie1bwe16babhf50jl0g0s 2451695 2451694 2026-08-26T09:55:39Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1022: Pool back pc2 2451695 wikitext text/x-wiki att11xeq92xptc8q9p1tbn4ppt6ebh3 2451697 2451695 2026-08-26T10:02:23Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.decommission 2451697 wikitext text/x-wiki owr6arokueulfhc03z98665ff4jgk6d 2451698 2451697 2026-08-26T10:02:27Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db-test2002.codfw.wmnet 2451698 wikitext text/x-wiki czt6livzags1amddi3xkuruci3i9j4u 2451699 2451698 2026-08-26T10:03:31Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2451699 wikitext text/x-wiki 6dp6hmuq5x4jfch5nm7lref274e39bj 2451701 2451699 2026-08-26T10:05:44Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2451701 wikitext text/x-wiki 98o9a4mpz9nlf4fmzg5xojo93onmgow 2451702 2451701 2026-08-26T10:07:39Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2451702 wikitext text/x-wiki t8a1kk7eslm5lyg3sbw1zl88n7a0s80 2451703 2451702 2026-08-26T10:15:54Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1010.eqiad.wmnet with OS bookworm 2451703 wikitext text/x-wiki lcf54mwlinf68xq0npqtjz0emgzucxi 2451704 2451703 2026-08-26T10:17:37Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply 2451704 wikitext text/x-wiki 8vzipgxmj3jlncuwt692xt5dkhhkfpp 2451705 2451704 2026-08-26T10:18:27Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply 2451705 wikitext text/x-wiki 7oa8357fr27np5gxdyevxen1ezt6f77 2451706 2451705 2026-08-26T10:18:34Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply 2451706 wikitext text/x-wiki o0gyhgab8cbdkjuqxx9pai8pne08qo3 2451707 2451706 2026-08-26T10:19:05Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply 2451707 wikitext text/x-wiki 7ghq1icnm4mfeqgkt1b7apxb2ibn3tj 2451708 2451707 2026-08-26T10:19:22Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-misc: apply 2451708 wikitext text/x-wiki 4r7en5gu3vrifyqxg1ian6di0wb5h11 2451710 2451708 2026-08-26T10:19:44Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-misc: apply 2451710 wikitext text/x-wiki kvtd36ned5k26iiogevx8rmiopuho9i 2451711 2451710 2026-08-26T10:19:50Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-misc: apply 2451711 wikitext text/x-wiki ny3oditsr8ajavpesb77smdkhfm7mij 2451712 2451711 2026-08-26T10:20:10Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-misc: apply 2451712 wikitext text/x-wiki 986m6txkgpx9ca57dfuihm5ff8zkdxo 2451713 2451712 2026-08-26T10:20:58Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply 2451713 wikitext text/x-wiki imqvw15hhspdzv8sioj2rlr5ws2jos2 2451714 2451713 2026-08-26T10:21:59Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply 2451714 wikitext text/x-wiki p4hnk639a4h5ahjdzh73yscwxr2a94l 2451715 2451714 2026-08-26T10:22:04Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply 2451715 wikitext text/x-wiki pd14nkjkp9979tedgkcri9w8du90lg0 2451716 2451715 2026-08-26T10:23:05Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply 2451716 wikitext text/x-wiki d5sohp9n1ud3jifo0rgd6oeblq19u9s 2451717 2451716 2026-08-26T10:24:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2451717 wikitext text/x-wiki 44aglwlkrkyoqkt51ibig0rxuy1gpaa 2451718 2451717 2026-08-26T10:25:01Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply 2451718 wikitext text/x-wiki 9xlh9d0zvpvef1ol3lku6ctui4mr1fw 2451719 2451718 2026-08-26T10:26:22Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply 2451719 wikitext text/x-wiki 6l571gj5bovxhnndmvgdsuhw8why7df 2451720 2451719 2026-08-26T10:27:24Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2451720 wikitext text/x-wiki e2x00ub384caqm80oek52jfkp7ja1i3 2451721 2451720 2026-08-26T10:28:59Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply 2451721 wikitext text/x-wiki iwmkht31tur822iep187y8gymftdpdn 2451722 2451721 2026-08-26T10:30:05Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply 2451722 wikitext text/x-wiki lrfobr1w5xlyvwpfka4gxqdynlrjmby 2451723 2451722 2026-08-26T10:32:00Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-int: apply 2451723 wikitext text/x-wiki gfkgdi86e24mrkzuv4itfygrogkrdez 2451724 2451723 2026-08-26T10:32:59Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply 2451724 wikitext text/x-wiki 44zmwrpfi5kpgwiaw637mwpc9zskdn0 2451725 2451724 2026-08-26T10:35:24Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply 2451725 wikitext text/x-wiki kwktgksviy2argnqqs62jpf360wyjlo 2451726 2451725 2026-08-26T10:36:25Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply 2451726 wikitext text/x-wiki aazkewtxbb7hwk4xey3fj237r43whm0 2451727 2451726 2026-08-26T10:37:09Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . 2451727 wikitext text/x-wiki a3ztfcoxx75otmytx6xlygt4winr7wc 2451728 2451727 2026-08-26T10:37:13Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply 2451728 wikitext text/x-wiki kfx3r015ehyfsp8620pwhc9k2h72qtz 2451729 2451728 2026-08-26T10:37:19Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . 2451729 wikitext text/x-wiki rbezucg24097exzdt4rm6xacds5k6hk 2451730 2451729 2026-08-26T10:38:27Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply 2451730 wikitext text/x-wiki 1s4gn3irgkgllbdbco9o5l8iabhb1uu 2451731 2451730 2026-08-26T10:38:48Z Stashbot 7414 slyngshede@puppetserver1001: conftool action : set/pooled=no; selector: name=dns5003.* 2451731 wikitext text/x-wiki ad0adpa6bsiq75ouo8nnv1kl05dz6e1 2451732 2451731 2026-08-26T10:38:59Z Stashbot 7414 slyngshede@puppetserver1001: conftool action : set/pooled=no; selector: name=dns5004.* 2451732 wikitext text/x-wiki rb27mpf8mtkwdybq2y8po8ddqw30htn 2451733 2451732 2026-08-26T10:39:27Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2451733 wikitext text/x-wiki 2w4ob5cq5ubt49ut3y3m2sgovkrd01u 2451734 2451733 2026-08-26T10:41:39Z Stashbot 7414 hnowlan: uploaded cortobot-1.2.0 to apt1002 2451734 wikitext text/x-wiki mg7a6vigxpn27lp2fahonzoeigd47im 2451735 2451734 2026-08-26T10:41:44Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2451735 wikitext text/x-wiki 1sjaifkns8nas395m6jgpvn52ddn7i8 2451737 2451735 2026-08-26T10:42:23Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply 2451737 wikitext text/x-wiki j0uqjkufvn2tx4sqobpa2rzkkbe8zug 2451738 2451737 2026-08-26T10:43:03Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2451738 wikitext text/x-wiki asz19f9jhe5grat7pnxwf3ss4j1kj5t 2451739 2451738 2026-08-26T10:43:35Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply 2451739 wikitext text/x-wiki ebhymn8t3h0h09lqkl4zlsyrefj6f2f 2451740 2451739 2026-08-26T10:45:14Z Stashbot 7414 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2451740 wikitext text/x-wiki 3hgmnlke9dy336rongv3ej5tpyj5aed 2451741 2451740 2026-08-26T11:04:21Z Stashbot 7414 denisse: Upgrading grafana in eqiad - T435816 2451741 wikitext text/x-wiki he1do78p4houzerb04mdxh8zn15y86l 2451742 2451741 2026-08-26T11:05:28Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply 2451742 wikitext text/x-wiki nt8ojulepgz7yhctfsvpfj3h5swnci7 2451743 2451742 2026-08-26T11:05:48Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply 2451743 wikitext text/x-wiki exhe2bb5j4g4we1jqsx4920tgs5zs76 2451744 2451743 2026-08-26T11:08:48Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply 2451744 wikitext text/x-wiki eqa6z6c5z45gdz1z9hop4x52h9w6uen 2451745 2451744 2026-08-26T11:09:15Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply 2451745 wikitext text/x-wiki c4deoch09cck7r4pq29wznp26pg30hz 2451746 2451745 2026-08-26T11:10:34Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply 2451746 wikitext text/x-wiki tqayacfjrixivgf2x2efdac4gfdyuhm 2451747 2451746 2026-08-26T11:11:02Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply 2451747 wikitext text/x-wiki 2x6ch26u0nwvk5bmksunuaykyarq2fv 2451748 2451747 2026-08-26T11:26:22Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host ganeti3005.esams.wmnet with OS bookworm 2451748 wikitext text/x-wiki rxk1igt4lzoujxf4v4rxvu1kqh3ojzc 2451749 2451748 2026-08-26T11:49:06Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2451749 wikitext text/x-wiki 7ihv4a99wvk206w8j5gx7xxi7sdgfy2 2451750 2451749 2026-08-26T11:50:48Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti3005.esams.wmnet with reason: host reimage 2451750 wikitext text/x-wiki msvwulsis3xlyhygvxv6jezpgqi43n7 2451751 2451750 2026-08-26T11:54:29Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti3005.esams.wmnet with reason: host reimage 2451751 wikitext text/x-wiki ezr1qiifpj1i38rrbohi5l8pdmmmvp9 2451752 2451751 2026-08-26T11:55:27Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1329531|UserGroupsSpecialPage: Check in_array in strict mode (T435907)]], [[gerrit:1329530|UserGroupsSpecialPage: Check in_array in strict mode (T435907)]] 2451752 wikitext text/x-wiki 6ejxf9nxx2u70jdwtiw055lbll0dec9 2451754 2451752 2026-08-26T11:58:03Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1329531|UserGroupsSpecialPage: Check in_array in strict mode (T435907)]], [[gerrit:1329530|UserGroupsSpecialPage: Check in_array in strict mode (T435907)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2451754 wikitext text/x-wiki hzr8q7kq7z240llv6a0osdu7hays6e1 2451755 2451754 2026-08-26T11:58:34Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2451755 wikitext text/x-wiki 9z8pz3pfxht0lft62ebqmy37vfx56m7 498 9925 2451567 2451247 2026-08-25T20:31:30Z Stashbot 7414 Southparkfan: T401839 - provisioned deployment-docker-wikifeeds01 (trixie) using Tofu (cloudvps-repos/deployment-prep/tofu-provisioning) 2451567 wikitext text/x-wiki qkmhj5fmo1jjfgkpi1x7rap5ai2ogeb 2451580 2451567 2026-08-25T21:03:28Z Stashbot 7414 brett: Switch acme-chief active to deployment-acme-chief07, passive to deployment-acme-chief08 2451580 wikitext text/x-wiki rczep0kwt647jx0iexfrhuysmm21uj9 2451587 2451580 2026-08-25T21:23:21Z Stashbot 7414 brett: revert deployment-prep acme-chief switch: acme-chief active to deployment-acme-chief05 and passive to deployment-acme-chief06 2451587 wikitext text/x-wiki 65gdekkvxwegbbebdb92ts4s45i5xym 2451589 2451587 2026-08-25T21:32:02Z Stashbot 7414 brett: Switch acme-chief active to deployment-acme-chief07, passive to deployment-acme-chief08 2451589 wikitext text/x-wiki jrugvoufz0119wzns6sqybavej76rwl 0 14218 2451594 2450911 2026-08-25T22:53:58Z 1234qwer1234qwer4 39576 grammar 2451594 wikitext text/x-wiki ljhk93xrk38d0enr8bl6exsjnj3js04 0 17290 2451495 2451246 2026-08-25T15:07:00Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/121 (T435368) 2451495 wikitext text/x-wiki 5m0cy9wm24mpklfwsvy5o95007dy6eg 2451566 2451495 2026-08-25T20:31:27Z Stashbot 7414 Southparkfan: T401839 - provisioned deployment-docker-wikifeeds01 (trixie) using Tofu (cloudvps-repos/deployment-prep/tofu-provisioning) 2451566 wikitext text/x-wiki p7uazvxvq32yqwe5nc0q51lbxhe909b 2451579 2451566 2026-08-25T21:03:27Z Stashbot 7414 brett: Switch acme-chief active to deployment-acme-chief07, passive to deployment-acme-chief08 2451579 wikitext text/x-wiki 9jorn17plnyegpmdiwivhqcjadsbgli 2451586 2451579 2026-08-25T21:23:21Z Stashbot 7414 brett: revert deployment-prep acme-chief switch: acme-chief active to deployment-acme-chief05 and passive to deployment-acme-chief06 2451586 wikitext text/x-wiki d2zar4zbz9vb9wyi603a8zo6tsvaphq 2451588 2451586 2026-08-25T21:32:02Z Stashbot 7414 brett: Switch acme-chief active to deployment-acme-chief07, passive to deployment-acme-chief08 2451588 wikitext text/x-wiki ll7t86bgnfbarmt1lzdwlndy7navzud 2451610 2451588 2026-08-26T07:11:44Z Stashbot 7414 hashar: integration: updated Quibble jobs to replace deprecated `--commands` option by 1..n `--command` option(s) | https://gerrit.wikimedia.org/r/c/integration/config/+/1293711 | T321617 2451610 wikitext text/x-wiki s021xviw46612hmlf08sjk8m3ddjh7f 0 20225 2451496 2435511 2026-08-25T15:08:21Z XCollazo-WMF 31846 /* How to deploy */ Updating to an-test-coord1002 2451496 wikitext text/x-wiki j16evb947g0asi8nj0b94yfrdkhu9fw 2451497 2451496 2026-08-25T15:08:45Z XCollazo-WMF 31846 /* How to deploy */ 2451497 wikitext text/x-wiki iklxt4sdj5pkpb82k1kcvimjox157bu 0 24248 2451623 2441359 2026-08-26T07:35:45Z ADenisse-WMF 31462 Add instructions for using devscripts for non-debian users. 2451623 wikitext text/x-wiki jsn3y3zo60fcuvwwtak64dvyulqolr9 498 30942 2451422 2451342 2026-08-25T13:06:08Z Stashbot 7414 dcaro: stopped keepalived on proxy-5 to force a handover to proxy-6 2451422 wikitext text/x-wiki 536k3b2u1k5vlf9r119b9s1rtyddglo 2451423 2451422 2026-08-25T13:06:36Z Stashbot 7414 dcaro: stopped puppet on proxy-5 to avoid restarting keepalived 2451423 wikitext text/x-wiki 96o87qvgg6gxdhdgl11g2xxynv4od39 2451429 2451423 2026-08-25T13:22:58Z Stashbot 7414 godog: failover dumps-nfs from clouddumps1002 to clouddumps1001 2451429 wikitext text/x-wiki t4hzmu16d2khaevg0v13t3qn7ijvdf6 498 55577 2451426 2450484 2026-08-25T13:18:04Z Stashbot 7414 godog: cordon and reboot paws-127d-mq2566hee6zs-node-1 2451426 wikitext text/x-wiki 6blqlsq964l32p0aq8dnnguutro2kmf 2451435 2451426 2026-08-25T13:28:49Z Stashbot 7414 godog: reboot remaining worker nodes to pick up nfs changes 2451435 wikitext text/x-wiki ksd3cyox2ksf09pf56jucbfd9977afb 0 202567 2451650 2432502 2026-08-26T09:04:51Z Tiziano Fogli 37623 /* Troubleshooting */ 2451650 wikitext text/x-wiki orml39eeg4mqleimv94k5tamojgsm7q 2451652 2451650 2026-08-26T09:05:59Z Tiziano Fogli 37623 /* Downtiming */ 2451652 wikitext text/x-wiki scipind3cwtmszqz6n9vrsx3tqt3rqi 2451653 2451652 2026-08-26T09:06:38Z Tiziano Fogli 37623 /* Downtiming */ 2451653 wikitext text/x-wiki omzwzezspota98mkvl8a3kqdknx0zw0 2451657 2451653 2026-08-26T09:14:11Z Tiziano Fogli 37623 /* Troubleshooting */ 2451657 wikitext text/x-wiki 1ww0yznhz5i9zs5x5eckv37fe5pnq5v 2451672 2451657 2026-08-26T09:34:51Z Tiziano Fogli 37623 /* deadmanswitchonamdb module [Alertmanager-(active|passive):(Prometheus|Thanos) Watchdog Alerts Presence] */ 2451672 wikitext text/x-wiki sinlzauo45q2b3z3w86xz5k3j0jz1yc 2451674 2451672 2026-08-26T09:35:24Z Tiziano Fogli 37623 /* deadmanswitchonamdb module [Alertmanager-(active|passive):(Prometheus|Thanos) Watchdog Alerts Presence] */ 2451674 wikitext text/x-wiki a9wgk4s8cz5gsdh7ybhqdxfa5mzzhvf 2451675 2451674 2026-08-26T09:35:36Z Tiziano Fogli 37623 /* deadmanswitchonamdb module [Alertmanager-(active|passive):(Prometheus|Thanos) Watchdog Alerts Presence] */ 2451675 wikitext text/x-wiki ohrllmiy3so7mqeqwiw6sju69uh1zey 2451736 2451675 2026-08-26T10:42:03Z Tiziano Fogli 37623 /* Downtiming */ 2451736 wikitext text/x-wiki 1o7l9rm4ji6p56yvvz2vgzif9lz6pqp 498 441263 2451532 2374450 2026-08-25T18:08:28Z Stashbot 7414 wmbot~bd808@tools-bastion-14: Restarted webservice after reports of slowness. Still seems slow. 2451532 wikitext text/x-wiki n5i0uc4wy8uay130xr8nnpkkddo3ayg 2451535 2451532 2026-08-25T18:14:49Z Stashbot 7414 wmbot~bd808@tools-bastion-14: Update to 24fb23a6 (T401818) 2451535 wikitext text/x-wiki so3qsvdmo8ld3z38t05c9misvkmcsfr 498 443628 2451461 2450742 2026-08-25T14:34:48Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32859995726 (https://github.com/cluebotng/component-configs/commits/8e54667ba485459bad6bd7798274c98ea487caee) 2451461 wikitext text/x-wiki nvufzs715smzglclvgfpb2hifdlf5x0 2451503 2451461 2026-08-25T15:37:23Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32866638493 (https://github.com/cluebotng/component-configs/commits/f0c0373d1612550e858604e6c40555d78d071318) 2451503 wikitext text/x-wiki gvfzhad3w95rfh6rs69wddf8j3uy8w7 498 443800 2451389 2450323 2026-08-25T12:00:37Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 5cc772f346 (use MariaDB INSERT…RETURNING) 2451389 wikitext text/x-wiki 4cqcenwpqv3r6273239wgpuw0kizsxe 0 446764 2451494 2201903 2026-08-25T15:05:43Z XCollazo-WMF 31846 /* Hosts and their roles */ Updating to an-test-coord1002.eqiad.wmnet 2451494 wikitext text/x-wiki o3ylvp082xdmj9k6qp6hyk6et88i8rx 0 448411 2451673 2430035 2026-08-26T09:34:57Z LSobanski (WMF) 18783 /* WMF upgrade path */ Removed explicit references to primary and replicas 2451673 wikitext text/x-wiki tutjxz8gfavobc6tyzegwos9vvhg83k 0 449160 2451595 2451180 2026-08-26T00:01:12Z Dexbot 30554 Bot: Updating the report 2451595 wikitext text/x-wiki 7cm4mq3rhucsjzv0izsmn7piokjr4rz 0 451075 2451550 2451143 2026-08-25T19:06:53Z WFan (WMF) 38087 2451550 wikitext text/x-wiki fcsd68e9mnxxsg6rxwlxoczowr5m53w 0 452797 2451681 2387549 2026-08-26T09:39:47Z FNegri-WMF 32595 /* Checking the replication status */ update replica hostname 2451681 wikitext text/x-wiki 8zck8cozgg2rs7kb2i89tompv1i7d3y 116 453906 2451481 2451252 2026-08-25T14:57:24Z Gitlabaccountapprovalbot 37332 @wilfredor2 was approved. 2451481 wikitext text/x-wiki sf9ai1cfg17h9i3c0696w5b9jxilre7 2451525 2451481 2026-08-25T17:27:26Z Gitlabaccountapprovalbot 37332 @skim was approved. 2451525 wikitext text/x-wiki kzdko58fjw9zv0k79311psfym0atlnl 0 454144 2451607 2400933 2026-08-26T07:11:26Z Filippo Giunchedi 2939 Update to include dumps-nfs.w.o 2451607 wikitext text/x-wiki osbgaprldp9vyzplhdoslhvfjp7iaho 4 458615 2451581 2450646 2026-08-25T21:03:48Z Jdforrester (WMF) 468 /* Inactive administrator removal */ Reply 2451581 wikitext text/x-wiki 4dcgpz3pqnuucznql69vyvd501ux5nj 0 459290 2451440 2450478 2026-08-25T13:41:21Z XCollazo-WMF 31846 In progress.. 2451440 wikitext text/x-wiki jtu6h7f9uab6efe7wvgr6kko1xprcgg 2451498 2451440 2026-08-25T15:11:11Z XCollazo-WMF 31846 /* 2026-08-25 Tuesday (in progress...) */ Marking 2026-08-25 as finished 2451498 wikitext text/x-wiki 0901u6nzrubi6v30dit4oztqv0bggr1 0 459560 2451465 2445674 2026-08-25T14:46:29Z DAdedoyin (WMF) 39150 Test 2451465 wikitext text/x-wiki o6c13dxu7brm65jrt2gw5zqvk0di0mw 0 459699 2451551 2451189 2026-08-25T19:12:20Z Stashbot 7414 larssandergreen: civicrm upgraded from 438f38be to 1c51768b 2451551 wikitext text/x-wiki fnvfzr07z8e7qv65nvdcc5goxl5v3bg 2451598 2451551 2026-08-26T02:18:24Z Stashbot 7414 eileen: civicrm upgraded from 1c51768b to 1fe235e5 2451598 wikitext text/x-wiki 55shv6dqolaqiotyyz07xd5sfln2g8t 2451599 2451598 2026-08-26T02:51:54Z Stashbot 7414 eileen: civicrm upgraded from 1fe235e5 to 92c276d6 2451599 wikitext text/x-wiki kugie7y546f2i63qst0unnv1pb1qmm1 0 460234 2451553 2448482 2026-08-25T19:23:48Z CMing (WMF) 23277 /* Step 5: Complete the Pre-Launch Checklist */ 2451553 wikitext text/x-wiki f3naw8ntdg7zbrbb9d3osle0fe9unvz 2451554 2451553 2026-08-25T19:30:50Z CMing (WMF) 23277 /* How do I…? */ 2451554 wikitext text/x-wiki 0hvgglx8f8c5e2nuwtva1yv38w30dg3 0 460375 2451507 2451140 2026-08-25T16:15:10Z TBurmeister (WMF) 27417 Add section links to specific rewrite docs and examples in Gitlab 2451507 wikitext text/x-wiki ivkjmmw8hugcpmq8chi96f68xyovn9g 1 460524 2451593 2444754 2026-08-25T22:16:46Z Meno25 1446 fix link 2451593 wikitext text/x-wiki 4n5dietubule70fafzw60ymlj02jnzv 498 460577 2451508 2450666 2026-08-25T16:22:21Z Stashbot 7414 wmbot~bd808@tools-bastion-14: Built new container for 6da7257e 2451508 wikitext text/x-wiki t71o68093do5kqkrzl1e653kfn5o84j 2451509 2451508 2026-08-25T16:23:54Z Stashbot 7414 wmbot~bd808@tools-bastion-14: `toolforge webservice restart` to pick up new container. 2451509 wikitext text/x-wiki 3dnya5cv9hvpl3yjbntbsn13ma8n8gz 0 460582 2451506 2451147 2026-08-25T15:58:37Z TBurmeister (WMF) 27417 Add links to Gitlab docs and examples for each construct 2451506 wikitext text/x-wiki fr6wuftr4f662v3vkna1a9alx320ttu 3 460588 2451529 2026-08-25T18:01:18Z StrikerBot 8475 Welcome to Toolforge! 2451529 wikitext text/x-wiki oiq5q0vghlzlktvpnv21u070s0odck6