Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.12 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 328 2441785 2418296 2026-07-28T14:48:13Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2441785 wikitext text/x-wiki 7o14up6ydlou8eckj6b2vbegmsivz0i 2441798 2441785 2026-07-28T14:58:17Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) 2441798 wikitext text/x-wiki cjr8g6my227k656xbqjjsvy4izw26i5 2441923 2441798 2026-07-28T17:20:27Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service 2441923 wikitext text/x-wiki omiklrixfqif83t1esw8o9n2240q74t 2441924 2441923 2026-07-28T17:21:01Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) 2441924 wikitext text/x-wiki p3gug5dqn2rr6ogqs199u5kyvmrxk5q 2441947 2441924 2026-07-28T17:39:07Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service 2441947 wikitext text/x-wiki hxmyglkckxrh92kun7ahjfg65lxz6ml 2441948 2441947 2026-07-28T17:39:16Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) 2441948 wikitext text/x-wiki 0dfcz5e4v8j69b8obzyqqt3pmfl2vcv 2441950 2441948 2026-07-28T17:40:41Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service 2441950 wikitext text/x-wiki dmts788t6mmg2kd8gmo50x63uhus87k 2441952 2441950 2026-07-28T17:42:46Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) 2441952 wikitext text/x-wiki scsqwk9e1jpzyv1lxmi7mf3ylvkzhd1 498 3086 2442161 2441451 2026-07-29T08:56:41Z Stashbot 7414 godog: enable 'dumps_use_nfs_lb: true' for bastion/static/mail prefixes - T432325 2442161 wikitext text/x-wiki 3ii2t4w8pkz9o7nj4bv6qbado78a8im 2442192 2442161 2026-07-29T09:40:10Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T433184) 2442192 wikitext text/x-wiki kzfx85aty7u29s882rpj36a55wipxau 2442193 2442192 2026-07-29T09:40:21Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T433184) 2442193 wikitext text/x-wiki peui9q3plpxei82jmpp6pky5avgdjmc 2442203 2442193 2026-07-29T09:45:27Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T433184) 2442203 wikitext text/x-wiki 0bxf5d9ohf5nuxrrc302sitr1e4mswh 2442205 2442203 2026-07-29T09:45:38Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T433184) 2442205 wikitext text/x-wiki 4hgvuaa3ootp57f3pfsibrlus89hefq 2442206 2442205 2026-07-29T09:48:01Z Stashbot 7414 godog: enable 'dumps_use_nfs_lb: true' for nfs worker prefix - T432325 2442206 wikitext text/x-wiki 7pbhpl1dppplrtr03sv6qxnhsyzq4lr 2442210 2442206 2026-07-29T09:54:37Z Stashbot 7414 godog: test dumps-nfs failover - T432325 2442210 wikitext text/x-wiki p4euug08ucxijau57mv0t4swha9pja1 0 4108 2441603 2441334 2026-07-28T12:18:41Z ScheduleDeploymentBot 37566 Add [[gerrit:1315908]] to Tuesday, July 28 UTC afternoon backport window 2441603 wikitext text/x-wiki forvm246qiy6njwhyenxzfd6w7vr5ch 2441604 2441603 2026-07-28T12:18:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1313960]] to Tuesday, July 28 UTC afternoon backport window 2441604 wikitext text/x-wiki nlsworjak9d0ptjdyeoyodgug3x4ojb 2441937 2441604 2026-07-28T17:34:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1318747]] to Tuesday, July 28 UTC late backport window 2441937 wikitext text/x-wiki 0zkmi5zsp3bx3ssdbsq8wlwct1hwgve 2441940 2441937 2026-07-28T17:35:22Z ABreault (WMF) 37749 /* {{Deployment_day|date=2026-07-28}} */ 2441940 wikitext text/x-wiki f8r8e88z9og2jwhydp8n7nu21emp6gk 2441957 2441940 2026-07-28T17:50:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1318753]] to Tuesday, July 28 UTC late backport window 2441957 wikitext text/x-wiki 3fzfrfjwwr3v8r8axaza8wjmmynm61c 2441959 2441957 2026-07-28T17:54:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1318754]] to Tuesday, July 28 UTC late backport window 2441959 wikitext text/x-wiki 216wpwgepbv09ieniuysr862ngjuuz7 2441960 2441959 2026-07-28T17:56:15Z ScheduleDeploymentBot 37566 Add [[gerrit:1318756]] to Tuesday, July 28 UTC late backport window 2441960 wikitext text/x-wiki cpl9fdpes0iiznyat1tjqccovak360v 2442038 2441960 2026-07-28T19:59:55Z ScheduleDeploymentBot 37566 Add [[gerrit:1318718]] to Tuesday, July 28 UTC late backport window 2442038 wikitext text/x-wiki oe6xe756hye8im7s085x7qpez728x77 2442099 2442038 2026-07-29T00:52:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1306988]] to Wednesday, July 29 UTC late backport window 2442099 wikitext text/x-wiki m9sm48km9tdpbui2j8csu77e4vpv6fv 2442100 2442099 2026-07-29T00:53:23Z ScheduleDeploymentBot 37566 Add [[gerrit:1312694]] to Wednesday, July 29 UTC late backport window 2442100 wikitext text/x-wiki t33365r2c919apk87c5owtpkz945ynj 2442107 2442100 2026-07-29T06:07:53Z ScheduleDeploymentBot 37566 Add [[gerrit:1309675]] to Wednesday, July 29 UTC morning backport window 2442107 wikitext text/x-wiki d38o1w023b09mjp2jn9q9asseps1db7 2442255 2442107 2026-07-29T11:41:39Z Daimona Eaytoy 11462 /* {{Deployment_day|date=2026-07-29}} */ +window for [[phab:T429339]] 2442255 wikitext text/x-wiki 96o7rgpl5qroe5ot4q780srpfqe9p0t 0 7919 2441581 2441579 2026-07-28T12:01:12Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2006.codfw.wmnet 2441581 wikitext text/x-wiki oyt393kxk53z1ciqh2pqve0q2jjywp1 2441582 2441581 2026-07-28T12:01:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2006.codfw.wmnet 2441582 wikitext text/x-wiki eyk40t0whbd7fj42pgwq9lv77o6sjw3 2441583 2441582 2026-07-28T12:01:19Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2007.codfw.wmnet 2441583 wikitext text/x-wiki 0mhf2jll2mt5n7lotzfjdgvz8ze4oj3 2441584 2441583 2026-07-28T12:02:46Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance 2441584 wikitext text/x-wiki i48njeho1y9cf9vq3iimn5ctijfwaos 2441585 2441584 2026-07-28T12:02:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2236 (T431660)', diff saved to https://phabricator.wikimedia.org/P95348 and previous config saved to /var/cache/conftool/dbconfig/20260728-120253-cwilliams.json 2441585 wikitext text/x-wiki r9yubnd45j56bs9aic4pe8wtl0fyypq 2441588 2441585 2026-07-28T12:06:26Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2007.codfw.wmnet 2441588 wikitext text/x-wiki rgbaz4ct29y96hninwdgkld81pa1nna 2441591 2441588 2026-07-28T12:09:17Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2236: Maintenance 2441591 wikitext text/x-wiki qj9zb2ucep2cu5vr0y3aiepge7giaam 2441592 2441591 2026-07-28T12:11:35Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2007.codfw.wmnet 2441592 wikitext text/x-wiki 75j3dae1mp4mz5a0oixgtb4yceknb8u 2441593 2441592 2026-07-28T12:11:37Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2007.codfw.wmnet 2441593 wikitext text/x-wiki mhrweuvxgbn1ualixhhu7lqywu74slc 2441594 2441593 2026-07-28T12:11:43Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2008.codfw.wmnet 2441594 wikitext text/x-wiki 0ij2b3ez8s60uf4wko2a7g9pb3yon37 2441597 2441594 2026-07-28T12:12:54Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1137.eqiad.wmnet with OS trixie 2441597 wikitext text/x-wiki 0qd37j7zav2o3gp1ypvkhgkw5nf4qyb 2441598 2441597 2026-07-28T12:14:54Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2236: Maintenance 2441598 wikitext text/x-wiki 0da5k7m25sgvqug4bkh7xrk2jea18bi 2441599 2441598 2026-07-28T12:15:00Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2236: Maintenance 2441599 wikitext text/x-wiki ppv7dajbdp72crzflkkkzd2b4rb80h7 2441600 2441599 2026-07-28T12:16:07Z Stashbot 7414 pt1979@cumin1003: START - Cookbook sre.network.tls for network device asw1-604-eqsin 2441600 wikitext text/x-wiki 9kw0l9j3wos9ks855zprz40s2gvxext 2441601 2441600 2026-07-28T12:16:49Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2008.codfw.wmnet 2441601 wikitext text/x-wiki 9w31df9c2sbluu3v5oun5tplg9x7gul 2441602 2441601 2026-07-28T12:16:54Z Stashbot 7414 pt1979@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-604-eqsin 2441602 wikitext text/x-wiki 1993zmw2cbcsbhetox5u5nrqwfyjey3 2441605 2441602 2026-07-28T12:21:58Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2008.codfw.wmnet 2441605 wikitext text/x-wiki t0z2irgrb7ewfatz7dlb2u8ghyd63ni 2441606 2441605 2026-07-28T12:22:01Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2008.codfw.wmnet 2441606 wikitext text/x-wiki d1pqi61luac5dfx4dojke2sybo9eg5l 2441607 2441606 2026-07-28T12:22:05Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2009.codfw.wmnet 2441607 wikitext text/x-wiki 1f62cgfrlggrc66qfkgw64bpagdf1jr 2441608 2441607 2026-07-28T12:27:13Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2009.codfw.wmnet 2441608 wikitext text/x-wiki 3ydxjfasqm17povcl9p9y1z0tl3jzz7 2441609 2441608 2026-07-28T12:31:53Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2009.codfw.wmnet 2441609 wikitext text/x-wiki tonju9a08h0n4g71rr6huagv6iem25c 2441610 2441609 2026-07-28T12:31:55Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2009.codfw.wmnet 2441610 wikitext text/x-wiki 960izb5zlwc459x9lpvwbp819ms871l 2441611 2441610 2026-07-28T12:32:02Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2010.codfw.wmnet 2441611 wikitext text/x-wiki mtssmtxtt205at43a3kbjfth3j3kea7 2441612 2441611 2026-07-28T12:32:07Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply 2441612 wikitext text/x-wiki nn4um5r2fvp6qpq505em4j308wo5vqh 2441613 2441612 2026-07-28T12:32:23Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply 2441613 wikitext text/x-wiki 1zx0tziu7evybk8wgave0cwxwwzaba0 2441614 2441613 2026-07-28T12:32:28Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply 2441614 wikitext text/x-wiki jnxsrj4bqe5v5tsdxkubjd0hvq2365o 2441615 2441614 2026-07-28T12:32:32Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply 2441615 wikitext text/x-wiki ojglx3l1kp0p4w4pw92zpy9u3ona5k3 2441616 2441615 2026-07-28T12:32:38Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/kartotherian: apply 2441616 wikitext text/x-wiki nw800f7lsznvcrstkg6w4jdr68ve1ua 2441617 2441616 2026-07-28T12:34:55Z Stashbot 7414 pt1979@cumin2003: START - Cookbook sre.dns.netbox 2441617 wikitext text/x-wiki ptpx4namti08x2rjqbw6byrluki4y4v 2441618 2441617 2026-07-28T12:37:07Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2010.codfw.wmnet 2441618 wikitext text/x-wiki k1uc950ggd2txluqyihbu5fcyhvmnax 2441619 2441618 2026-07-28T12:41:39Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2010.codfw.wmnet 2441619 wikitext text/x-wiki 6z52qmfcwvwv87j1fc7emjccmhg3o5n 2441620 2441619 2026-07-28T12:41:41Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2010.codfw.wmnet 2441620 wikitext text/x-wiki qhcvp3rqp3itde65qlw63g98s11uh9q 2441621 2441620 2026-07-28T12:41:45Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2011.codfw.wmnet 2441621 wikitext text/x-wiki 0np9x9u1cwdy7pkvqj7con8ikbovy7z 2441622 2441621 2026-07-28T12:42:54Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply 2441622 wikitext text/x-wiki 83k7pvtgs1pxpq7oyakfbn286gxgd0e 2441623 2441622 2026-07-28T12:44:29Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Maintenance 2441623 wikitext text/x-wiki rjwmc7ibaz1v6nq8z96t9o10ibkzsiu 2441624 2441623 2026-07-28T12:44:51Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance 2441624 wikitext text/x-wiki ikur69sfrla8x581td1jgmzk98ljao3 2441625 2441624 2026-07-28T12:44:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2167 (T431660)', diff saved to https://phabricator.wikimedia.org/P95354 and previous config saved to /var/cache/conftool/dbconfig/20260728-124457-cwilliams.json 2441625 wikitext text/x-wiki c2y79oduwma85uo9pdo8yvrksh1vjm8 2441626 2441625 2026-07-28T12:45:32Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Maintenance 2441626 wikitext text/x-wiki avx2varmegjgy50w39mttct4bichi2m 2441627 2441626 2026-07-28T12:45:54Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance 2441627 wikitext text/x-wiki ggcuvlc8rysv8h05d1xsyziwh6n3pjn 2441628 2441627 2026-07-28T12:46:03Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2221 (T431660)', diff saved to https://phabricator.wikimedia.org/P95357 and previous config saved to /var/cache/conftool/dbconfig/20260728-124601-cwilliams.json 2441628 wikitext text/x-wiki mmefk5mlvn8haqlwom1aj7eeiy81al5 2441629 2441628 2026-07-28T12:46:52Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2011.codfw.wmnet 2441629 wikitext text/x-wiki 87g297ihzi88ounj2nwbed44uqfod3d 2441630 2441629 2026-07-28T12:48:33Z Stashbot 7414 pt1979@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add loopback for asw1-603 - pt1979@cumin2003" 2441630 wikitext text/x-wiki 1hscspusof6ot5g08twtspbfck4azzt 2441631 2441630 2026-07-28T12:49:16Z Stashbot 7414 pt1979@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add loopback for asw1-603 - pt1979@cumin2003" 2441631 wikitext text/x-wiki 5uc88wir4w0ab7gl4i4y5xregzh0uh4 2441632 2441631 2026-07-28T12:49:17Z Stashbot 7414 pt1979@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441632 wikitext text/x-wiki cimo67i92480mq756f3mwnf1m2sxu3e 2441633 2441632 2026-07-28T12:51:21Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2011.codfw.wmnet 2441633 wikitext text/x-wiki 06bfenp0hjtns38y63ne7gmuvd8gh97 2441634 2441633 2026-07-28T12:51:23Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2011.codfw.wmnet 2441634 wikitext text/x-wiki foysvs6kd8h89w3q0sbeva8oqoh7vef 2441635 2441634 2026-07-28T12:51:25Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-codfw 2441635 wikitext text/x-wiki kbz0bpmj60kxng04l05uncwjh7d9004 2441636 2441635 2026-07-28T12:51:28Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2167: Maintenance 2441636 wikitext text/x-wiki g82z25df5nswedhrw4rvrn1rv475itj 2441637 2441636 2026-07-28T12:52:08Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance 2441637 wikitext text/x-wiki hnmyso27ay76cg0ek71enae1p5civhz 2441638 2441637 2026-07-28T12:52:16Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2153 (T431660)', diff saved to https://phabricator.wikimedia.org/P95358 and previous config saved to /var/cache/conftool/dbconfig/20260728-125214-cwilliams.json 2441638 wikitext text/x-wiki ccodc9k8egx88oy5ivmzjtaqes1m390 2441639 2441638 2026-07-28T12:52:18Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2221: Maintenance 2441639 wikitext text/x-wiki lep0apbvuyvf8ocngs29dokubcrrnnb 2441647 2441639 2026-07-28T12:58:50Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Maintenance 2441647 wikitext text/x-wiki k3al4sq4g4rmq9byp4q0g12xa1eyvtb 2441652 2441647 2026-07-28T13:00:40Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2236: Maintenance 2441652 wikitext text/x-wiki 66934bt2vt4p2ykox1zpynpidu696jh 2441655 2441652 2026-07-28T13:01:01Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance 2441655 wikitext text/x-wiki 0wnoax87l2ihsdbpyhaqmtgklpz4oat 2441656 2441655 2026-07-28T13:01:09Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2237 (T431660)', diff saved to https://phabricator.wikimedia.org/P95362 and previous config saved to /var/cache/conftool/dbconfig/20260728-130107-cwilliams.json 2441656 wikitext text/x-wiki 3am6xq4uxvuqcbz75lbawc3xvm7g9te 2441658 2441656 2026-07-28T13:02:34Z Stashbot 7414 esanders@deploy1003: Started scap sync-world: Backport for [[gerrit:1309727|Set wgMFFallbackEditor to 'visual' on enwiki (T431858)]] 2441658 wikitext text/x-wiki dasv4fzr2pi356j5598xzg8vw3v301z 2441665 2441658 2026-07-28T13:04:27Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2153: Maintenance 2441665 wikitext text/x-wiki 6zpkckb8aycfep81qt7pltqssl0lag0 2441667 2441665 2026-07-28T13:04:35Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Maintenance 2441667 wikitext text/x-wiki lp4x1xm3cf21mf3tjvk0p8cisi6hoyw 2441668 2441667 2026-07-28T13:04:43Z Stashbot 7414 esanders@deploy1003: esanders: Backport for [[gerrit:1309727|Set wgMFFallbackEditor to 'visual' on enwiki (T431858)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441668 wikitext text/x-wiki pn9q5ad4u66ob6ew0w93qyx880ciagp 2441673 2441668 2026-07-28T13:06:23Z Stashbot 7414 esanders@deploy1003: esanders: Continuing with deployment 2441673 wikitext text/x-wiki 46qrh2lkgj8cctjh0sexenncb4wwpnu 2441674 2441673 2026-07-28T13:07:43Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2237: Maintenance 2441674 wikitext text/x-wiki fhsyfu1iwupuo2ebfgt5r494dp0um0z 2441675 2441674 2026-07-28T13:08:16Z Stashbot 7414 pt1979@cumin2003: START - Cookbook sre.dns.netbox 2441675 wikitext text/x-wiki bi786zjzcj9cywmwp51zxiufak1dwhb 2441676 2441675 2026-07-28T13:10:45Z Stashbot 7414 esanders@deploy1003: Finished scap sync-world: Backport for [[gerrit:1309727|Set wgMFFallbackEditor to 'visual' on enwiki (T431858)]] (duration: 08m 11s) 2441676 wikitext text/x-wiki 0a5cmskgp30mnotjp2q7wwikr4uk35m 2441677 2441676 2026-07-28T13:11:39Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2441677 wikitext text/x-wiki rkma8i62ppm9qq0vtgrblqa3ijmxqdp 2441678 2441677 2026-07-28T13:11:45Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 20s) 2441678 wikitext text/x-wiki hebe7pzn850jr7yddbsifhy3x2r5htt 2441680 2441678 2026-07-28T13:12:50Z Stashbot 7414 pt1979@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add loopback IPV6 for asw1-604 - pt1979@cumin2003" 2441680 wikitext text/x-wiki r19osylk1a1cibwf2koo34eiwaqo6dr 2441681 2441680 2026-07-28T13:12:53Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2441681 wikitext text/x-wiki 0ugcsthytz8ckl8lqe0oc6t2x5tdebv 2441682 2441681 2026-07-28T13:12:55Z Stashbot 7414 pt1979@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add loopback IPV6 for asw1-604 - pt1979@cumin2003" 2441682 wikitext text/x-wiki 0ub7xbsroad1zi7504antlqr79dmaco 2441683 2441682 2026-07-28T13:12:57Z Stashbot 7414 pt1979@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441683 wikitext text/x-wiki 8gxne8qwelhqmr4140uraumv8reitks 2441684 2441683 2026-07-28T13:13:13Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2441684 wikitext text/x-wiki b5l8f5hdaqk3gvkonga6p42gicdt38k 2441685 2441684 2026-07-28T13:13:20Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2237: Maintenance 2441685 wikitext text/x-wiki sw3tows9eg79xge2p56fffqcjj02e9n 2441686 2441685 2026-07-28T13:13:26Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2237: Maintenance 2441686 wikitext text/x-wiki oesny1mwcpp1861gz74rkrh6x7wx5g1 2441688 2441686 2026-07-28T13:14:06Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1315908|magwiki: update logos (T433011)]], [[gerrit:1313960|bolwiki: add logo, sitename, projectnamespace and timezone (T429951)]] 2441688 wikitext text/x-wiki 1s15oljwx246rpa3d2y89eco3j1hq53 2441689 2441688 2026-07-28T13:14:11Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003 2441689 wikitext text/x-wiki 3bof5j8iu3rhyzju8k838t2acf2po3d 2441690 2441689 2026-07-28T13:16:07Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, anzx: Backport for [[gerrit:1315908|magwiki: update logos (T433011)]], [[gerrit:1313960|bolwiki: add logo, sitename, projectnamespace and timezone (T429951)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2441690 wikitext text/x-wiki pf1xudtmnnl5j61i4bgy6wz32h4cw9s 2441691 2441690 2026-07-28T13:16:23Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441691 wikitext text/x-wiki 5gbi95z1qwqnai5gqlcjw1yk5gjhmv6 2441692 2441691 2026-07-28T13:16:25Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2012.codfw.wmnet 57.48.192.10.in-addr.arpa 7.5.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441692 wikitext text/x-wiki ibwb7u09esebgkovb6pc4ji0ya9j2dt 2441693 2441692 2026-07-28T13:16:27Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2012.codfw.wmnet 57.48.192.10.in-addr.arpa 7.5.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441693 wikitext text/x-wiki e7t253nbe3z7w5kve8qivdikaz4oxo3 2441694 2441693 2026-07-28T13:16:29Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2012 2441694 wikitext text/x-wiki 6t75a4amdsaswcof2omml4jh7frvc8r 2441695 2441694 2026-07-28T13:16:41Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2441695 wikitext text/x-wiki 7ovxlkdyikekmyillsx3851b8xhskj3 2441696 2441695 2026-07-28T13:17:10Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, anzx: Continuing with deployment 2441696 wikitext text/x-wiki dap27ol9qfcu9zfh42stdehutnhgi5s 2441697 2441696 2026-07-28T13:18:56Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2441697 wikitext text/x-wiki o6oajk56lj6gd9rubw5fimxpgk93hx9 2441698 2441697 2026-07-28T13:20:09Z Stashbot 7414 swfrench-wmf: authdns-update to direct codfw, eqsin, ulsfo etcd clients to eqiad - T428495 2441698 wikitext text/x-wiki 1ac526h8gieso9luev14bxnumikzlgr 2441699 2441698 2026-07-28T13:21:30Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1315908|magwiki: update logos (T433011)]], [[gerrit:1313960|bolwiki: add logo, sitename, projectnamespace and timezone (T429951)]] (duration: 07m 19s) 2441699 wikitext text/x-wiki ajvzjb35zswi4gfzoxsoatmbfv0bmxw 2441700 2441699 2026-07-28T13:21:49Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: mwscript-k8s job started: cleanupTitles bolwiki # T429951 2441700 wikitext text/x-wiki 83ydxqvotjy9mffmrkge9e1lcwz6qg6 2441702 2441700 2026-07-28T13:27:02Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2012 2441702 wikitext text/x-wiki 63ipn5i8tjirxt1bq4cvjfjtcj8qrr9 2441703 2441702 2026-07-28T13:27:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2012 2441703 wikitext text/x-wiki abq1xfgx0utlbdmi8mkhz706l48ks74 2441704 2441703 2026-07-28T13:27:14Z Stashbot 7414 swfrench-wmf: begin rolling restart of confd in codfw, eqsin, ulsfo - T428495 2441704 wikitext text/x-wiki pejhtq36b6b9ifxu0w83a1pbowylk24 2441706 2441704 2026-07-28T13:27:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1022.eqiad.wmnet, repooling source-only afterwards 2441706 wikitext text/x-wiki diwfl26aasmr60gytcaycezeqvjtpkq 2441707 2441706 2026-07-28T13:28:59Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: router upgrade, T431749] 2441707 wikitext text/x-wiki o3gdropi8004awci0wwkhck4xozoo9i 2441708 2441707 2026-07-28T13:29:05Z Stashbot 7414 ayounsi@cumin1003: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool esams [reason: router upgrade, T431749] 2441708 wikitext text/x-wiki e8zggusz8j1xv2vobd7pf2msmjzc7mk 2441709 2441708 2026-07-28T13:29:20Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: router upgrade, T431751] 2441709 wikitext text/x-wiki t3wtb9bwikcrwtwc8odxc2ui8ngxtpm 2441710 2441709 2026-07-28T13:29:34Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: router upgrade, T431751] 2441710 wikitext text/x-wiki simadaiwgti2crqjkjii7w4smuer8gj 2441711 2441710 2026-07-28T13:30:23Z Stashbot 7414 swfrench-wmf: restarted navtiming on webperf2003 - T428495 2441711 wikitext text/x-wiki 1bcaq2zsvxk9ei2onm6xuzfwqs18fgw 2441712 2441711 2026-07-28T13:38:29Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Maintenance 2441712 wikitext text/x-wiki dl60yilddyhd8tk38xlvvsdxmvcse2d 2441713 2441712 2026-07-28T13:38:51Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance 2441713 wikitext text/x-wiki jbybod4dimzbq4gul014c88ngc8b1dy 2441714 2441713 2026-07-28T13:38:59Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2181 (T431660)', diff saved to https://phabricator.wikimedia.org/P95374 and previous config saved to /var/cache/conftool/dbconfig/20260728-133857-cwilliams.json 2441714 wikitext text/x-wiki rre6w0z0mczws2b3sgi00way9uxbxx8 2441715 2441714 2026-07-28T13:39:20Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Maintenance 2441715 wikitext text/x-wiki e36o8w03tb7yjaffrvaho9s4ki9zyow 2441716 2441715 2026-07-28T13:39:42Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance 2441716 wikitext text/x-wiki q8hrg7mzrr0qsls3a76y05whxqm48ga 2441717 2441716 2026-07-28T13:39:50Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2222 (T431660)', diff saved to https://phabricator.wikimedia.org/P95376 and previous config saved to /var/cache/conftool/dbconfig/20260728-133948-cwilliams.json 2441717 wikitext text/x-wiki m5816menx4xlsqalh8ovu3qi77bw57s 2441718 2441717 2026-07-28T13:42:10Z Stashbot 7414 btullis@dns1004: START - running authdns-update 2441718 wikitext text/x-wiki hhy4yn6faid7x7axkw6fomwvfhb5frp 2441719 2441718 2026-07-28T13:42:21Z Stashbot 7414 sukhe: restart pybal on lvs2014 2441719 wikitext text/x-wiki 2i5grxjl8nzxgkeybzcefw7qbbi0rdh 2441720 2441719 2026-07-28T13:44:20Z Stashbot 7414 btullis@dns1004: END - running authdns-update 2441720 wikitext text/x-wiki 2fg8qsyaun9g05ima4p9rknaw035ekp 2441721 2441720 2026-07-28T13:45:07Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2181: Maintenance 2441721 wikitext text/x-wiki 4na6que3i2vuf29hlui2l80hka6z4rw 2441722 2441721 2026-07-28T13:45:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2012.codfw.wmnet with reason: host reimage 2441722 wikitext text/x-wiki 1lwt24388c517l6g4wxe4p06fdmmoea 2441723 2441722 2026-07-28T13:45:22Z Stashbot 7414 sukhe: restart pybal on A:lvs-codfw 2441723 wikitext text/x-wiki j3y95ckprwxxvixu163ba9t7lmko82p 2441724 2441723 2026-07-28T13:46:10Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2222: Maintenance 2441724 wikitext text/x-wiki cwvd8gxkvk3i7wuqft5xqksoqwoef1g 2441725 2441724 2026-07-28T13:49:05Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2441725 wikitext text/x-wiki ceu9x3xn9aphloaukmqyqu0xwr1unpn 2441726 2441725 2026-07-28T13:49:53Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2441726 wikitext text/x-wiki o79u0vuzxu6bt9e257fnpyqifbtsy8e 2441727 2441726 2026-07-28T13:50:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2012.codfw.wmnet with reason: host reimage 2441727 wikitext text/x-wiki m4dto836dqpqbfxokp6ao7s7xepo8oh 2441728 2441727 2026-07-28T13:50:14Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Maintenance 2441728 wikitext text/x-wiki bh3jm8p56mqjv71cfgfrbcaxwai3bl2 2441729 2441728 2026-07-28T13:50:18Z Stashbot 7414 XioNoX: drain cr2-esams - T431751 2441729 wikitext text/x-wiki 35dqjnyjq6bvz6vtdwj2glhh5z19a4k 2441730 2441729 2026-07-28T13:50:36Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance 2441730 wikitext text/x-wiki b3v1d3vgbs5uomr7o6xriizke6drwd6 2441731 2441730 2026-07-28T13:50:44Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2170 (T431660)', diff saved to https://phabricator.wikimedia.org/P95381 and previous config saved to /var/cache/conftool/dbconfig/20260728-135043-cwilliams.json 2441731 wikitext text/x-wiki gyfkyp9webjyn24ynbd7ysxs7ebbebb 2441732 2441731 2026-07-28T13:51:02Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T428495) 2441732 wikitext text/x-wiki bbl2f9gn5e78vpzdjvumaglzs1d3x5r 2441733 2441732 2026-07-28T13:51:46Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: router upgrade 2441733 wikitext text/x-wiki 5ytqe964zh4vwqrq0f5c502ejt14ed6 2441734 2441733 2026-07-28T13:52:54Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T428495) 2441734 wikitext text/x-wiki qbvkuzqyrhn9pd357ayfz09tlep4blb 2441735 2441734 2026-07-28T13:55:40Z Stashbot 7414 XioNoX: reboot cr2-esams - T431751 2441735 wikitext text/x-wiki njzh2bis2ygdlffld87xcom1jjpinfm 2441736 2441735 2026-07-28T13:56:58Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin (T428495) 2441736 wikitext text/x-wiki eir89twu31khsg1zjwt99xp5i0cr5w2 2441737 2441736 2026-07-28T13:57:21Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2170: Maintenance 2441737 wikitext text/x-wiki 6tc5v7ii3x2983exvl4bitwilmk1u1f 2441738 2441737 2026-07-28T13:59:40Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2237: Maintenance 2441738 wikitext text/x-wiki iuga2z9w8nrmy7qwcalaqxk3di0yvic 2441739 2441738 2026-07-28T14:00:04Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2240.codfw.wmnet with reason: Maintenance 2441739 wikitext text/x-wiki 63482krclzzqpyrt5yj7tp4j5xlcn1d 2441740 2441739 2026-07-28T14:00:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2240 (T431660)', diff saved to https://phabricator.wikimedia.org/P95384 and previous config saved to /var/cache/conftool/dbconfig/20260728-140011-cwilliams.json 2441740 wikitext text/x-wiki c9igxh4m5t9ym1tocicmh36yz2b91hg 2441741 2441740 2026-07-28T14:01:09Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin (T428495) 2441741 wikitext text/x-wiki boe0wk6xhtbo6b9i9jd6w8uj485mejs 2441742 2441741 2026-07-28T14:02:56Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad 2441742 wikitext text/x-wiki a8he8awe18qm13s62eqbtqnoz5zckl7 2441743 2441742 2026-07-28T14:03:00Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet 2441743 wikitext text/x-wiki r1hr3vv45qx1a4r623smfeysragp6q7 2441744 2441743 2026-07-28T14:05:49Z Stashbot 7414 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. 2441744 wikitext text/x-wiki j0l899hbgo83m4vnox5tbhdbyqn9t55 2441745 2441744 2026-07-28T14:06:17Z Stashbot 7414 XioNoX: un-drain cr2-esams - T431751 2441745 wikitext text/x-wiki mn9v2xtjgigmjo0ya5yzu3d4xnpb7pq 2441746 2441745 2026-07-28T14:06:44Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2240: Maintenance 2441746 wikitext text/x-wiki jjw5q2dhvv6cbdpl8zjhsncawmf3rcs 2441747 2441746 2026-07-28T14:06:46Z Stashbot 7414 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. 2441747 wikitext text/x-wiki 1f6go1gzy0r1qo9vnumwa0xhwxcl23q 2441748 2441747 2026-07-28T14:07:32Z Stashbot 7414 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. 2441748 wikitext text/x-wiki f91tb1s5fj4dcfi1nvo37tuhzxtzw6p 2441749 2441748 2026-07-28T14:07:55Z Stashbot 7414 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. 2441749 wikitext text/x-wiki tkr5yybsdxo9ofi9edy8i0aaq9expk4 2441750 2441749 2026-07-28T14:08:02Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet 2441750 wikitext text/x-wiki 9afh3uh6pylnfsto92a8cm25wryh62s 2441751 2441750 2026-07-28T14:11:19Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: router upgrade, T431751] 2441751 wikitext text/x-wiki gpkxt90iaq0q4d8h1dc3czd2fqg2v0v 2441752 2441751 2026-07-28T14:11:23Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: router upgrade, T431751] 2441752 wikitext text/x-wiki fs05ojjytm543dtnfe4vmgutevwup1c 2441753 2441752 2026-07-28T14:13:22Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet 2441753 wikitext text/x-wiki l1zbt1ecd90rquh5h9zbx080a10w9t6 2441754 2441753 2026-07-28T14:13:24Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet 2441754 wikitext text/x-wiki at4zb66dkuiiifpv4z6c4tjnnogbmr8 2441755 2441754 2026-07-28T14:13:29Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet 2441755 wikitext text/x-wiki ctqevas0ibzm34oyi2trvlmbwr2zehy 2441756 2441755 2026-07-28T14:17:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS bookworm 2441756 wikitext text/x-wiki dot37zia4g0v27c2cd9ielta50s863i 2441757 2441756 2026-07-28T14:18:31Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet 2441757 wikitext text/x-wiki bi55sj3amx37kmhqwiyd51wu9x72lmv 2441758 2441757 2026-07-28T14:23:38Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2441758 wikitext text/x-wiki c2v3kxru4exjtj0wovyg09n8aue0rn7 2441759 2441758 2026-07-28T14:23:47Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 19s) 2441759 wikitext text/x-wiki h11velw720njppqv6q5djnxw6nmp2rz 2441760 2441759 2026-07-28T14:23:50Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet 2441760 wikitext text/x-wiki rhebwvgzz98s4z6r1ggnizmehurxsto 2441761 2441760 2026-07-28T14:23:52Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet 2441761 wikitext text/x-wiki ddmzix5l54c199izuz8qwavgccu1itk 2441762 2441761 2026-07-28T14:23:57Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet 2441762 wikitext text/x-wiki i4nez66919evabiwzkjcutkspgojrvv 2441763 2441762 2026-07-28T14:25:29Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply 2441763 wikitext text/x-wiki qesxjf12np376vx7q99ao6bs08r3iz5 2441764 2441763 2026-07-28T14:25:32Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply 2441764 wikitext text/x-wiki mi6f3g16dc2cqgrlenkct4fyx1e1jwl 2441765 2441764 2026-07-28T14:25:39Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/kartotherian: apply 2441765 wikitext text/x-wiki nbma9hin9vf31pawkfndtsag4l9ejw6 2441766 2441765 2026-07-28T14:30:54Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2441766 wikitext text/x-wiki 4vsnam6cteo8sagoolaowysx76nxf0k 2441767 2441766 2026-07-28T14:31:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2441767 wikitext text/x-wiki f4ucs4q8hw04b2w9dcsw5uebzs7nc72 2441768 2441767 2026-07-28T14:31:50Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Maintenance 2441768 wikitext text/x-wiki 0zqkzob9jx9imy4r57yupb04390xxm0 2441769 2441768 2026-07-28T14:32:12Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance 2441769 wikitext text/x-wiki fgtf9k9i0dwqhv6gy6dvh78tuqp3gfu 2441770 2441769 2026-07-28T14:32:20Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2195 (T431660)', diff saved to https://phabricator.wikimedia.org/P95394 and previous config saved to /var/cache/conftool/dbconfig/20260728-143218-cwilliams.json 2441770 wikitext text/x-wiki 7ydtmmjcaaibh6ed6znpy8tumsi54k1 2441771 2441770 2026-07-28T14:33:15Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Maintenance 2441771 wikitext text/x-wiki 1yd75gisl65edzrtr83minf5bdrh6un 2441772 2441771 2026-07-28T14:34:00Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet 2441772 wikitext text/x-wiki fj2zyiyd5nk5iwkci6xshmizyg7ydu9 2441773 2441772 2026-07-28T14:36:46Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply 2441773 wikitext text/x-wiki 5m8oj0v097t3uzl30648vj486kz20l0 2441774 2441773 2026-07-28T14:36:50Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1022.eqiad.wmnet, repooling source-only afterwards 2441774 wikitext text/x-wiki jrmt9mwcy7dtvtj6lmzd82l5s8jnhaj 2441775 2441774 2026-07-28T14:38:39Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2195: Maintenance 2441775 wikitext text/x-wiki nu1g9iasiswwvnb8uthzi9l3i8cy9eo 2441776 2441775 2026-07-28T14:38:53Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet 2441776 wikitext text/x-wiki 8hytclw1f0huyy6adrkrbuuo4c20m32 2441777 2441776 2026-07-28T14:38:55Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet 2441777 wikitext text/x-wiki dyiaspqun8ow821q9pxqksbx4hlp7gr 2441778 2441777 2026-07-28T14:39:00Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet 2441778 wikitext text/x-wiki gtkg20z0dxnf6t5jk4wcfr5id2vl5dl 2441779 2441778 2026-07-28T14:39:04Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.reboot-single for host conf2004.codfw.wmnet 2441779 wikitext text/x-wiki h4doz9o913c1klfvqivfboo3frrx258 2441780 2441779 2026-07-28T14:44:02Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet 2441780 wikitext text/x-wiki onzz63dpjttbx02t2eaagsc70rsl1jh 2441781 2441780 2026-07-28T14:44:25Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2170: Maintenance 2441781 wikitext text/x-wiki n30ybm4p1eee1aeskluo51ngla6zu1f 2441782 2441781 2026-07-28T14:44:47Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance 2441782 wikitext text/x-wiki g6frz2jd1eshc50vazeqgt465osyjx9 2441783 2441782 2026-07-28T14:44:55Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2173 (T431660)', diff saved to https://phabricator.wikimedia.org/P95399 and previous config saved to /var/cache/conftool/dbconfig/20260728-144453-cwilliams.json 2441783 wikitext text/x-wiki 9jl548yc38ge0wcp8sfcsrx5nsnhb0b 2441784 2441783 2026-07-28T14:45:24Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2004.codfw.wmnet 2441784 wikitext text/x-wiki hv4f5v6l1ae05zl51xpnhjmceb0y3bx 2441786 2441784 2026-07-28T14:49:45Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 13 hosts 2441786 wikitext text/x-wiki l0cgwmccs6ya4c1pefrf96s9g6blvvd 2441787 2441786 2026-07-28T14:50:33Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet 2441787 wikitext text/x-wiki 0lnyj0yxftk2hpgf148e95qu9xweefi 2441788 2441787 2026-07-28T14:50:35Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet 2441788 wikitext text/x-wiki org7fo8upnsmsp6wqmgoty7z5nfonhe 2441789 2441788 2026-07-28T14:50:40Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet 2441789 wikitext text/x-wiki rlya1enet342gbc56fw65ur3x2yvw7a 2441790 2441789 2026-07-28T14:51:32Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 13 hosts 2441790 wikitext text/x-wiki 4ffpklxluhwy0e0xl3x8y3p7d9r90q8 2441791 2441790 2026-07-28T14:51:34Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2173: Maintenance 2441791 wikitext text/x-wiki 820yqx60pe14oxdubpl5rqavi9lcfni 2441793 2441791 2026-07-28T14:54:44Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2240: Maintenance 2441793 wikitext text/x-wiki 8cxb38hk2kaa8es5oks75m4y6gjv7h1 2441794 2441793 2026-07-28T14:55:07Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance 2441794 wikitext text/x-wiki 7mwo0zby5kv5g6j0ly153e49l0yeooc 2441795 2441794 2026-07-28T14:55:25Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2245-2247].codfw.wmnet with reason: Maintenance 2441795 wikitext text/x-wiki 0jqegrw3npb20abexipn9uuxlnk0zyv 2441796 2441795 2026-07-28T14:55:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2248 (T431660)', diff saved to https://phabricator.wikimedia.org/P95403 and previous config saved to /var/cache/conftool/dbconfig/20260728-145532-cwilliams.json 2441796 wikitext text/x-wiki skiyobfb80s4514iik2yve8jh4dxden 2441797 2441796 2026-07-28T14:55:43Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet 2441797 wikitext text/x-wiki 5pynnva8v8hjlhrj02dku0ngv67ycsn 2441799 2441797 2026-07-28T14:59:17Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host conf2004.codfw.wmnet with OS bookworm 2441799 wikitext text/x-wiki tslwk2miup3ib2blgva4l0dedabnshm 2441800 2441799 2026-07-28T14:59:52Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab1004.eqiad.wmnet with reason: deployment 2441800 wikitext text/x-wiki dj3z4e1lme8k72764ceer7ojk2gwm4v 2441801 2441800 2026-07-28T14:59:53Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2248: Maintenance 2441801 wikitext text/x-wiki 9nr0o4m888kmptz3zvdq85e9pp5vs3z 2441802 2441801 2026-07-28T15:00:49Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet 2441802 wikitext text/x-wiki n05y3ojjtzptgq09byuu4vfjpe211bc 2441803 2441802 2026-07-28T15:00:50Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet 2441803 wikitext text/x-wiki rvxnp15kg2ev0du2tujzrkw5h8yrzfb 2441804 2441803 2026-07-28T15:00:55Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab1006.eqiad.wmnet with reason: deployment 2441804 wikitext text/x-wiki nfmepzfy5v4rf5q95g9eui06rt1r7n6 2441805 2441804 2026-07-28T15:00:57Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet 2441805 wikitext text/x-wiki j5vcurf3ut628w527y3ucw5ibrf1wuc 2441806 2441805 2026-07-28T15:01:26Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab1005.eqiad.wmnet with reason: deployment 2441806 wikitext text/x-wiki o442aj3pstpljwwi4r5ci478s7xc4st 2441807 2441806 2026-07-28T15:01:59Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab2003.codfw.wmnet with reason: deployment 2441807 wikitext text/x-wiki s76hrjb3qyqotuyzrv9kivtux1o02xe 2441808 2441807 2026-07-28T15:05:19Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs1022\.eqiad\.wmnet,dc=eqiad,cluster=wdqs\-main,service=wdqs\-main 2441808 wikitext text/x-wiki 51qpqrplkoioj1ftmt434czbosd2luo 2441809 2441808 2026-07-28T15:05:58Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet 2441809 wikitext text/x-wiki 0e1s7vivgntl22xf846g9z4xl3brzzl 2441810 2441809 2026-07-28T15:06:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1137.eqiad.wmnet 2441810 wikitext text/x-wiki blfsh4kn8v665wdlhfnunlh1enqtqx4 2441811 2441810 2026-07-28T15:06:21Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1137.eqiad.wmnet 2441811 wikitext text/x-wiki edkw583j56goz1vuvt3szv68az8c8bm 2441812 2441811 2026-07-28T15:06:23Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1137.eqiad.wmnet 2441812 wikitext text/x-wiki 2wjlz7psysufzixpjkecmjv58kiqy93 2441813 2441812 2026-07-28T15:07:13Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 13 hosts 2441813 wikitext text/x-wiki t30i0ixwby77tr42zabunbrk9ccrce4 2441814 2441813 2026-07-28T15:07:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling source-only afterwards 2441814 wikitext text/x-wiki 9sa5m0i2erbq31mhjshyhs1eceb35no 2441815 2441814 2026-07-28T15:08:19Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@f8b349f]: deploy phab2003 for T433382 2441815 wikitext text/x-wiki 59drbxp95iksqy5l4edbynygbh5xz7m 2441816 2441815 2026-07-28T15:09:14Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@f8b349f]: deploy phab2003 for T433382 (duration: 00m 55s) 2441816 wikitext text/x-wiki euzz17xeps5a1zhef1wpk5yjrfkcans 2441817 2441816 2026-07-28T15:09:26Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 13 hosts 2441817 wikitext text/x-wiki 37kgel4yh3beskb8h62w14g8f0x1sat 2441818 2441817 2026-07-28T15:10:07Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@f8b349f]: deploy phab1004 for T433382 2441818 wikitext text/x-wiki 82n0rfjbumfziz5vxsozeplojo69nec 2441819 2441818 2026-07-28T15:10:51Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@f8b349f]: deploy phab1004 for T433382 (duration: 00m 43s) 2441819 wikitext text/x-wiki g65mye99a8b14whfk8b5pgxzcildwgz 2441820 2441819 2026-07-28T15:11:01Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1138.eqiad.wmnet 2441820 wikitext text/x-wiki 1y1fo5014ih5x2rhi9v72391qwsfh87 2441821 2441820 2026-07-28T15:11:05Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1138.eqiad.wmnet 2441821 wikitext text/x-wiki 1c8e43rf45qveo9bf5al8qg9g5x1j5i 2441822 2441821 2026-07-28T15:11:35Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003 2441822 wikitext text/x-wiki tbbokyvj43s16xb571z9h8c8hn3hedy 2441823 2441822 2026-07-28T15:11:41Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1138.eqiad.wmnet 2441823 wikitext text/x-wiki toiqwqum20uk7r9sqfd542x9gsiq39n 2441824 2441823 2026-07-28T15:12:39Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet 2441824 wikitext text/x-wiki sgydomdk8opn7f8ll5kjl7xjglp5jzm 2441825 2441824 2026-07-28T15:12:41Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet 2441825 wikitext text/x-wiki qaxj6eyp7gisrgu1ahjrlneca0uyjf2 2441826 2441825 2026-07-28T15:12:46Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet 2441826 wikitext text/x-wiki 0xncunfosgaw85uvzzshy56bmr34fj0 2441827 2441826 2026-07-28T15:17:48Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet 2441827 wikitext text/x-wiki e2qayayuyvj2fcebaxajymk3gkai1bu 2441828 2441827 2026-07-28T15:19:41Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2441828 wikitext text/x-wiki m07s28qawmwy0leaj92uepfyi5jq6t1 2441829 2441828 2026-07-28T15:19:43Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on conf2004.codfw.wmnet with reason: host reimage 2441829 wikitext text/x-wiki fehj30ly9amqe5gumqk64wac20jilno 2441830 2441829 2026-07-28T15:20:00Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2441830 wikitext text/x-wiki p8r6gjkbu5o4blxsuo70z1z3127zvbi 2441831 2441830 2026-07-28T15:21:54Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2013.codfw.wmnet with OS bookworm 2441831 wikitext text/x-wiki 3t5e62wgh1iqnbc5obkutusajktrqmr 2441832 2441831 2026-07-28T15:22:32Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2004.codfw.wmnet with reason: host reimage 2441832 wikitext text/x-wiki 3w9nwwa14pzx929219vg5k8c17o2mxr 2441833 2441832 2026-07-28T15:24:28Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet 2441833 wikitext text/x-wiki qix8sism211v6sb2dw5j6hwsa9ttslw 2441834 2441833 2026-07-28T15:24:29Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet 2441834 wikitext text/x-wiki 2153gankj92qjl33qtw7pww0rvcq7bl 2441835 2441834 2026-07-28T15:24:34Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet 2441835 wikitext text/x-wiki 0rob0m5e2m7tlqh8sqy67efc0sbbfbp 2441836 2441835 2026-07-28T15:25:20Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Maintenance 2441836 wikitext text/x-wiki 9pl0sqyy6qclc98vcflucy7x7obvdm1 2441837 2441836 2026-07-28T15:25:34Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2013 2441837 wikitext text/x-wiki es8e09vzv117o9oqekxmhub9m7q4j2w 2441838 2441837 2026-07-28T15:27:32Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 290 hosts 2441838 wikitext text/x-wiki t02zgfh972wgi5molu959hnjersvups 2441839 2441838 2026-07-28T15:29:37Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet 2441839 wikitext text/x-wiki dqa9sbkmazr6e4wm99m0c5h314ggpc5 2441841 2441839 2026-07-28T15:31:30Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2441841 wikitext text/x-wiki s4g618zm6g9scn4qk5j1n36br1m96ib 2441842 2441841 2026-07-28T15:31:50Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2441842 wikitext text/x-wiki 1n3k28u3ufxp36md8z5kbarfnx3nnv1 2441843 2441842 2026-07-28T15:34:44Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet 2441843 wikitext text/x-wiki bttmcj04g7suzkt8we0l9rcocjliu3c 2441844 2441843 2026-07-28T15:34:45Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet 2441844 wikitext text/x-wiki ahckui8alzewscj9q3tz769o98469fm 2441845 2441844 2026-07-28T15:34:50Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet 2441845 wikitext text/x-wiki nxkg85l4ad6zokstyqtq840k7vngm6d 2441846 2441845 2026-07-28T15:35:26Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2441846 wikitext text/x-wiki 5ctwzv4fbz9cyh0u653wui55qexxuxe 2441847 2441846 2026-07-28T15:37:47Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2441847 wikitext text/x-wiki hu1nnjjesqrnryi2clxlw1hu9qr8lyc 2441848 2441847 2026-07-28T15:38:15Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2173: Maintenance 2441848 wikitext text/x-wiki m61vldwi9t2ru3k096ucsw0mwamp872 2441849 2441848 2026-07-28T15:38:38Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance 2441849 wikitext text/x-wiki tbm8kqf1m4rqb4j8vzr5wnpjiet2uc5 2441850 2441849 2026-07-28T15:38:46Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2174 (T431660)', diff saved to https://phabricator.wikimedia.org/P95413 and previous config saved to /var/cache/conftool/dbconfig/20260728-153844-cwilliams.json 2441850 wikitext text/x-wiki 8ue6ynpmripu8lp6te8aj7j2kasw7kp 2441851 2441850 2026-07-28T15:39:39Z Stashbot 7414 robh@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on arclamp2001.codfw.wmnet with reason: ram upgrade 2441851 wikitext text/x-wiki skjdf02l8ds7ispu2ulzmkua8mayxv0 2441852 2441851 2026-07-28T15:39:54Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet 2441852 wikitext text/x-wiki 8zo13b0vxfykyci361z9t5ao2r377qo 2441853 2441852 2026-07-28T15:41:46Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1138.eqiad.wmnet with OS trixie 2441853 wikitext text/x-wiki loi3s2p0uotublj0dp38ja9tdk59amy 2441854 2441853 2026-07-28T15:42:16Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1138 2441854 wikitext text/x-wiki spqsoauv61d1m365r81gz312r54uowj 2441855 2441854 2026-07-28T15:44:34Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet 2441855 wikitext text/x-wiki 0utds0jpxjeubm8pp6zi29oebtuzai4 2441856 2441855 2026-07-28T15:44:36Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet 2441856 wikitext text/x-wiki 4z88ethxn4y338z9s8ba9m65emg8sy6 2441857 2441856 2026-07-28T15:44:41Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet 2441857 wikitext text/x-wiki al48mjz36cp2syfnq1tre7byzwk2t0w 2441858 2441857 2026-07-28T15:45:23Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2174: Maintenance 2441858 wikitext text/x-wiki i5oj7u0qu0wam7hhra40jat8g8svqt4 2441859 2441858 2026-07-28T15:46:36Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2248: Maintenance 2441859 wikitext text/x-wiki qrlkb33lnj33f8s5tz6xe3ssa4ze1ek 2441860 2441859 2026-07-28T15:48:17Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2441860 wikitext text/x-wiki 0580oz7qyaamuprdziklcvvrvo3k35s 2441861 2441860 2026-07-28T15:48:22Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2441861 wikitext text/x-wiki setpbqv8e0ivozsjqdmw037r30xlwd0 2441862 2441861 2026-07-28T15:49:44Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet 2441862 wikitext text/x-wiki pjd2zr6pb5it1l5msyp5ydgyqx49avr 2441863 2441862 2026-07-28T15:50:56Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2441863 wikitext text/x-wiki lcy7ngjnzcdl88o61gp1ucwloynga3f 2441864 2441863 2026-07-28T15:54:04Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2441864 wikitext text/x-wiki fis8pqskju9odqagji5gzi6jkexlycg 2441865 2441864 2026-07-28T15:54:10Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet 2441865 wikitext text/x-wiki hbkr9k1g197q3vnzpa97khiodlbe7br 2441866 2441865 2026-07-28T15:54:12Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet 2441866 wikitext text/x-wiki pv1668p2xn09haaj9hj6ffxzlngf2d1 2441867 2441866 2026-07-28T15:54:16Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet 2441867 wikitext text/x-wiki eretswbwk9hcuv2njgyl0sro3mrtacf 2441868 2441867 2026-07-28T15:54:53Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2441868 wikitext text/x-wiki gqyz5a6rxnblh167cmh6z3glykfstdw 2441869 2441868 2026-07-28T15:55:52Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2441869 wikitext text/x-wiki jgrlc11tlz3ovsy047lvvdmqawxcs4a 2441870 2441869 2026-07-28T15:56:28Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2441870 wikitext text/x-wiki aecnknnq22n31tukg1tr5ga6q9qwd51 2441871 2441870 2026-07-28T15:59:19Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet 2441871 wikitext text/x-wiki 6lwccozo45ui2x4jtawz3vb5ii21yo3 2441872 2441871 2026-07-28T16:00:06Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2004.codfw.wmnet with OS bookworm 2441872 wikitext text/x-wiki 2cp0daka3eq79oyb4gun4oy8rkqaywx 2441873 2441872 2026-07-28T16:03:46Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet 2441873 wikitext text/x-wiki hjoewnoh7mae26k5wzqkxvaiwcs6xz6 2441874 2441873 2026-07-28T16:03:48Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet 2441874 wikitext text/x-wiki 920bqdpa45wiyxrv63lgfkszn297bid 2441875 2441874 2026-07-28T16:03:52Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1012.eqiad.wmnet 2441875 wikitext text/x-wiki kbdfbwsebxoodl6lm2h5gaohfc76wgw 2441876 2441875 2026-07-28T16:13:56Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1012.eqiad.wmnet 2441876 wikitext text/x-wiki ttqg03kneapxdgc5jzj5rwgztvxyrm2 2441877 2441876 2026-07-28T16:20:41Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1012.eqiad.wmnet 2441877 wikitext text/x-wiki rnx9r7snrrva5cza0ap8jikqk997enl 2441878 2441877 2026-07-28T16:20:43Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1012.eqiad.wmnet 2441878 wikitext text/x-wiki bh3bahchfwaijstapzp32ndvzsclcod 2441879 2441878 2026-07-28T16:20:48Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1013.eqiad.wmnet 2441879 wikitext text/x-wiki 5kjpnqq1e5gdy0d8m4trtc4v1y36t9b 2441880 2441879 2026-07-28T16:25:53Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet 2441880 wikitext text/x-wiki gz9mkuh76ujs0miw145n83nwk8bhz0m 2441881 2441880 2026-07-28T16:28:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling source-only afterwards 2441881 wikitext text/x-wiki rpwp528qgwx9cnf3llhfo82uhw7bv8h 2441882 2441881 2026-07-28T16:32:06Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2174: Maintenance 2441882 wikitext text/x-wiki e182qoelw63w0w3t0i2xjgcbrjfnt5y 2441883 2441882 2026-07-28T16:32:12Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet 2441883 wikitext text/x-wiki br8fgip33l5wb3kyhmng9jcc0xxnre0 2441884 2441883 2026-07-28T16:32:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet 2441884 wikitext text/x-wiki oglhhphpqgq9uf8uz7yb0niy2ximv54 2441885 2441884 2026-07-28T16:32:18Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet 2441885 wikitext text/x-wiki 0uw5zy0er39ucuot2ks8ziwthbnhk4g 2441886 2441885 2026-07-28T16:32:28Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance 2441886 wikitext text/x-wiki 3y9v7k5s95g9m7o3u1ik20jdpzt0t4h 2441887 2441886 2026-07-28T16:32:37Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2176 (T431660)', diff saved to https://phabricator.wikimedia.org/P95420 and previous config saved to /var/cache/conftool/dbconfig/20260728-163235-cwilliams.json 2441887 wikitext text/x-wiki pivbu0cuat3s5c6wj08lcihhvo2ikev 2441889 2441887 2026-07-28T16:37:24Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet 2441889 wikitext text/x-wiki jcrzjl3ooas4ta6urm2za7d0obh7s5e 2441890 2441889 2026-07-28T16:37:40Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1138 - jiji@cumin1003" 2441890 wikitext text/x-wiki o5w1fud7zba1sdbngnhmiobaz3xlb8h 2441891 2441890 2026-07-28T16:37:45Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1138 - jiji@cumin1003" 2441891 wikitext text/x-wiki si01v18tcb7za6i0660jo7bwmksa4l9 2441892 2441891 2026-07-28T16:37:47Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441892 wikitext text/x-wiki gi9mzlark974tiyg1tn67o3k59esk4v 2441893 2441892 2026-07-28T16:37:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1138.eqiad.wmnet 193.32.64.10.in-addr.arpa 3.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441893 wikitext text/x-wiki 0sq34q2qf7elv25jl9rk4elsgmmq9cl 2441894 2441893 2026-07-28T16:37:50Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1138.eqiad.wmnet 193.32.64.10.in-addr.arpa 3.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2441894 wikitext text/x-wiki 3f2sdsb14pss89mgnpo8eg20nszwo9u 2441895 2441894 2026-07-28T16:37:52Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1138 2441895 wikitext text/x-wiki 71eaqus5jg6pe1z9d1lk98nrhq0k8jc 2441896 2441895 2026-07-28T16:38:19Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1138 2441896 wikitext text/x-wiki 2j613idpftnefkhvl848ihfw52y0xgq 2441897 2441896 2026-07-28T16:38:21Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1138 2441897 wikitext text/x-wiki nnvdu72naozzmpuk12naj7ar1e2uvb2 2441898 2441897 2026-07-28T16:38:57Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2176: Maintenance 2441898 wikitext text/x-wiki r6l2rzppcfpzwipntyb21bk3ippm5wz 2441899 2441898 2026-07-28T16:42:06Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 290 hosts 2441899 wikitext text/x-wiki qu5aeasb00tg85ckw9ns6ssoe3jyt9b 2441900 2441899 2026-07-28T16:43:11Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns3003.wikimedia.org [reason: depooling for reimage to trixie] 2441900 wikitext text/x-wiki 3fe1iej8bzoq89n5kg6poydya4p5jdt 2441901 2441900 2026-07-28T16:43:35Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet 2441901 wikitext text/x-wiki mffpmzprmmc05rxotbp20keyqvn3z1f 2441902 2441901 2026-07-28T16:43:37Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet 2441902 wikitext text/x-wiki 6k899cab6z5egev7i2qr8rrdosepvzs 2441903 2441902 2026-07-28T16:43:42Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet 2441903 wikitext text/x-wiki kse6s26vvw0bdocys31e9tduezqa4f4 2441904 2441903 2026-07-28T16:43:58Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host dns3003.wikimedia.org with OS trixie 2441904 wikitext text/x-wiki fjwj6hgei0ub0yqvft8ux99fecs54tm 2441907 2441904 2026-07-28T16:53:49Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet 2441907 wikitext text/x-wiki q7y88cdf4g0972j0s6o5vrumzmrlud9 2441908 2441907 2026-07-28T16:54:28Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1138.eqiad.wmnet with reason: host reimage 2441908 wikitext text/x-wiki akr9iqyxpbkil2twb29rmo3kecnuoss 2441909 2441908 2026-07-28T16:55:16Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003 2441909 wikitext text/x-wiki drunswdpnyud3sqn2dchb8mg022e46a 2441910 2441909 2026-07-28T16:59:03Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1138.eqiad.wmnet with reason: host reimage 2441910 wikitext text/x-wiki 8mwn8hegx6jwl8s1nutm3uv7d0vx7d0 2441911 2441910 2026-07-28T17:00:10Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet 2441911 wikitext text/x-wiki 27gd4b0pdx9e1n8b3vf34p46nyyfuwr 2441912 2441911 2026-07-28T17:00:12Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet 2441912 wikitext text/x-wiki 9n98u8p9j34kodzbkefeortzqsnpcxv 2441913 2441912 2026-07-28T17:00:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad 2441913 wikitext text/x-wiki ietgff7emiiz8x48l8pe4ugd14q4gpy 2441916 2441913 2026-07-28T17:07:43Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns3003.wikimedia.org with reason: host reimage 2441916 wikitext text/x-wiki iw8um76i9kdi4r26wfsvt2afq6f1kph 2441917 2441916 2026-07-28T17:13:41Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns3003.wikimedia.org with reason: host reimage 2441917 wikitext text/x-wiki 914xcspw2ij5ls4qmfajwq2a58nlgyc 2441918 2441917 2026-07-28T17:15:20Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2441918 wikitext text/x-wiki 82pkcxkj1gt0j04tctwaubvqks6xdqw 2441919 2441918 2026-07-28T17:18:24Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441919 wikitext text/x-wiki qh5oqwaweo1hw43om7mg8tjtsvdhf5n 2441920 2441919 2026-07-28T17:18:38Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1005 2441920 wikitext text/x-wiki 9uptywnzue97jr3acm3e51jt76253gd 2441921 2441920 2026-07-28T17:18:57Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1005 2441921 wikitext text/x-wiki 52ofo27yae8a5eoj5ln1t096eofb8cn 2441922 2441921 2026-07-28T17:19:14Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1138.eqiad.wmnet with OS trixie 2441922 wikitext text/x-wiki bb27r7zobbazxw5u32mc87iwzki7btl 2441925 2441922 2026-07-28T17:25:41Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: Maintenance 2441925 wikitext text/x-wiki jom3wutnku85x4i6nrkdd69xkdvqvtv 2441926 2441925 2026-07-28T17:26:02Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance 2441926 wikitext text/x-wiki cme0vroiad4kvf0ez8kuymiqc0sv483 2441927 2441926 2026-07-28T17:26:11Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2188 (T431660)', diff saved to https://phabricator.wikimedia.org/P95425 and previous config saved to /var/cache/conftool/dbconfig/20260728-172609-cwilliams.json 2441927 wikitext text/x-wiki ghnz9rgl0hhnskl039wlhrlje6ieexz 2441928 2441927 2026-07-28T17:29:22Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2441928 wikitext text/x-wiki aqkd4hhlk1ivn6qedgpmtcxpw09a4dl 2441929 2441928 2026-07-28T17:29:47Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2441929 wikitext text/x-wiki 7xpts5ss4yohh2vmcaag55vc3f55d2m 2441931 2441929 2026-07-28T17:32:46Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2188: Maintenance 2441931 wikitext text/x-wiki e26f4uctnz0v7yzy02o7nq9g83jiht1 2441932 2441931 2026-07-28T17:34:05Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1005] - vriley@cumin1003" 2441932 wikitext text/x-wiki cmaiikuutf2uu1iun4dtta9iprywt6a 2441933 2441932 2026-07-28T17:34:09Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1005] - vriley@cumin1003" 2441933 wikitext text/x-wiki sp4v6ccs3od94uoncz65dphrkfjtpdt 2441934 2441933 2026-07-28T17:34:11Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441934 wikitext text/x-wiki geqtu4kzs5b7be6twk6okk7t8uur2gt 2441935 2441934 2026-07-28T17:34:19Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1005 2441935 wikitext text/x-wiki 9d2osy5whdyjieiitv6udw5klns1vr8 2441936 2441935 2026-07-28T17:34:26Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2441936 wikitext text/x-wiki obadhbjtnrx0fsa5svw729prk86fswz 2441938 2441936 2026-07-28T17:34:50Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1005 2441938 wikitext text/x-wiki 4wvadx7sze50tw1wcigvqv7lzitowxu 2441939 2441938 2026-07-28T17:34:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2441939 wikitext text/x-wiki 95ugmcc44zzuemihx727vnbs410ms62 2441941 2441939 2026-07-28T17:35:45Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2441941 wikitext text/x-wiki 06gsycktidbzblr67nw6t56mlp77elt 2441943 2441941 2026-07-28T17:35:57Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: cluster=urldownloader,service=squid 2441943 wikitext text/x-wiki 131xsifihnxvkuyjkc1ope7d7dycd0g 2441944 2441943 2026-07-28T17:36:13Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441944 wikitext text/x-wiki a93b7yij6zlaro2xaiorso8yzbiv8qv 2441945 2441944 2026-07-28T17:36:14Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=1; selector: cluster=urldownloader,service=squid 2441945 wikitext text/x-wiki h7gzajicp5bpmonkkdt9lzlksseblpf 2441949 2441945 2026-07-28T17:39:49Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2441949 wikitext text/x-wiki q262g7zjjp4lw60b4q9yq3lph3dy4dc 2441951 2441949 2026-07-28T17:41:59Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2441951 wikitext text/x-wiki 45wik6zls80549vl2qahid3h5ovw0lf 2441954 2441951 2026-07-28T17:45:28Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441954 wikitext text/x-wiki pmchbkiib162pefgmm0gjnlq46y4sae 2441955 2441954 2026-07-28T17:45:58Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie 2441955 wikitext text/x-wiki 1j0yi4pm75dwszo93nzwr41bc40mdpe 2441961 2441955 2026-07-28T17:56:51Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003 2441961 wikitext text/x-wiki 6tm3xu7d89g0bsioo6dqibqf81idbmj 2441962 2441961 2026-07-28T18:03:41Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage 2441962 wikitext text/x-wiki jfr02y843rhrgtffwuwraqf08p9cywp 2441964 2441962 2026-07-28T18:05:28Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns3003.wikimedia.org with OS trixie 2441964 wikitext text/x-wiki ht2vu48yetsmmar13ntjr8dwp3ygdm2 2441965 2441964 2026-07-28T18:08:10Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage 2441965 wikitext text/x-wiki rcc8dbzpuyfw42zrd2thb999cpkdzdl 2441966 2441965 2026-07-28T18:10:04Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2441966 wikitext text/x-wiki 4y9nu0fevd6u25riv5eg827gk2020w8 2441967 2441966 2026-07-28T18:15:46Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding ms-be2097-8 to codfw - jhancock@cumin2002" 2441967 wikitext text/x-wiki 267e6wzd7hc0xehfcjanjnhokkgw3uz 2441968 2441967 2026-07-28T18:15:52Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding ms-be2097-8 to codfw - jhancock@cumin2002" 2441968 wikitext text/x-wiki onav5v6ultn4o854fmqsp5o9pxj4ldf 2441969 2441968 2026-07-28T18:15:53Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2441969 wikitext text/x-wiki s66z3zbyne61pje5smsowtj8gr0zud8 2441971 2441969 2026-07-28T18:16:41Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2097 2441971 wikitext text/x-wiki dnwbwwzawvf0u7w8qp5ndwrfe1pq6kr 2441972 2441971 2026-07-28T18:17:10Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2097 2441972 wikitext text/x-wiki 7hkqe7e1vj6q7u3azuw0yemxzax3zb7 2441973 2441972 2026-07-28T18:17:14Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2098 2441973 wikitext text/x-wiki kf31r2wu8zpkhyrsqsncm9eilozrzrk 2441974 2441973 2026-07-28T18:17:59Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2098 2441974 wikitext text/x-wiki s87gjshd5nkt0ycc98f7rsd8huclzo8 2441975 2441974 2026-07-28T18:18:46Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2097.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441975 wikitext text/x-wiki ae42auwjg760yskf8iit9izpig7c0i2 2441976 2441975 2026-07-28T18:19:27Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: Maintenance 2441976 wikitext text/x-wiki s3vpi1dw0td62xr2pi9w6xxmuy6qvwo 2441977 2441976 2026-07-28T18:19:52Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2203.codfw.wmnet with reason: Maintenance 2441977 wikitext text/x-wiki 1haztrol0mvqf3leyrx1ehr5r39rbnh 2441978 2441977 2026-07-28T18:20:00Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2203 (T431660)', diff saved to https://phabricator.wikimedia.org/P95430 and previous config saved to /var/cache/conftool/dbconfig/20260728-181958-cwilliams.json 2441978 wikitext text/x-wiki 7dw2edagehgfi1zq0tgvqxeh9eahnl9 2441979 2441978 2026-07-28T18:26:29Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2203: Maintenance 2441979 wikitext text/x-wiki girrxzk6v7ng669znd1aadymsa3mr7m 2441980 2441979 2026-07-28T18:27:11Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns3003.wikimedia.org,service=authdns-update [reason: pool authdns-update after reimaging] 2441980 wikitext text/x-wiki qbye0nw40fg64zzccaqbjoayh9z1drv 2441981 2441980 2026-07-28T18:27:23Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2441981 wikitext text/x-wiki n2otvwmy7relg4ya7xz4g9uecbtxeyt 2441982 2441981 2026-07-28T18:29:25Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2441982 wikitext text/x-wiki smhh99nau9swbhcm7vspx5scdaspywo 2441983 2441982 2026-07-28T18:30:09Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns3003.wikimedia.org [reason: pool for all services after reimaging] 2441983 wikitext text/x-wiki 9kwtm04z42ll3l6oxjguje5qd8o6zzb 2441984 2441983 2026-07-28T18:32:24Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2097.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441984 wikitext text/x-wiki 2bt8trqxcehrvwc1ppm9i58t49yaifo 2441985 2441984 2026-07-28T18:32:50Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441985 wikitext text/x-wiki pr45r9c3kw2lvw5l0aqnn5ltn095fei 2441986 2441985 2026-07-28T18:33:32Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.275.0" for 3 host(s) 2441986 wikitext text/x-wiki 8kp6y7xz6p12cte59vwaszgkasknu31 2441987 2441986 2026-07-28T18:35:28Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.275.0" completed for 3 hosts 2441987 wikitext text/x-wiki mp1e4jvztpbymqh3tx8hdyliq6wnvcv 2441988 2441987 2026-07-28T18:36:59Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003 2441988 wikitext text/x-wiki 9eix479gpmqnsmd5zql2yffu7y7u5ox 2441989 2441988 2026-07-28T18:40:10Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441989 wikitext text/x-wiki jmwvrqtzac3ztwn2xc9unevjcj03j6h 2441990 2441989 2026-07-28T18:43:42Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441990 wikitext text/x-wiki nthtsrle8gpqbnio1tb2rhjzmsydslc 2441991 2441990 2026-07-28T18:45:35Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2441991 wikitext text/x-wiki 0jqftyihujun5hfdvhducr82ra9aflu 2441992 2441991 2026-07-28T18:45:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2441992 wikitext text/x-wiki 4tnuhytamqmidlgivvra7f2fwn3ry42 2441993 2441992 2026-07-28T18:46:33Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.13 refs T430832 2441993 wikitext text/x-wiki grbz9qtmppg6720iqqtjqn682fdsim9 2441994 2441993 2026-07-28T19:06:19Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2441994 wikitext text/x-wiki s7foskkyrgim2r5gpdl9grwzj4kxkvd 2441995 2441994 2026-07-28T19:06:56Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2441995 wikitext text/x-wiki g57q5ge71zk9pans5gejca1msb8bjie 2441996 2441995 2026-07-28T19:06:58Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie 2441996 wikitext text/x-wiki j4m3f4ojxm0p0gsvrv2gmr3xf5z9aqy 2441997 2441996 2026-07-28T19:13:14Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2203: Maintenance 2441997 wikitext text/x-wiki ou2zz10czsfvkilut0amxk45xisfpgj 2441998 2441997 2026-07-28T19:13:36Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance 2441998 wikitext text/x-wiki dz4xpg9k9sn8vtfolv2hbns09wbjc72 2441999 2441998 2026-07-28T19:13:45Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db2216 (T431660)', diff saved to https://phabricator.wikimedia.org/P95435 and previous config saved to /var/cache/conftool/dbconfig/20260728-191343-cwilliams.json 2441999 wikitext text/x-wiki 4qhy72ysh3jvgwg056phy2d0ute525k 2442000 2441999 2026-07-28T19:20:18Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db2216: Maintenance 2442000 wikitext text/x-wiki 29na9icv12kvk2kwhphmyb0vv1v0b5b 2442001 2442000 2026-07-28T19:21:21Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2442001 wikitext text/x-wiki 62vswywxgiez6jdv8r4d7fu76dnbef7 2442002 2442001 2026-07-28T19:21:36Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2013 - bking@cumin2003" 2442002 wikitext text/x-wiki ar2jv1vnx6yoarta5z4ekh2cio8xo08 2442003 2442002 2026-07-28T19:21:40Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2013 - bking@cumin2003" 2442003 wikitext text/x-wiki 3y7xpl2ukrbappbmmajj9zciazphccv 2442004 2442003 2026-07-28T19:21:42Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442004 wikitext text/x-wiki hse248jdffjpz84n1mtkkjfh9tqkcw5 2442005 2442004 2026-07-28T19:21:43Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2013.codfw.wmnet 84.0.192.10.in-addr.arpa 4.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2442005 wikitext text/x-wiki jy3no1st0bo3jaunaryk64vs0eeoiqy 2442006 2442005 2026-07-28T19:21:46Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2013.codfw.wmnet 84.0.192.10.in-addr.arpa 4.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2442006 wikitext text/x-wiki 3bc5mq60kqz7g77bo5g0eefs4umcmog 2442007 2442006 2026-07-28T19:21:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2013 2442007 wikitext text/x-wiki epusmndwt8zjqddwr3j2x1a1lm7dohn 2442008 2442007 2026-07-28T19:21:56Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2013 2442008 wikitext text/x-wiki qxnfmlikyb2fyptkm4oxsjnzbpq9e9j 2442009 2442008 2026-07-28T19:21:57Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2013 2442009 wikitext text/x-wiki kx5fen29pqpvya2ia7bf67f5edeuw6o 2442010 2442009 2026-07-28T19:25:57Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1006] - vriley@cumin1003" 2442010 wikitext text/x-wiki p5f1cx7ku45or5xvrms5mc5bwnm3sox 2442011 2442010 2026-07-28T19:26:02Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1006] - vriley@cumin1003" 2442011 wikitext text/x-wiki r6zokyso5frua8sddd0pw2zq4pyn7o5 2442012 2442011 2026-07-28T19:26:04Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2442012 wikitext text/x-wiki jtslhjp8lu3jqxju8enoax5vu53xlao 2442013 2442012 2026-07-28T19:26:11Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1006 2442013 wikitext text/x-wiki 4f3rfslm3d24qr6hhpphhinzlnf62d7 2442014 2442013 2026-07-28T19:26:29Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2012\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main 2442014 wikitext text/x-wiki sd00xzomic7vfnb8i85gx52lvvb1wtj 2442015 2442014 2026-07-28T19:27:37Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1006 2442015 wikitext text/x-wiki ps23un5mbqno22wwchviveei0j04ljv 2442016 2442015 2026-07-28T19:28:01Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442016 wikitext text/x-wiki j5vrqr5w0ugttack0tj2aymvtj6b6m8 2442017 2442016 2026-07-28T19:29:55Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2014.codfw.wmnet with OS bookworm 2442017 wikitext text/x-wiki kxl0gw3752lmrpvs5oh22d0qi9qp4cm 2442018 2442017 2026-07-28T19:33:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2014 2442018 wikitext text/x-wiki lib852ko6502iahcw6k576m7swxfvs8 2442019 2442018 2026-07-28T19:33:18Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2014 2442019 wikitext text/x-wiki r1bkwi23t47te1m2a3cahacut8yuh4x 2442020 2442019 2026-07-28T19:38:54Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442020 wikitext text/x-wiki e4ujs937rmv6lom94k84f82mea1pewl 2442021 2442020 2026-07-28T19:39:20Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003 2442021 wikitext text/x-wiki 9xghms1o20nm4i40304660crjnkk15w 2442022 2442021 2026-07-28T19:39:25Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442022 wikitext text/x-wiki gf14gbjvb8waeqqm6uofmcatngih3kw 2442023 2442022 2026-07-28T19:39:36Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442023 wikitext text/x-wiki pptzjd85xfdl8rmbbdrsoq2npd7ic2e 2442024 2442023 2026-07-28T19:41:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2013.codfw.wmnet with reason: host reimage 2442024 wikitext text/x-wiki 9tr0s7m8oa5279o061ckl38o3vn5rhu 2442025 2442024 2026-07-28T19:41:46Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442025 wikitext text/x-wiki a322ce3j9zqrficnqkpfeh9rza6gcx0 2442026 2442025 2026-07-28T19:42:57Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2442026 wikitext text/x-wiki gz1ixysix63mkd8xkua274u4at8xfmd 2442027 2442026 2026-07-28T19:45:00Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2013.codfw.wmnet with reason: host reimage 2442027 wikitext text/x-wiki bibv7died9su4wodddr9cqxx016fw5t 2442028 2442027 2026-07-28T19:48:16Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442028 wikitext text/x-wiki pqnsmxp5v55a7bzz9plhwisu37mzt8s 2442029 2442028 2026-07-28T19:48:35Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442029 wikitext text/x-wiki qrpcyqv1q8qvlglmlz5eim0uhfa4hx2 2442030 2442029 2026-07-28T19:49:38Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442030 wikitext text/x-wiki fdl84k5dx6c8xb36afjm72s2e3u0519 2442031 2442030 2026-07-28T19:52:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2014.codfw.wmnet with reason: host reimage 2442031 wikitext text/x-wiki l1nxvzsxwvynnamjuyrv0eq6ohmeh46 2442032 2442031 2026-07-28T19:52:51Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442032 wikitext text/x-wiki 2m33a56ty4l2fnrk02mrwkuwoyuakez 2442033 2442032 2026-07-28T19:53:31Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442033 wikitext text/x-wiki 6mbr0wybw56gaml87w7hr0nh9wqfu5o 2442034 2442033 2026-07-28T19:54:06Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442034 wikitext text/x-wiki 5wxpdg29be236klei87bwf2prrr0y0p 2442035 2442034 2026-07-28T19:54:16Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442035 wikitext text/x-wiki n1pa1ilsu6gugwdkvvz9d2nqv4b955q 2442036 2442035 2026-07-28T19:57:50Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2442036 wikitext text/x-wiki 5nbfjag6f99jhj2dpgz6j4u2ylxvliu 2442037 2442036 2026-07-28T19:59:29Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2014.codfw.wmnet with reason: host reimage 2442037 wikitext text/x-wiki 21aatuj9yuexodq33ilxgx80x8ft84z 2442039 2442037 2026-07-28T20:07:02Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2216: Maintenance 2442039 wikitext text/x-wiki n9rgobxcucqnjyuqaeampolkf42wcjm 2442040 2442039 2026-07-28T20:07:20Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1318718|bolwiki: set correct version of logo (T429951)]], [[gerrit:1318756|Remove icon beside reporting link on desktop view (T433303)]], [[gerrit:1318754|Remove icon beside reporting link on desktop view (T433303)]] 2442040 wikitext text/x-wiki 7f26i4rvez7egv48bfuyrlo4gpuh9uf 2442042 2442040 2026-07-28T20:09:22Z Stashbot 7414 arlolra@deploy1003: anzx, arlolra: Backport for [[gerrit:1318718|bolwiki: set correct version of logo (T429951)]], [[gerrit:1318756|Remove icon beside reporting link on desktop view (T433303)]], [[gerrit:1318754|Remove icon beside reporting link on desktop view (T433303)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442042 wikitext text/x-wiki 216rghzu7yoopucp0gp5jcbwxbi5k9p 2442043 2442042 2026-07-28T20:10:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2013.codfw.wmnet with OS bookworm 2442043 wikitext text/x-wiki 4mg88kydcudcteu40nmc74ks5tx9dnf 2442044 2442043 2026-07-28T20:11:22Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442044 wikitext text/x-wiki 48gh1o09g4o4ekc2cbhp8u22wnmjl9u 2442045 2442044 2026-07-28T20:11:27Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442045 wikitext text/x-wiki jap6o579yjkpimr99c9u3knzvdlwra2 2442046 2442045 2026-07-28T20:11:28Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442046 wikitext text/x-wiki pbt25or7df9us7b5p669xiojf9270hn 2442047 2442046 2026-07-28T20:11:35Z Stashbot 7414 arlolra@deploy1003: anzx, arlolra: Continuing with deployment 2442047 wikitext text/x-wiki qcqk4ar8kg847mwrsiq74w8yhplrpj3 2442048 2442047 2026-07-28T20:11:43Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442048 wikitext text/x-wiki e0q6q5owd6iqhgyo5u7z5rnptkzxzwg 2442049 2442048 2026-07-28T20:11:51Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442049 wikitext text/x-wiki m1zaan45qjbnmqs5509c45plmjbhnva 2442050 2442049 2026-07-28T20:12:03Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442050 wikitext text/x-wiki 6b7biq4p1yqnpbhsx42n6iocvjixc0a 2442051 2442050 2026-07-28T20:15:46Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318718|bolwiki: set correct version of logo (T429951)]], [[gerrit:1318756|Remove icon beside reporting link on desktop view (T433303)]], [[gerrit:1318754|Remove icon beside reporting link on desktop view (T433303)]] (duration: 08m 26s) 2442051 wikitext text/x-wiki 797lwaxsl1jgzpchb4r255wd8rfinso 2442052 2442051 2026-07-28T20:17:24Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442052 wikitext text/x-wiki iu81yyfsn0j6yx7tip565h6h468ctoz 2442053 2442052 2026-07-28T20:17:32Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442053 wikitext text/x-wiki ind7z4waqjyfzmu4p5rfkoih62ihvoj 2442054 2442053 2026-07-28T20:19:24Z Stashbot 7414 swfrench-wmf: switched etcd-mirror replication from conf2005 to conf2004 - T428495 2442054 wikitext text/x-wiki d7khw94ir1kpif9ssr1q97vbkr0cgk2 2442055 2442054 2026-07-28T20:19:34Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442055 wikitext text/x-wiki mbj5jgay1ivddvktgd2u12ivwz85fgs 2442056 2442055 2026-07-28T20:20:37Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2442056 wikitext text/x-wiki 39kyp67w6n76gf1xoksjb7wl8qfuy6k 2442057 2442056 2026-07-28T20:21:49Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442057 wikitext text/x-wiki qi56u3hxiuskm1ae5pkets1jg9jqst1 2442058 2442057 2026-07-28T20:22:00Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2442058 wikitext text/x-wiki 3n7lw05ct8olw6fm7rf9t20iikh1nly 2442059 2442058 2026-07-28T20:26:54Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2014.codfw.wmnet with OS bookworm 2442059 wikitext text/x-wiki cpienny4hrhky3yydzh7gw7kgv9z7lt 2442060 2442059 2026-07-28T20:27:36Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1318753|Support older config values in $wgJsonConfigModels (T433008)]] 2442060 wikitext text/x-wiki olbyomkw19e1s96b87mhpzlb1mper51 2442061 2442060 2026-07-28T20:29:35Z Stashbot 7414 arlolra@deploy1003: arlolra: Backport for [[gerrit:1318753|Support older config values in $wgJsonConfigModels (T433008)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442061 wikitext text/x-wiki qfwsrywc7jnwthua7znddubawlkfumj 2442062 2442061 2026-07-28T20:30:24Z Stashbot 7414 arlolra@deploy1003: arlolra: Continuing with deployment 2442062 wikitext text/x-wiki 6jxq02833s3gm98jnoi2z57ifeo21vz 2442063 2442062 2026-07-28T20:30:30Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442063 wikitext text/x-wiki s7tzm5hkmr5uxo1hipuz2wq6i1yhkeg 2442064 2442063 2026-07-28T20:30:40Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442064 wikitext text/x-wiki s2ll673dvnzigvx5er0vje0hfrrrhbd 2442065 2442064 2026-07-28T20:31:39Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2442065 wikitext text/x-wiki r8mwsnk6k0k8unuokxp3i7mxdvk6tc5 2442066 2442065 2026-07-28T20:31:48Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2442066 wikitext text/x-wiki c12mhbryhubetjvosmyzmnpxdn7cstg 2442067 2442066 2026-07-28T20:32:42Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2442067 wikitext text/x-wiki qo63uhmpcxijfzctgx2d3r5a96rwthf 2442068 2442067 2026-07-28T20:32:52Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2442068 wikitext text/x-wiki db0atq7ojet7uqtf237dv9kukndgp1l 2442069 2442068 2026-07-28T20:33:45Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2442069 wikitext text/x-wiki 1jk1k3sgnxh89p9jvoweq5cdkknvbte 2442070 2442069 2026-07-28T20:33:57Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2442070 wikitext text/x-wiki 7lrbp1m5plm29qar5tdysvrl5ux57rv 2442071 2442070 2026-07-28T20:34:10Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2442071 wikitext text/x-wiki 4fwt6bnucnlti7ysv2vsin4req03wes 2442072 2442071 2026-07-28T20:34:23Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2442072 wikitext text/x-wiki 46qe5bh4ot83ofv6j4nx2vryni4fkmo 2442073 2442072 2026-07-28T20:34:31Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318753|Support older config values in $wgJsonConfigModels (T433008)]] (duration: 06m 54s) 2442073 wikitext text/x-wiki ls4wej3rcxosznaq1j6rwt0kq5lj7jc 2442074 2442073 2026-07-28T20:43:35Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie 2442074 wikitext text/x-wiki 4yy1t66yn9rp8t4or8caocltwdd3t4x 2442075 2442074 2026-07-28T20:49:13Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2442075 wikitext text/x-wiki hmhm10jy06o6jgq8lpv6bh3w8u72ejl 2442076 2442075 2026-07-28T20:50:26Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1318746|Bump wikimedia/parsoid to 0.24.0-a17 (T19006 T234299 T360814 T385317 T432477 T433018)]], [[gerrit:1318747|Bump wikimedia/parsoid to 0.24.0-a17 (T433018)]] 2442076 wikitext text/x-wiki fctp49mlyyncltwhyjdojcbej6xnwm7 2442077 2442076 2026-07-28T20:51:16Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2442077 wikitext text/x-wiki 7dnxyotbgbjn6u763oaki1dh3m61imc 2442078 2442077 2026-07-28T20:52:21Z Stashbot 7414 arlolra@deploy1003: arlolra: Backport for [[gerrit:1318746|Bump wikimedia/parsoid to 0.24.0-a17 (T19006 T234299 T360814 T385317 T432477 T433018)]], [[gerrit:1318747|Bump wikimedia/parsoid to 0.24.0-a17 (T433018)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442078 wikitext text/x-wiki ak4qtfkortl8nv0g9t9bfv7mqyaot17 2442079 2442078 2026-07-28T20:53:00Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2442079 wikitext text/x-wiki 6wb2exnzoc3w68i2wrhe58bi5rxx34c 2442080 2442079 2026-07-28T20:53:08Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 30s) 2442080 wikitext text/x-wiki n7y4q65j9ivok5tiky7agrzeukmt057 2442081 2442080 2026-07-28T20:54:05Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2442081 wikitext text/x-wiki d4x9yhxpq1yannrkxyygfp0uzjx51f4 2442082 2442081 2026-07-28T20:54:13Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 14s) 2442082 wikitext text/x-wiki bgawikpffhuo57045orhzp3d1448mnm 2442083 2442082 2026-07-28T20:54:27Z Stashbot 7414 arlolra@deploy1003: arlolra: Continuing with deployment 2442083 wikitext text/x-wiki dxf4zn6hulfhs1j8nd71vg1eja49cpt 2442084 2442083 2026-07-28T20:55:21Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling source-only afterwards 2442084 wikitext text/x-wiki e6wuvrlomuepbm6d8uc1lmrv9fiddab 2442085 2442084 2026-07-28T20:55:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards 2442085 wikitext text/x-wiki 3fp4rqyixg4126mtcbqx6sg20oj1yda 2442086 2442085 2026-07-28T20:58:41Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1318746|Bump wikimedia/parsoid to 0.24.0-a17 (T19006 T234299 T360814 T385317 T432477 T433018)]], [[gerrit:1318747|Bump wikimedia/parsoid to 0.24.0-a17 (T433018)]] (duration: 08m 19s) 2442086 wikitext text/x-wiki amo0p5vastemtwvfc6d7wdbk5hfl6g3 2442087 2442086 2026-07-28T22:03:50Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie 2442087 wikitext text/x-wiki sem7tyymz1pyd3nn9ol6kzjvj3lsh96 2442088 2442087 2026-07-28T22:08:08Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling source-only afterwards 2442088 wikitext text/x-wiki 56p6cskpuevf0pzevc3phrzjv0awsiz 2442089 2442088 2026-07-28T22:11:22Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards 2442089 wikitext text/x-wiki 8ol8exldnnw3xjh0au3mw32zo2pnh7u 2442091 2442089 2026-07-28T22:49:59Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1138.eqiad.wmnet 2442091 wikitext text/x-wiki ct5ey2dvle0rbmjcnryvtel7hxu9yxs 2442092 2442091 2026-07-28T22:50:01Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1138.eqiad.wmnet 2442092 wikitext text/x-wiki tnrr8z7env0x3uou8sbpkudvml5woii 2442093 2442092 2026-07-28T22:50:03Z Stashbot 7414 jiji@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1138.eqiad.wmnet 2442093 wikitext text/x-wiki dsx45ssfnbafyfjvgkqwk4xudzm4ovx 2442103 2442093 2026-07-29T05:46:54Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1164.eqiad.wmnet with reason: Reimage 2442103 wikitext text/x-wiki ec47rbzbboy6397n8nvc3uz19jyqyfd 2442104 2442103 2026-07-29T05:47:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1164.eqiad.wmnet with OS trixie 2442104 wikitext text/x-wiki md2mwyry8l3o9ras4q7qkhbkd2vyqy1 2442105 2442104 2026-07-29T06:01:24Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage 2442105 wikitext text/x-wiki nw1xzjth0ak04uw78bzik4yzp5t5p9k 2442106 2442105 2026-07-29T06:05:12Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage 2442106 wikitext text/x-wiki 5ovlq87tz6g1xhqrj6zatja1vcvlroq 2442108 2442106 2026-07-29T06:26:59Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1164.eqiad.wmnet with OS trixie 2442108 wikitext text/x-wiki dhjbc9hlxk3eg3b5pjqzpe72i8edvr3 2442110 2442108 2026-07-29T06:58:15Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2160,2232].codfw.wmnet with reason: Reimage 2442110 wikitext text/x-wiki 2rp7op6jgp7h5f86v3qwkyqtodrqe67 2442111 2442110 2026-07-29T06:58:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2232.codfw.wmnet with OS trixie 2442111 wikitext text/x-wiki 92pe2cjmbquzgkwzzbayr8klyir8j3s 2442112 2442111 2026-07-29T07:15:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2232.codfw.wmnet with reason: host reimage 2442112 wikitext text/x-wiki la1hnpe0nmu501nj97txuwbiry9s2bc 2442113 2442112 2026-07-29T07:19:31Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2232.codfw.wmnet with reason: host reimage 2442113 wikitext text/x-wiki 73xzt8yl3hes4oxz4ogndk8olab9yi4 2442114 2442113 2026-07-29T07:31:09Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2442114 wikitext text/x-wiki qgnrp54tz2vp6ytxgvtcm9dd70nic3b 2442115 2442114 2026-07-29T07:33:58Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2442115 wikitext text/x-wiki ffe7b4fm9liw3j3rfyao36n52x0fsmc 2442116 2442115 2026-07-29T07:34:10Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2442116 wikitext text/x-wiki isj1qvqr0j05mm34qv0qunioiqiqnld 2442117 2442116 2026-07-29T07:34:27Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2442117 wikitext text/x-wiki 04rwxna2tzq69gxkaiwiz03ea4guonw 2442118 2442117 2026-07-29T07:42:13Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2232.codfw.wmnet with OS trixie 2442118 wikitext text/x-wiki q028tu3cvfhkh2yy4482coey4i5bnfi 2442119 2442118 2026-07-29T07:42:54Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet 2442119 wikitext text/x-wiki 8mf0im3513w5dzdhk9xs8utnqwwaav1 2442120 2442119 2026-07-29T07:46:36Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet 2442120 wikitext text/x-wiki hfxp2wuk5n67t91jphrmlihn0zomxyw 2442121 2442120 2026-07-29T07:46:46Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: Maintenance 2442121 wikitext text/x-wiki bh6tlchxfdhser6imecnl59wmfuca2i 2442122 2442121 2026-07-29T07:46:54Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1163 (T431660)', diff saved to https://phabricator.wikimedia.org/P95441 and previous config saved to /var/cache/conftool/dbconfig/20260729-074652-cwilliams.json 2442122 wikitext text/x-wiki 6juxcapdwk5eb9ui5mfxc2jc9w1tk3z 2442123 2442122 2026-07-29T07:48:49Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance 2442123 wikitext text/x-wiki pztc82f8n2sx9755klncwrdv9ql76sl 2442124 2442123 2026-07-29T07:49:00Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2442124 wikitext text/x-wiki mh3zyy4ti1favs5imu0ba388e1t6wii 2442125 2442124 2026-07-29T07:49:07Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance 2442125 wikitext text/x-wiki j9ok1nka82zw9484zr00oc8yqb5bm4p 2442126 2442125 2026-07-29T07:49:09Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1156 (T431660)', diff saved to https://phabricator.wikimedia.org/P95442 and previous config saved to /var/cache/conftool/dbconfig/20260729-074906-cwilliams.json 2442126 wikitext text/x-wiki pggjehhynlipsq0aq96ot0ele2cp4g8 2442127 2442126 2026-07-29T07:49:16Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1157 (T431660)', diff saved to https://phabricator.wikimedia.org/P95443 and previous config saved to /var/cache/conftool/dbconfig/20260729-074914-cwilliams.json 2442127 wikitext text/x-wiki h3ltl5y0zdnohmga0hxk0dcnysiz81j 2442128 2442127 2026-07-29T07:49:24Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance 2442128 wikitext text/x-wiki 9unj92vlyazbia5im79ivmuxafaodjq 2442129 2442128 2026-07-29T07:49:32Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1190 (T431660)', diff saved to https://phabricator.wikimedia.org/P95444 and previous config saved to /var/cache/conftool/dbconfig/20260729-074930-cwilliams.json 2442129 wikitext text/x-wiki 9u3eao8iratame9d582c7s29xbhm820 2442131 2442129 2026-07-29T07:53:23Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1163: Maintenance 2442131 wikitext text/x-wiki 58cnbgbo358y1xgp0ojz61gzhbmscgy 2442132 2442131 2026-07-29T07:55:35Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Maintenance 2442132 wikitext text/x-wiki jz5hi41e2n2cejb5bvg56qaf1dhyjat 2442133 2442132 2026-07-29T07:55:39Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1156: Maintenance 2442133 wikitext text/x-wiki jgl5pi04wxg2sbyuuaptklnwvj76ux8 2442134 2442133 2026-07-29T07:57:12Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1190: Maintenance 2442134 wikitext text/x-wiki ecd4vdcmlrx6ntjz8300f4y9dh4bokv 2442135 2442134 2026-07-29T07:58:32Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2072.codfw.wmnet with OS trixie 2442135 wikitext text/x-wiki 1wm66odujbm9dzqnlccv7i22vt0l2lj 2442136 2442135 2026-07-29T08:19:41Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2072.codfw.wmnet with reason: host reimage 2442136 wikitext text/x-wiki 02ql7hglakws44ii6yfd9y9gki2fcip 2442137 2442136 2026-07-29T08:20:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1098 hosts 2442137 wikitext text/x-wiki gwxgj4yw6adzxtpfhq9zan41za446m3 2442138 2442137 2026-07-29T08:23:47Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2072.codfw.wmnet with reason: host reimage 2442138 wikitext text/x-wiki sf503s1qqzmjj86twugvjvx4qvx8hok 2442139 2442138 2026-07-29T08:30:29Z Stashbot 7414 btullis@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 11 hosts with reason: Replacing the namenodes 2442139 wikitext text/x-wiki 0iu33jqk6h360winn8t7ww7av6klw8z 2442140 2442139 2026-07-29T08:41:21Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Maintenance 2442140 wikitext text/x-wiki a8a17ckek3t62797oo6hm7dh6f49emy 2442141 2442140 2026-07-29T08:41:41Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance 2442141 wikitext text/x-wiki 8cqryqro5k8n82au55vgw69kj97vk0k 2442142 2442141 2026-07-29T08:41:49Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1169 (T431660)', diff saved to https://phabricator.wikimedia.org/P95458 and previous config saved to /var/cache/conftool/dbconfig/20260729-084147-cwilliams.json 2442142 wikitext text/x-wiki 1rbkm8ec9zxnriulymiyr2yrwuok5ao 2442143 2442142 2026-07-29T08:43:33Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Maintenance 2442143 wikitext text/x-wiki mavyp5rp2q6wr73o1kha4u2bznyt74m 2442144 2442143 2026-07-29T08:43:53Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance 2442144 wikitext text/x-wiki lg7jg9482f7hsce0vpuzxrzmygkwgx7 2442145 2442144 2026-07-29T08:44:02Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1166 (T431660)', diff saved to https://phabricator.wikimedia.org/P95460 and previous config saved to /var/cache/conftool/dbconfig/20260729-084400-cwilliams.json 2442145 wikitext text/x-wiki eo56kj7jdatr00k26ldkiee90k0lhed 2442146 2442145 2026-07-29T08:44:09Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1156: Maintenance 2442146 wikitext text/x-wiki n0cpq6z4dv0lq7sg46b6y8gvrzb6hha 2442147 2442146 2026-07-29T08:44:29Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance 2442147 wikitext text/x-wiki ev99ii6t5usvxihw5ikrvd232y5u1nx 2442148 2442147 2026-07-29T08:44:38Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1182 (T431660)', diff saved to https://phabricator.wikimedia.org/P95462 and previous config saved to /var/cache/conftool/dbconfig/20260729-084436-cwilliams.json 2442148 wikitext text/x-wiki gs6x58pzi0ot286uk4r5904p0aaei0h 2442149 2442148 2026-07-29T08:45:08Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1190: Maintenance 2442149 wikitext text/x-wiki 5cn8qdpcqeq4yik78cn6emorxf3ia6p 2442150 2442149 2026-07-29T08:45:29Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: router upgrade, T431750] 2442150 wikitext text/x-wiki di950bt18uw7p2j4v4x3s1b4guwowvb 2442151 2442150 2026-07-29T08:45:32Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance 2442151 wikitext text/x-wiki cuo8z09i3cr6qz2u97zpeuxbcw64zpz 2442152 2442151 2026-07-29T08:45:34Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: router upgrade, T431750] 2442152 wikitext text/x-wiki dm9jebmr7qw52u00i85hchzry8axzjf 2442153 2442152 2026-07-29T08:45:36Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1199 (T431660)', diff saved to https://phabricator.wikimedia.org/P95464 and previous config saved to /var/cache/conftool/dbconfig/20260729-084534-cwilliams.json 2442153 wikitext text/x-wiki k1psk7ad8a67236f0021p1eafer3mne 2442154 2442153 2026-07-29T08:45:42Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2072.codfw.wmnet with OS trixie 2442154 wikitext text/x-wiki l3bxllwnfg9acxb84f7ldsyaozlfh7m 2442155 2442154 2026-07-29T08:46:13Z Stashbot 7414 ayounsi@cumin1003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on cr1-magru,cr1-magru IPv6,cr1-magru.mgmt with reason: router upgrade 2442155 wikitext text/x-wiki omtto427q8qibvr15kl2ikwwtvpyb9o 2442156 2442155 2026-07-29T08:47:20Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1169: Maintenance 2442156 wikitext text/x-wiki 0gqkwm13b1jbl5syirokzve91dflhmn 2442157 2442156 2026-07-29T08:50:20Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1166: Maintenance 2442157 wikitext text/x-wiki nru5bfdmgkd9jno748nte4eia8dtefq 2442158 2442157 2026-07-29T08:50:53Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1182: Maintenance 2442158 wikitext text/x-wiki ju038n9ev9vlns5jgxhge5m2crtb3w3 2442159 2442158 2026-07-29T08:53:11Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1199: Maintenance 2442159 wikitext text/x-wiki 2nchc7wfyuv28ayli34tab0e3u3969u 2442162 2442159 2026-07-29T09:00:03Z Stashbot 7414 marostegui: Dropping renamed tables T426341 2442162 wikitext text/x-wiki jf8zwzso9xdeyinastn5egoahyx5bbu 2442163 2442162 2026-07-29T09:00:15Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr1-magru,cr1-magru IPv6,cr1-magru.mgmt with reason: router upgrade 2442163 wikitext text/x-wiki f76e74zi66xvnzx1fo6o825pju7co2o 2442165 2442163 2026-07-29T09:00:48Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host idp2005.wikimedia.org 2442165 wikitext text/x-wiki 39oqx2jgblgkcuj731wqrgmyupqj9vw 2442166 2442165 2026-07-29T09:04:48Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2005.wikimedia.org 2442166 wikitext text/x-wiki dwwl6fsour982ahfsbl8xr78f85ioqy 2442167 2442166 2026-07-29T09:07:18Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host idp1005.wikimedia.org 2442167 wikitext text/x-wiki 8m976dqe34vtqydsenkne8ycoov58mn 2442168 2442167 2026-07-29T09:11:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1005.wikimedia.org 2442168 wikitext text/x-wiki hm319myvpg210wxpsgzyfabm6mbyr22 2442169 2442168 2026-07-29T09:11:47Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host idm1001.wikimedia.org 2442169 wikitext text/x-wiki h1vl906yjv3qseb05bd70sjsnacznyw 2442170 2442169 2026-07-29T09:11:58Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-magru,cr2-magru IPv6,cr2-magru.mgmt with reason: router upgrade 2442170 wikitext text/x-wiki ev5suu7r8cf3y4hejctqhpjdmxo07ww 2442171 2442170 2026-07-29T09:13:10Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2442171 wikitext text/x-wiki kz1j1faxje25y6fll5f7ktqyojpmciu 2442172 2442171 2026-07-29T09:14:00Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2442172 wikitext text/x-wiki s90n72l6bxry4gmw78668sfmi0blpeh 2442173 2442172 2026-07-29T09:14:51Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply 2442173 wikitext text/x-wiki ni91a74w7ium7yluv6kqwpo8t99c0ie 2442174 2442173 2026-07-29T09:15:39Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply 2442174 wikitext text/x-wiki 0t75o0l5pj1y3wsrk1dyy232b1csati 2442175 2442174 2026-07-29T09:15:45Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm1001.wikimedia.org 2442175 wikitext text/x-wiki mre524zdtj4g2bxf2fl6vd1u1dg5hh5 2442176 2442175 2026-07-29T09:17:19Z Stashbot 7414 XioNoX: drain cr1-magru - T431750 2442176 wikitext text/x-wiki 4g0zdsxry37c8huri8yp6qonadd1elj 2442177 2442176 2026-07-29T09:21:08Z Stashbot 7414 XioNoX: reboot cr1-magru - T431750 2442177 wikitext text/x-wiki 0yvmvsib7x8bgi56cmwk3zj1ozkq3c4 2442179 2442177 2026-07-29T09:21:45Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442179 wikitext text/x-wiki dg0amdx05ahgs0uu3k2uaeukipilepj 2442180 2442179 2026-07-29T09:33:18Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1033.eqiad.wmnet,service=s5 2442180 wikitext text/x-wiki e2s5d9hbva7a1ds3ibtl92rvzn98dje 2442181 2442180 2026-07-29T09:33:21Z Stashbot 7414 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1033.eqiad.wmnet,service=s8 2442181 wikitext text/x-wiki 21sgykc06sfn9wz34ne7ng1cnqumz85 2442182 2442181 2026-07-29T09:33:28Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1033.eqiad.wmnet,service=s5 2442182 wikitext text/x-wiki imqzb4bv1za22w27ahlb8dxyxvspmj3 2442183 2442182 2026-07-29T09:33:33Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1033.eqiad.wmnet,service=s8 2442183 wikitext text/x-wiki 126fqnx58mcy23vwgozb4zrg59b8gz9 2442184 2442183 2026-07-29T09:35:17Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1169: Maintenance 2442184 wikitext text/x-wiki 8j20g50bncvbwvrpzkb9xo4b0ifjrtk 2442185 2442184 2026-07-29T09:38:16Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1166: Maintenance 2442185 wikitext text/x-wiki 8q4mbmoduc2yueakp7qflcr0j3xk0qf 2442186 2442185 2026-07-29T09:38:36Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance 2442186 wikitext text/x-wiki oxnj1a9yenjh616wfe53nf0immgqwa6 2442187 2442186 2026-07-29T09:38:44Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1175 (T431660)', diff saved to https://phabricator.wikimedia.org/P95479 and previous config saved to /var/cache/conftool/dbconfig/20260729-093842-cwilliams.json 2442187 wikitext text/x-wiki h850357sqmds85rhl7gswg4vcxyfsu8 2442188 2442187 2026-07-29T09:38:50Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1182: Maintenance 2442188 wikitext text/x-wiki lcn7qciwdiftjfg2jhu1xobvq78xh35 2442189 2442188 2026-07-29T09:39:11Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance 2442189 wikitext text/x-wiki slr4s59emk4d61j8r6izjiudp959cka 2442190 2442189 2026-07-29T09:39:19Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1188 (T431660)', diff saved to https://phabricator.wikimedia.org/P95481 and previous config saved to /var/cache/conftool/dbconfig/20260729-093917-cwilliams.json 2442190 wikitext text/x-wiki q4h3fm8or7gwe10lj33elx67mez0fi4 2442194 2442190 2026-07-29T09:41:10Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1199: Maintenance 2442194 wikitext text/x-wiki 8i5lp9h6l0jjsndt3gf5bgg9wtfazi2 2442195 2442194 2026-07-29T09:41:27Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet 2442195 wikitext text/x-wiki cg0b98j9mrthnclmhzix3knk5pqc6de 2442196 2442195 2026-07-29T09:41:30Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance 2442196 wikitext text/x-wiki id2di4w7rmr5zy3nnny56prspkago3t 2442197 2442196 2026-07-29T09:41:53Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance 2442197 wikitext text/x-wiki 3bchy2ddio0vqugt2toou9fg9eahgfs 2442198 2442197 2026-07-29T09:42:02Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1221 (T431660)', diff saved to https://phabricator.wikimedia.org/P95483 and previous config saved to /var/cache/conftool/dbconfig/20260729-094200-cwilliams.json 2442198 wikitext text/x-wiki 3icrp48rc4wgp84b8g3f8mgqrgw2ibv 2442199 2442198 2026-07-29T09:42:46Z Stashbot 7414 btullis@dns1004: START - running authdns-update 2442199 wikitext text/x-wiki 4whhcldowr89hbpzpfw44nu4fk44a7j 2442200 2442199 2026-07-29T09:44:55Z Stashbot 7414 btullis@dns1004: END - running authdns-update 2442200 wikitext text/x-wiki p60ty9tnnpjh7mw9rb2qfitycitmo9l 2442201 2442200 2026-07-29T09:45:06Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1175: Maintenance 2442201 wikitext text/x-wiki ey19cavslmb3fq74k2m3528vn8u7pmd 2442202 2442201 2026-07-29T09:45:08Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1188: Maintenance 2442202 wikitext text/x-wiki 4cypijtvp2eibbbdvsuibcjij129wru 2442204 2442202 2026-07-29T09:45:29Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet 2442204 wikitext text/x-wiki 4zf2atn4xsvu0q2y3d5gvahjrhdiz5d 2442207 2442204 2026-07-29T09:48:21Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1221: Maintenance 2442207 wikitext text/x-wiki 0mvwake4fmyl3t4etli37kovfbbpx7m 2442208 2442207 2026-07-29T09:52:41Z Stashbot 7414 XioNoX: drain cr2-magru - T431750 2442208 wikitext text/x-wiki 36bqh4im65rm0tdiwaeg8x96caf298u 2442209 2442208 2026-07-29T09:53:33Z Stashbot 7414 XioNoX: reboot cr2-magru - T431750 2442209 wikitext text/x-wiki dgno5tqmvbbbcvft3simvpsj1oacyv7 2442213 2442209 2026-07-29T10:14:04Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: router upgrade, T431750] 2442213 wikitext text/x-wiki csd0ds71f5zllpzpd5v6zapjqxb3428 2442214 2442213 2026-07-29T10:14:10Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: router upgrade, T431750] 2442214 wikitext text/x-wiki dw7kyg0e9h4zi8q64tiz6wpas6fje5a 2442215 2442214 2026-07-29T10:21:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance 2442215 wikitext text/x-wiki 0pba9ruxguwos4dp1ah5xoq0c76lafz 2442216 2442215 2026-07-29T10:21:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1186 (T431660)', diff saved to https://phabricator.wikimedia.org/P95493 and previous config saved to /var/cache/conftool/dbconfig/20260729-102111-cwilliams.json 2442216 wikitext text/x-wiki ezq7coxmo0nb6lkkg2q33trlke0rg1p 2442218 2442216 2026-07-29T10:27:31Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1186: Maintenance 2442218 wikitext text/x-wiki dxhaetq9mo662mduwqhum1lbwiweski 2442219 2442218 2026-07-29T10:31:51Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1188: Maintenance 2442219 wikitext text/x-wiki 732zfld9xihxvwvrz8x8rwpiulao81t 2442220 2442219 2026-07-29T10:32:10Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance 2442220 wikitext text/x-wiki 99cvifk4lx9kxouhls2345n5mv84mek 2442221 2442220 2026-07-29T10:32:19Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1197 (T431660)', diff saved to https://phabricator.wikimedia.org/P95496 and previous config saved to /var/cache/conftool/dbconfig/20260729-103217-cwilliams.json 2442221 wikitext text/x-wiki gbw3mfvllkuciv0ykgf8ud2cj56olv6 2442222 2442221 2026-07-29T10:33:04Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1175: Maintenance 2442222 wikitext text/x-wiki o0ugprazergnzwpu3u6oullmhxi7sge 2442223 2442222 2026-07-29T10:33:24Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance 2442223 wikitext text/x-wiki hblzfiakdgmnex479y02wlydfin0zh7 2442224 2442223 2026-07-29T10:33:33Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1198 (T431660)', diff saved to https://phabricator.wikimedia.org/P95499 and previous config saved to /var/cache/conftool/dbconfig/20260729-103330-cwilliams.json 2442224 wikitext text/x-wiki ovs1cz0qkl6h18g2aakqdcmwkhrtjic 2442225 2442224 2026-07-29T10:35:05Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1221: Maintenance 2442225 wikitext text/x-wiki a84jaw3lj1n92ax8sjfg96z11t86kbe 2442226 2442225 2026-07-29T10:35:25Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance 2442226 wikitext text/x-wiki i4dcagw9b2q81srn1s1x600vfyt0sn2 2442227 2442226 2026-07-29T10:35:34Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1238 (T431660)', diff saved to https://phabricator.wikimedia.org/P95501 and previous config saved to /var/cache/conftool/dbconfig/20260729-103532-cwilliams.json 2442227 wikitext text/x-wiki 0q9uci31tqmeuqs8bgtewg8yud6f2ye 2442228 2442227 2026-07-29T10:38:21Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1197: Maintenance 2442228 wikitext text/x-wiki h9vdfg4tbgl4cnpb8yyzc8mtb5nid5a 2442229 2442228 2026-07-29T10:39:25Z Stashbot 7414 Amir1: ran https://phabricator.wikimedia.org/T432509#12149723 in production (T432509) 2442229 wikitext text/x-wiki 7zsmt39vq85ti2rpsh8wxh7z5fj7j9c 2442230 2442229 2026-07-29T10:39:53Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1198: Maintenance 2442230 wikitext text/x-wiki 48obkvpc0qnspxn9xf2ssme75a8qwj8 2442231 2442230 2026-07-29T10:41:58Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1238: Maintenance 2442231 wikitext text/x-wiki sixqbbhqj05ikm9z5y089mz6iyg89fh 2442232 2442231 2026-07-29T10:54:36Z Stashbot 7414 marostegui: Dropping renamed tables T425066 2442232 wikitext text/x-wiki hnnpwa70bp3mj0qfakfur42k0gtq1rw 2442233 2442232 2026-07-29T11:05:44Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442233 wikitext text/x-wiki gi3ngbyxkfln6kuydtjbaj9b1bl6gbn 2442234 2442233 2026-07-29T11:12:52Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2442234 wikitext text/x-wiki oesfpgybkk6wz1md4xqlghgrb9cq2i4 2442235 2442234 2026-07-29T11:13:26Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2442235 wikitext text/x-wiki ov1orj6m6iy0up8kw8o8t3vw5sxdp44 2442236 2442235 2026-07-29T11:14:13Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Maintenance 2442236 wikitext text/x-wiki 5xv9ptmr9gegxlhplk3feqox6hs9gsa 2442237 2442236 2026-07-29T11:14:44Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance 2442237 wikitext text/x-wiki kj8vr8a0irqhse6d8b9xtia29ycaitj 2442238 2442237 2026-07-29T11:14:52Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1195 (T431660)', diff saved to https://phabricator.wikimedia.org/P95514 and previous config saved to /var/cache/conftool/dbconfig/20260729-111450-cwilliams.json 2442238 wikitext text/x-wiki 3p39qnbmvn15epi62ktus9sp4zmptrn 2442239 2442238 2026-07-29T11:21:17Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1195: Maintenance 2442239 wikitext text/x-wiki dd5fr8wep4k6p7eplkcyfjrxczompam 2442240 2442239 2026-07-29T11:22:01Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2442240 wikitext text/x-wiki pd1h3tpzmmjjs3njucg7a5cjznw2lcz 2442241 2442240 2026-07-29T11:23:52Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2442241 wikitext text/x-wiki m2gghu1ghzck92zjjgyo23e71ld10jq 2442242 2442241 2026-07-29T11:26:38Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1198: Maintenance 2442242 wikitext text/x-wiki 9s2wlnowgs4b7egvtx8tg5dfg3rh9kd 2442243 2442242 2026-07-29T11:26:59Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance 2442243 wikitext text/x-wiki 8nqfs9htdlfcroi6thmdfqs1y1gksnq 2442244 2442243 2026-07-29T11:27:21Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2442244 wikitext text/x-wiki gtlyyhtlrxi2qzujymkho98yewuw1u9 2442245 2442244 2026-07-29T11:27:29Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1212 (T431660)', diff saved to https://phabricator.wikimedia.org/P95517 and previous config saved to /var/cache/conftool/dbconfig/20260729-112727-cwilliams.json 2442245 wikitext text/x-wiki cz12hxvpfoi97kebtkadqlwouwbikbx 2442246 2442245 2026-07-29T11:27:49Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2442246 wikitext text/x-wiki fegzy53zpbf2mc0kzlz2de5r57lmxff 2442247 2442246 2026-07-29T11:28:33Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2442247 wikitext text/x-wiki cfp6z7wybhnwgrbn1mn2on8ecdfggze 2442248 2442247 2026-07-29T11:29:03Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1238: Maintenance 2442248 wikitext text/x-wiki igxqhnpj28xufbfojxkcvnwr4ypog2b 2442249 2442248 2026-07-29T11:29:12Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance 2442249 wikitext text/x-wiki 2xmf9msnit55akb4od7rnsjx7at2qo2 2442250 2442249 2026-07-29T11:29:20Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling db1241 (T431660)', diff saved to https://phabricator.wikimedia.org/P95520 and previous config saved to /var/cache/conftool/dbconfig/20260729-112918-cwilliams.json 2442250 wikitext text/x-wiki t2zu8xftgwo4ns3womau8lypy5b6moe 2442251 2442250 2026-07-29T11:29:46Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2442251 wikitext text/x-wiki 5a8w1sq0dmyceaboc6xdxeu4tu9mzz2 2442252 2442251 2026-07-29T11:30:12Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2442252 wikitext text/x-wiki 82r7a275ka2lg28sn8oamyghb7x2edm 2442253 2442252 2026-07-29T11:33:27Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1212: Maintenance 2442253 wikitext text/x-wiki 6pxdab937rjvk429wig49rq3hxvh7ie 2442254 2442253 2026-07-29T11:35:44Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool db1241: Maintenance 2442254 wikitext text/x-wiki 7j9yhb1i8dadr2z4an5xs3canl8q63v 2442256 2442254 2026-07-29T11:49:20Z Stashbot 7414 elukey@deploy1003: helmfile [staging] START helmfile.d/services/proton: sync 2442256 wikitext text/x-wiki 5yrp017kbgjxcpgvggofwxh44g8gtql 2442257 2442256 2026-07-29T11:50:11Z Stashbot 7414 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: sync 2442257 wikitext text/x-wiki 6ps5qwa0gqi5yznyjb8er1adfj01ohf 2442258 2442257 2026-07-29T11:50:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.decommission for hosts an-test-coord1001.eqiad.wmnet 2442258 wikitext text/x-wiki bzuwy8qtjnoeqor90r6frz509ftj0l2 2442259 2442258 2026-07-29T11:51:27Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2442259 wikitext text/x-wiki h945zmoe20jridx1k3ws8sbwupxmklw 2442260 2442259 2026-07-29T11:51:44Z Stashbot 7414 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/proton: sync 2442260 wikitext text/x-wiki 4w9daz42dmqdvsam44je4fsgx1r7a7f 2442261 2442260 2026-07-29T11:51:52Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2442261 wikitext text/x-wiki prawrrx3jpwg95fthryxuyeq1h28pay 2442262 2442261 2026-07-29T11:52:54Z Stashbot 7414 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: sync 2442262 wikitext text/x-wiki bxb71ycpml2z8dkukun7uxncrey75zo 2442263 2442262 2026-07-29T11:55:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2442263 wikitext text/x-wiki 69aybtv2tje7y3v90kgugjyp7w5s4uf 2442264 2442263 2026-07-29T11:58:57Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1197: Maintenance 2442264 wikitext text/x-wiki 6bx9vakryr3p6s76bbchiivfu42ih87 12 18678 2441914 2441546 2026-07-28T17:02:28Z FNegri-WMF 32595 Merge "Databases" section into lead paragraph, as they contained similar info. 2441914 wikitext text/x-wiki r9ruwp5psvwcf1uqyu5pmk1jsumd5x3 2441915 2441914 2026-07-28T17:04:58Z FNegri-WMF 32595 Remove empty line 2441915 wikitext text/x-wiki 4ta3dts6t73fxt20njcnct52yysbdlv 2441930 2441915 2026-07-28T17:32:33Z FNegri-WMF 32595 Move "Credentials" section higher up in the page 2441930 wikitext text/x-wiki efdqifp1yhiwehinjen1j4402xaw3d7 2441942 2441930 2026-07-28T17:35:51Z FNegri-WMF 32595 Move ToolsDB connection example to the right section 2441942 wikitext text/x-wiki 287v0g9angme9hyqbxyh5jdo4e2ltlk 2441946 2441942 2026-07-28T17:38:13Z FNegri-WMF 32595 /* Connecting to the user databases */ Specify you need to log into the bastion first 2441946 wikitext text/x-wiki 82x9k2pkit9sqxnm97jwne5pa8vi1y6 2441956 2441946 2026-07-28T17:46:30Z FNegri-WMF 32595 Move "See also" links into "Replica databases" section 2441956 wikitext text/x-wiki q16wrik4dboepw7elndkx1f2d5n2i1w 498 30942 2441580 2441577 2026-07-28T12:00:15Z Stashbot 7414 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=1) 2441580 wikitext text/x-wiki oelhjpddgqffwc0kijosztq70venyj0 2441586 2441580 2026-07-28T12:06:00Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441586 wikitext text/x-wiki 73avlk1n6n49ip35do8asad9545rs8e 2441587 2441586 2026-07-28T12:06:10Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441587 wikitext text/x-wiki 09yfxgod14h2y494p92mpdiq7o989xn 2441589 2441587 2026-07-28T12:08:04Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441589 wikitext text/x-wiki 0nu59xkxkd76y9mt05lka6gj986trp4 2441590 2441589 2026-07-28T12:08:08Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441590 wikitext text/x-wiki 5r0jrh0swjem58uqbgca1vm8mjqngho 2441595 2441590 2026-07-28T12:12:31Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441595 wikitext text/x-wiki 1jjs3s63t3r211lx7uqbo0s90loosof 2441596 2441595 2026-07-28T12:12:34Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441596 wikitext text/x-wiki 2ja3jlvqfq6hwvo0uygtdpg96sz43zs 2441641 2441596 2026-07-28T12:54:14Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441641 wikitext text/x-wiki kv2m2qnlr377sgln9n5mr0nwwf2w51x 2441642 2441641 2026-07-28T12:54:18Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441642 wikitext text/x-wiki cqwdu8t9nl1vc3jqwa13aybtbnieelx 2441643 2441642 2026-07-28T12:54:27Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441643 wikitext text/x-wiki 57twkneug0gj2xnbaluv0f5wxect3bi 2441644 2441643 2026-07-28T12:54:31Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441644 wikitext text/x-wiki g25jfxm42vz5ibsnyib45uw8k23vy7c 2441645 2441644 2026-07-28T12:55:16Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441645 wikitext text/x-wiki pj15va2wjenup0g1jf4prlnidjuzebd 2441646 2441645 2026-07-28T12:55:19Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441646 wikitext text/x-wiki j1xod6w6vsdrdvvikv5c70v3s5exf33 2441648 2441646 2026-07-28T12:59:23Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441648 wikitext text/x-wiki khr1hxvu6v5ozbf0f0oi7xdeo2biujo 2441649 2441648 2026-07-28T12:59:27Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441649 wikitext text/x-wiki 861c4invz4lopnsobv82nscp5y224w7 2441650 2441649 2026-07-28T12:59:59Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441650 wikitext text/x-wiki 2sxo7774stcuoimtost0u5hnxnzhajz 2441651 2441650 2026-07-28T13:00:10Z Stashbot 7414 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=0) 2441651 wikitext text/x-wiki o2o7kqbt96cr7gf9xrve1pgbn8dcifp 2441653 2441651 2026-07-28T13:00:49Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441653 wikitext text/x-wiki 034ekoaoet6rqtts7s8mh9c9at0f8ym 2441654 2441653 2026-07-28T13:00:56Z Stashbot 7414 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=0) 2441654 wikitext text/x-wiki suvdh040imqlwtu1tau7kxqqlj2t8at 2441657 2441654 2026-07-28T13:02:29Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441657 wikitext text/x-wiki l046s1946jgduyqmci6uv01o5a3nfa4 2441659 2441657 2026-07-28T13:02:36Z Stashbot 7414 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=0) 2441659 wikitext text/x-wiki jz043hgd8230vscpyfzj4f3l1flxk4r 2441660 2441659 2026-07-28T13:03:25Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441660 wikitext text/x-wiki 9er0o8wntxtsrwvtkk4fce4d8edw9hs 2441661 2441660 2026-07-28T13:03:33Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=99) 2441661 wikitext text/x-wiki 00ju8opqn6vtjephqup9tbc6kyxcjlt 2441662 2441661 2026-07-28T13:03:53Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441662 wikitext text/x-wiki 4blw6f56p9nztnm2wepgr03qa4hverb 2441663 2441662 2026-07-28T13:04:12Z Stashbot 7414 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=1) 2441663 wikitext text/x-wiki fsleoqwuwd1uey8953fm1dfhzo1z6rs 2441664 2441663 2026-07-28T13:04:20Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441664 wikitext text/x-wiki e0mk0vpysjez6q2o3naems45e2yvc5b 2441666 2441664 2026-07-28T13:04:32Z Stashbot 7414 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=0) 2441666 wikitext text/x-wiki hqy2idvajaz3s1ggfmde25qsiecx77c 2441669 2441666 2026-07-28T13:04:47Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441669 wikitext text/x-wiki lvgwcndsgwwnrtflgkyonlf8f5e3dp7 2441670 2441669 2026-07-28T13:04:58Z Stashbot 7414 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=0) 2441670 wikitext text/x-wiki ac7le3l3v4h85o2au8l18rpuc7a0xl3 2441671 2441670 2026-07-28T13:05:55Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.get_project_for_proxy 2441671 wikitext text/x-wiki g8s37lsn770nvmx8qf2l5e7c9aj92en 2441672 2441671 2026-07-28T13:06:09Z Stashbot 7414 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.get_project_for_proxy (exit_code=0) 2441672 wikitext text/x-wiki kiboezs54xn2ae4qfs6mo2bn5rd1p2n 2441679 2441672 2026-07-28T13:11:49Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons (T431659) 2441679 wikitext text/x-wiki k06dxas6xugszs6hqq0nlsp96rtkbzq 2441687 2441679 2026-07-28T13:14:01Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) 2441687 wikitext text/x-wiki jcialo5qnqbdnx86rk165q6bwg6u1zf 2441701 2441687 2026-07-28T13:24:44Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) (T431659) 2441701 wikitext text/x-wiki 3mbssq2zalaf7ttm6d90ef0keope8l6 0 35071 2442101 2401120 2026-07-29T02:57:20Z SWilson (WMF) 57863 Remove local development info, to keep everything in README.md. Also update link to the font list. 2442101 wikitext text/x-wiki sjdpsna96dex1s5b836kufp4h9ufdh0 0 441012 2442102 2433327 2026-07-29T03:21:55Z SSapaty (WMF) 28955 /* Community Requests */ link to account request approval checklist 2442102 wikitext text/x-wiki tsn08rv4sgftnmgbfw63th18heshfqh 2442160 2442102 2026-07-29T08:56:08Z FNegri-WMF 32595 /* Community Requests */ remove link to "Users and community" (the other link in the section is more specific); add link to Admin/#Quota_management 2442160 wikitext text/x-wiki dzvn84rocre8guetllgmeenk531558x 2442164 2442160 2026-07-29T09:00:17Z FNegri-WMF 32595 /* Community Requests */ Toolforge quota requests: reorder, fix one link 2442164 wikitext text/x-wiki 1vsn6xefqn3iv4cp05ksqd6p26jkdpu 498 442301 2441705 2386655 2026-07-28T13:27:37Z Stashbot 7414 taavi: migrating to new opensearch cluster 2441705 wikitext text/x-wiki 9qa42gwfxosvxbuxl7l74axka7yj4q4 0 443680 2442178 2441433 2026-07-29T09:21:40Z JWodstrcil (WMF) 24826 add a sentence about updating the Dockerfile.template for helm-linter. 2442178 wikitext text/x-wiki l2wng3edngxaltrjbtenaut81zx9gu5 498 448635 2441963 2199526 2026-07-28T18:04:23Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2441963 wikitext text/x-wiki cytqw9thrhpjjqxhz4bgbw8gi9nyz4o 2441970 2441963 2026-07-28T18:16:18Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) 2441970 wikitext text/x-wiki liuk9everdb7v8g4vf7g3faeui6n4xm 0 448785 2442090 2439166 2026-07-28T22:15:30Z EMcnaughton (WMF) 6074 2442090 wikitext text/x-wiki rk1uprbsjbyofl1mqbs4p21i0kuttk3 0 449160 2441640 2441424 2026-07-28T12:53:18Z Dexbot 30554 Bot: Updating the report 2441640 wikitext text/x-wiki iaji55mjzc8ohi7q4vpn9ugnynmx0ty 2442098 2441640 2026-07-29T00:00:49Z Dexbot 30554 Bot: Updating the report 2442098 wikitext text/x-wiki mp1v4ankziv7zi6jgmciu93z9w8u2lp 2442130 2442098 2026-07-29T07:49:50Z Dexbot 30554 Bot: Updating the report 2442130 wikitext text/x-wiki rqq10xsnqadlcey9in7oh4gbdr1vuhx 0 451533 2441792 2439789 2026-07-28T14:53:35Z DCaro (WMF) 18271 /* Next meeting agenda */ 2441792 wikitext text/x-wiki c843kkm9ubza7prxj3p8iiyjmkwhi68 0 452383 2441906 2436501 2026-07-28T16:53:09Z HNowlan (WMF) 15760 /* What constitutes emergency cover? */ decentre IC involvement in emergency decisions - the emergency is the event we are concerned with. 2441906 wikitext text/x-wiki 7kjwbdio4mkbxz99kfcu4sutisvp50t 0 453382 2442191 2440629 2026-07-29T09:39:29Z CMooney (WMF) 24288 /* Show Nokia's current gNMI subscriptions */ 2442191 wikitext text/x-wiki m14riwt5rr11lep09t1eus8wes4zemr 2442217 2442191 2026-07-29T10:23:11Z CMooney (WMF) 24288 /* Restart gnmic daemon on switch */ 2442217 wikitext text/x-wiki 4ipta3ywa91tp3o5dxpzewptqlfncke 116 453906 2441905 2441489 2026-07-28T16:51:38Z Gitlabaccountapprovalbot 37332 @for-each-next was approved. 2441905 wikitext text/x-wiki 053tm1v5zrj9uoump5xk8s1a9lwsniy 2442109 2441905 2026-07-29T06:33:16Z Gitlabaccountapprovalbot 37332 rafiul1 was rejected. 2442109 wikitext text/x-wiki tv5u6az61sv40nculyc8tsntvk9mubf 2442212 2442109 2026-07-29T10:03:34Z Gitlabaccountapprovalbot 37332 @piastu was approved. 2442212 wikitext text/x-wiki rdb57xm6mo0p12w3dn5vs9nqr1r41mp 0 459290 2441840 2441576 2026-07-28T15:31:07Z AMastilovic-WMF 40067 Details on 2026-07-22 deployment 2441840 wikitext text/x-wiki 2wo3ssftj61liq0uv2k7ixwatslrlya 0 459517 2442094 2367755 2026-07-28T22:55:44Z EMcnaughton (WMF) 6074 /* Audit Process Documentation for Intact / Sage */ 2442094 wikitext text/x-wiki dh5meb6daf3vvhct96j48yjx3ppotvt 0 459560 2441953 2436560 2026-07-28T17:43:22Z DAdedoyin (WMF) 39150 Test 2441953 wikitext text/x-wiki brv5dq7w2u7bl4mxbv1cpfvkhcnj0rw 0 459699 2441888 2440555 2026-07-28T16:34:29Z Stashbot 7414 ejegg: standalone (IPN listener) SmashPig upgraded from 038018d4 to 0c2593b5 2441888 wikitext text/x-wiki hb1ca82tnt8ctrx76qqhzzvzm0uurpm 2442041 2441888 2026-07-28T20:08:55Z Stashbot 7414 ejegg: fundraising civicrm upgraded from 98ef4abd to 26bd5ab6 2442041 wikitext text/x-wiki eutgtqrxfmd2gf79uqctqc54wpyjwc9 2 460394 2441958 2432035 2026-07-28T17:51:12Z Milimetric 612 2441958 wikitext text/x-wiki 28gl716d5s6q6qoa5ap70xp2y37sdlm 3 460497 2442211 2026-07-29T09:58:53Z StrikerBot 8475 Welcome to Toolforge! 2442211 wikitext text/x-wiki fzt0blciyn46x2fz6l5n0wwb2mjb1lb