Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.11 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 1516 2438055 2435482 2026-07-16T17:12:26Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2438055 wikitext text/x-wiki dgw8zleu98wuq8aowo4zg7p5j03g2ee 2438061 2438055 2026-07-16T17:22:18Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) 2438061 wikitext text/x-wiki kgrtf4016tiji8fnvqts8ifn328jvpe 2438063 2438061 2026-07-16T17:24:46Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service (T430589) 2438063 wikitext text/x-wiki ohj2qgx11x4pcfl0ebugd83ghfocal0 2438064 2438063 2026-07-16T17:24:56Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589) 2438064 wikitext text/x-wiki ovo7t8i35jpy6xb11c01sosuxmbam5v 2438065 2438064 2026-07-16T17:26:14Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service (T430589) 2438065 wikitext text/x-wiki iv9zj0706nhlbqhxlz7uevsnny38402 2438066 2438065 2026-07-16T17:26:18Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.migrate_service (exit_code=97) (T430589) 2438066 wikitext text/x-wiki 3iornqaq1uf89spl4ybqu5y5lyk4pwd 2438067 2438066 2026-07-16T17:26:46Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service (T430589) 2438067 wikitext text/x-wiki 7uhpgdpepgayif2m7z6kp01wr024exq 2438073 2438067 2026-07-16T17:30:28Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589) 2438073 wikitext text/x-wiki 00pruu47k4mzkh70iikkidjjz9zlppo 2438074 2438073 2026-07-16T17:30:39Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service (T430589) 2438074 wikitext text/x-wiki 2d3zvpmdrdeoytxjmudejtczsy8yuor 2438075 2438074 2026-07-16T17:30:41Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589) 2438075 wikitext text/x-wiki c1k0te8s07bw80thve1bvvcgz4wxct1 2438171 2438075 2026-07-16T20:04:14Z Stashbot 7414 Krinkle: Delete cvn-nfs1 per T430589 2438171 wikitext text/x-wiki d29ji23tmb3vgeaszjtro21r1npml4a 498 3086 2437694 2436713 2026-07-16T12:26:34Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2437694 wikitext text/x-wiki ii24y9s51sivpfo2lacxq109sydyvod 2437701 2437694 2026-07-16T12:39:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2437701 wikitext text/x-wiki qihemlkb2bgiwwrrxediju27q75xm3o 2437724 2437701 2026-07-16T13:01:41Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2437724 wikitext text/x-wiki 735tqprbxiit68snlmtgktpz31to5mj 2437742 2437724 2026-07-16T13:13:20Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2437742 wikitext text/x-wiki sb04dlbsstuabc6rcx1j1xtaesb85pu 2437780 2437742 2026-07-16T13:42:41Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2437780 wikitext text/x-wiki qoj7qezb8dvrqw1ybpbycxr4r1tbh9z 2437793 2437780 2026-07-16T13:55:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2437793 wikitext text/x-wiki rinivyunyaccx42nk7gp7gyovod29os 2437917 2437793 2026-07-16T15:02:18Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2437917 wikitext text/x-wiki 4w2dhgrave35aqzndoaez3ygkepd0di 2437929 2437917 2026-07-16T15:14:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2437929 wikitext text/x-wiki hqnqv7jpef2sr7len8a0j68he27rjbm 0 3170 2437713 2428672 2026-07-16T12:50:56Z SSingh (WMF) 13928 /* Depool a host */ 2437713 wikitext text/x-wiki 5bg73wraxubrq2cf7lj8wq3apf0m1vn 2437714 2437713 2026-07-16T12:51:29Z SSingh (WMF) 13928 /* Depool /pool a host */ 2437714 wikitext text/x-wiki m2ythtk8knjpbe4s27webor8yplc88q 2437716 2437714 2026-07-16T12:55:02Z SSingh (WMF) 13928 /* Pool or Depool a DNS host */ update instructions for maintenance a bit more clear 2437716 wikitext text/x-wiki 033mvwgq0foa5pr62kj3hv4vhmzn1l5 2437718 2437716 2026-07-16T12:57:00Z SSingh (WMF) 13928 /* DNS Host Maintenance */ use a proper hostname 2437718 wikitext text/x-wiki 66qrr8i9g5ki8f29lv0xlfjgm7npa34 2437719 2437718 2026-07-16T12:57:34Z SSingh (WMF) 13928 /* DNS Host Maintenance */ 2437719 wikitext text/x-wiki er61amzcdrm6z3b92dt1gftkqhhwrbj 0 3205 2438442 2433913 2026-07-17T10:26:22Z BJensen-WMF 48942 /* Remove a load balanced service */ Add clarifying step numbers. 2438442 wikitext text/x-wiki crqbrp29r9wgjz4o5t8iw6fj3ls1s77 0 4108 2437847 2436990 2026-07-16T14:27:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1311472]] to Thursday, July 16 UTC late backport window 2437847 wikitext text/x-wiki 28ihp9axexeym1j2f29d45viq49qztf 2437977 2437847 2026-07-16T16:08:46Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-07-16}} */ UTC-late infra window 2437977 wikitext text/x-wiki fjwrzg7llfsrr0iwkcza0frxxaraplm 2438022 2437977 2026-07-16T16:50:15Z ScheduleDeploymentBot 37566 Add [[gerrit:1311481]] to Thursday, July 16 UTC late backport window 2438022 wikitext text/x-wiki bhduscf0xm2rzkcm8pzuehx2zss31rg 2438023 2438022 2026-07-16T16:50:22Z ScheduleDeploymentBot 37566 Add [[gerrit:1311482]] to Thursday, July 16 UTC late backport window 2438023 wikitext text/x-wiki 6ssi2dynhd4ljw8bvbxfxh98l7u3i3f 498 6726 2437690 2436706 2026-07-16T12:14:36Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2437690 wikitext text/x-wiki d4agl2ec8gbopg21fekotodfhudxj1b 2437693 2437690 2026-07-16T12:25:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2437693 wikitext text/x-wiki bmw6c8oofljb7t3mchxl1aw9622apiu 2437712 2437693 2026-07-16T12:49:50Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2437712 wikitext text/x-wiki 2jm1pkrpea85z32yq3omj6giodhgf7s 2437723 2437712 2026-07-16T13:01:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2437723 wikitext text/x-wiki 8zreadopcx90yjcdopnmnzaropupg6s 2437771 2437723 2026-07-16T13:30:54Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2437771 wikitext text/x-wiki kknrfd440h5bq0x6r7v38o7hvizjbss 2437779 2437771 2026-07-16T13:42:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2437779 wikitext text/x-wiki 0y0ffyu8792iowfqi2u9f3na5c4z6ec 2437859 2437779 2026-07-16T14:36:54Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2437859 wikitext text/x-wiki 3r9r7n4uzrncu5fr7eiw18ph1i6pcuy 2437874 2437859 2026-07-16T14:48:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2437874 wikitext text/x-wiki j89c8hdc3ndc5zxt2zx9zl58ivswjl4 0 7195 2438347 2412355 2026-07-17T05:32:19Z Elli 25566 /* {{anchor|Server clusters}} Data centers */ link improvements 2438347 wikitext text/x-wiki ry6z33im3hpn9jigi4ecczhvfi95wot 0 7919 2437674 2437673 2026-07-16T11:59:13Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1267-1276].eqiad.wmnet 2437674 wikitext text/x-wiki howm4c89zu44twwy2i9vmox0a4f01iy 2437675 2437674 2026-07-16T12:00:00Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet 2437675 wikitext text/x-wiki hid324khhlc38btrsjxl8wszo0r5ata 2437676 2437675 2026-07-16T12:00:58Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet 2437676 wikitext text/x-wiki 1e0wlb1v9gkn2olsv8m1vfkwo97xhws 2437677 2437676 2026-07-16T12:01:14Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet 2437677 wikitext text/x-wiki 6l156m9wc6g64p9lhrx7tthqa450iw4 2437678 2437677 2026-07-16T12:01:17Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-codfw 2437678 wikitext text/x-wiki dxuvj1fq8clsem8lhivx5qvs3tfod6g 2437679 2437678 2026-07-16T12:01:38Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2437679 wikitext text/x-wiki fn2kgv67ezh3j7jbhfzgmamykske60a 2437680 2437679 2026-07-16T12:01:46Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet 2437680 wikitext text/x-wiki 6dg0iplmfgiy6h455786dpl347a8dha 2437681 2437680 2026-07-16T12:01:57Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet 2437681 wikitext text/x-wiki rtcc5ehocu2xh6ss1985t0wrhuv2wub 2437682 2437681 2026-07-16T12:02:07Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet 2437682 wikitext text/x-wiki 6lznv6l2w92icm0kvpf04gty52ova0o 2437683 2437682 2026-07-16T12:02:46Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2437683 wikitext text/x-wiki szm2qnwmw7g5ji2kpazxg0bcmjjafi0 2437684 2437683 2026-07-16T12:03:42Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns5004.wikimedia.org 2437684 wikitext text/x-wiki dxezmstgypjdjedl88wpqc68rtxq4qx 2437685 2437684 2026-07-16T12:04:45Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1267-1276].eqiad.wmnet 2437685 wikitext text/x-wiki dpa4yctfp07o8aftosijc4h4zomq1jy 2437687 2437685 2026-07-16T12:13:46Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1267-1276].eqiad.wmnet 2437687 wikitext text/x-wiki 434fgwzu39crntguw1lo98ivg7rvhb9 2437688 2437687 2026-07-16T12:13:49Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1267-1276].eqiad.wmnet 2437688 wikitext text/x-wiki th3rgemyzlt6p4umccopbqosfyyxwov 2437689 2437688 2026-07-16T12:14:09Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1277-1286].eqiad.wmnet 2437689 wikitext text/x-wiki der1qfcfxh0ppy8o7xvzzp2n0o6q2hb 2437691 2437689 2026-07-16T12:18:42Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns6001.wikimedia.org 2437691 wikitext text/x-wiki llshyasebg1nujc6pdx6augtbm1lssc 2437692 2437691 2026-07-16T12:20:09Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1277-1286].eqiad.wmnet 2437692 wikitext text/x-wiki 64j56wpos2glh5awnye11s96ezoebc1 2437695 2437692 2026-07-16T12:28:03Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1277-1286].eqiad.wmnet 2437695 wikitext text/x-wiki 2sjijpm6dsnrk4kkh2zhxbcrsmmgj39 2437696 2437695 2026-07-16T12:28:06Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1277-1286].eqiad.wmnet 2437696 wikitext text/x-wiki q2u31yesx7ofij6cpfmdl3csglpsy9p 2437697 2437696 2026-07-16T12:28:26Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1287-1289,1291-1297].eqiad.wmnet 2437697 wikitext text/x-wiki 9lum7ck6sinh5sn100i3artv77bjcha 2437699 2437697 2026-07-16T12:34:38Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns6001.wikimedia.org 2437699 wikitext text/x-wiki h8gyc9rboy6779ttjzao9mpthfk5r99 2437700 2437699 2026-07-16T12:35:01Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1287-1289,1291-1297].eqiad.wmnet 2437700 wikitext text/x-wiki sffakpmj6azqbp7aonyo1lo169n450a 2437702 2437700 2026-07-16T12:39:38Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet 2437702 wikitext text/x-wiki nhsqfrxg68f17micoiidwfgzjvj5taw 2437703 2437702 2026-07-16T12:42:01Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet 2437703 wikitext text/x-wiki a8lgce1m9yv1wi48f7y90jglcnyrtpl 2437704 2437703 2026-07-16T12:43:10Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet 2437704 wikitext text/x-wiki ilrfckemqft7fkdk0pwx9hor71j55na 2437705 2437704 2026-07-16T12:43:29Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet 2437705 wikitext text/x-wiki gkgrk1z0s9gop6qo9sr554m9wglsl9z 2437706 2437705 2026-07-16T12:43:45Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet 2437706 wikitext text/x-wiki 5idcy76jzhnvceg572iguc0kz66ogig 2437707 2437706 2026-07-16T12:43:57Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet 2437707 wikitext text/x-wiki lcdxd43252x2cw5z3p2u3ar2u2moa28 2437708 2437707 2026-07-16T12:46:05Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1287-1289,1291-1297].eqiad.wmnet 2437708 wikitext text/x-wiki 1yqit7wb13r5cywksas71ozr52eui0k 2437709 2437708 2026-07-16T12:46:08Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1287-1289,1291-1297].eqiad.wmnet 2437709 wikitext text/x-wiki eikuu93mu4c7hklwjkdwzz2rcxms5pv 2437710 2437709 2026-07-16T12:46:32Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1298-1307].eqiad.wmnet 2437710 wikitext text/x-wiki 51tmwe0vh40nxe14zkghiud9wvv0611 2437711 2437710 2026-07-16T12:49:38Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns6002.wikimedia.org 2437711 wikitext text/x-wiki 5ry4bodc4lgvloew29bnj0p16sqmmxy 2437715 2437711 2026-07-16T12:52:37Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1298-1307].eqiad.wmnet 2437715 wikitext text/x-wiki ee7wb3k2u8z9mei0e5w9clwcmiyrj14 2437720 2437715 2026-07-16T12:59:42Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1262.eqiad.wmnet 2437720 wikitext text/x-wiki 2pttee6te9caz773vmceyj513y390ek 2437721 2437720 2026-07-16T12:59:44Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1262.eqiad.wmnet 2437721 wikitext text/x-wiki b4u9yqlepugwrdnndndrf5wxn3hoh02 2437722 2437721 2026-07-16T12:59:46Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1262.eqiad.wmnet 2437722 wikitext text/x-wiki te7hbx4ipkamy4ypcku3s2lypekd57v 2437725 2437722 2026-07-16T13:02:59Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1298-1307].eqiad.wmnet 2437725 wikitext text/x-wiki kp386ukpd2zkjhbnzotcz3chpy0xwg6 2437726 2437725 2026-07-16T13:03:01Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1298-1307].eqiad.wmnet 2437726 wikitext text/x-wiki q5ubfbqohsq9phwldsa1diqnda1r9gk 2437727 2437726 2026-07-16T13:03:22Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1308-1317].eqiad.wmnet 2437727 wikitext text/x-wiki 9shfm9h377csdq555m50fke3ikyj9ob 2437728 2437727 2026-07-16T13:03:33Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns6002.wikimedia.org 2437728 wikitext text/x-wiki aw4msup5j7040cyplq976p08r88am53 2437729 2437728 2026-07-16T13:05:48Z Stashbot 7414 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1311054|Enable Article Guidance on Polish Wikipedia (T432137)]] 2437729 wikitext text/x-wiki 07fap7jfyc57f25418jb2trx2tb3hob 2437730 2437729 2026-07-16T13:07:47Z Stashbot 7414 sbisson@deploy2003: sbisson: Backport for [[gerrit:1311054|Enable Article Guidance on Polish Wikipedia (T432137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2437730 wikitext text/x-wiki lwc7lg6tvijldeb1qta0j1oqosb8543 2437731 2437730 2026-07-16T13:08:55Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1308-1317].eqiad.wmnet 2437731 wikitext text/x-wiki jkbbwcdfhz62gc6wtxcol0s3kd3w0se 2437732 2437731 2026-07-16T13:09:31Z Stashbot 7414 sbisson@deploy2003: sbisson: Continuing with deployment 2437732 wikitext text/x-wiki o7vzl06gmwgmdkcanpmwq6bs4t17u5g 2437733 2437732 2026-07-16T13:10:37Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1263.eqiad.wmnet 2437733 wikitext text/x-wiki f94z7ga15zfkmf4scr1k2qukzfy89wr 2437734 2437733 2026-07-16T13:10:41Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1263.eqiad.wmnet 2437734 wikitext text/x-wiki foygu4fs78oky4gjf8ogn5uz4mkktwf 2437735 2437734 2026-07-16T13:11:12Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1263.eqiad.wmnet 2437735 wikitext text/x-wiki dsbd9j8iorc4xlww7wvrrfcp1mfcnx1 2437736 2437735 2026-07-16T13:11:31Z Stashbot 7414 cdobbins@cumin2003: conftool action : set/pooled=no; selector: name=dns7002.* 2437736 wikitext text/x-wiki o4xjg8louyzl09tuds3sy4359zrcws0 2437737 2437736 2026-07-16T13:11:40Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1263.eqiad.wmnet with OS trixie 2437737 wikitext text/x-wiki dwsdyijn6qajq3cg3kidkqm73mctm0c 2437738 2437737 2026-07-16T13:12:08Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1263 2437738 wikitext text/x-wiki k92v26jfkhhgw7x8p6te0gfpd8qbwp7 2437739 2437738 2026-07-16T13:12:13Z Stashbot 7414 cdobbins@cumin2003: conftool action : set/pooled=yes; selector: name=dns7002.*,service=authdns-update 2437739 wikitext text/x-wiki 4hm5avpyanoyp5yvvnpxwmnpn0q3npr 2437740 2437739 2026-07-16T13:12:24Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.netbox 2437740 wikitext text/x-wiki ox28cztcosvxocxmf6jpbk299gauxia 2437741 2437740 2026-07-16T13:13:07Z Stashbot 7414 cdobbins@dns1004: START - running authdns-update 2437741 wikitext text/x-wiki ndt1k5vr1pws31x5308zysnw8l81uf0 2437743 2437741 2026-07-16T13:13:51Z Stashbot 7414 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311054|Enable Article Guidance on Polish Wikipedia (T432137)]] (duration: 08m 03s) 2437743 wikitext text/x-wiki t4en7jb9bv9r248qv6d2bk00j4myitd 2437744 2437743 2026-07-16T13:14:58Z Stashbot 7414 cdobbins@dns1004: END - running authdns-update 2437744 wikitext text/x-wiki ntfqyv2fjro29kx1h5jqiwh53lguq4d 2437745 2437744 2026-07-16T13:16:13Z Stashbot 7414 cdobbins@cumin2003: conftool action : set/pooled=yes; selector: name=dns7002.* 2437745 wikitext text/x-wiki bx0co9t1nq0gi6y3asbr2w2ua64baqb 2437746 2437745 2026-07-16T13:18:33Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot begin reboot of dns7001.wikimedia.org 2437746 wikitext text/x-wiki p48m21l8y5t74u37d1rvekhtq8gzewr 2437747 2437746 2026-07-16T13:18:59Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1308-1317].eqiad.wmnet 2437747 wikitext text/x-wiki 779mxnapji8bs2ez5oouo1hwlfe8ke9 2437748 2437747 2026-07-16T13:19:02Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1308-1317].eqiad.wmnet 2437748 wikitext text/x-wiki a8ykb61b3c08fbkkksoxiv97gen042y 2437749 2437748 2026-07-16T13:19:22Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1318-1327].eqiad.wmnet 2437749 wikitext text/x-wiki kqnm35j2gih3biy0tn8prup3kgrq5ye 2437750 2437749 2026-07-16T13:19:25Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet 2437750 wikitext text/x-wiki 16mf5xajtrzgyf3mgsy9bzhld97lwos 2437751 2437750 2026-07-16T13:20:15Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1263 - kamila@cumin1003" 2437751 wikitext text/x-wiki fh15i1gg98op23aaolxsabg6cnluf3h 2437752 2437751 2026-07-16T13:20:20Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1263 - kamila@cumin1003" 2437752 wikitext text/x-wiki f876xmvlwearpaw7b1ovzpu6yetuoyh 2437753 2437752 2026-07-16T13:20:21Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2437753 wikitext text/x-wiki kc31pzdo8dj44a2m19yjsqp03mffhay 2437754 2437753 2026-07-16T13:20:23Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1263.eqiad.wmnet 73.32.64.10.in-addr.arpa 3.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437754 wikitext text/x-wiki b2uupjt1qkwn7z1sgbmwmo88dm3i20g 2437755 2437754 2026-07-16T13:20:41Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1263.eqiad.wmnet 73.32.64.10.in-addr.arpa 3.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437755 wikitext text/x-wiki carlgdips05b44iu1lfj3kyoxr1whu9 2437756 2437755 2026-07-16T13:21:32Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1263.eqiad.wmnet 73.32.64.10.in-addr.arpa 3.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437756 wikitext text/x-wiki 5xqzv7i2v79crl71ufu5hdaqiddfj7j 2437757 2437756 2026-07-16T13:21:52Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1263.eqiad.wmnet 73.32.64.10.in-addr.arpa 3.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437757 wikitext text/x-wiki c6000h0urb2k4hwuae542rhuobogx8z 2437758 2437757 2026-07-16T13:22:03Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet 2437758 wikitext text/x-wiki 36hxgwtf6dqj3g2cclcaeyi4jh64mzj 2437759 2437758 2026-07-16T13:23:54Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1263.eqiad.wmnet 73.32.64.10.in-addr.arpa 3.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437759 wikitext text/x-wiki nfcoju5mzzhovonql5gh1li3w8tb7nc 2437760 2437759 2026-07-16T13:23:57Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1263.eqiad.wmnet 73.32.64.10.in-addr.arpa 3.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437760 wikitext text/x-wiki n8vz934ykydwvksro5njh69mx30d4l0 2437761 2437760 2026-07-16T13:23:59Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1263 2437761 wikitext text/x-wiki odyj19vc2dl6pf8l5vvcyrez4oqlis6 2437762 2437761 2026-07-16T13:24:39Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1263 2437762 wikitext text/x-wiki ewgaitagj8ds5rzaedfkxrl0kbnjmff 2437763 2437762 2026-07-16T13:24:41Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1263 2437763 wikitext text/x-wiki 7ihvltcvf3mx3nttiz18hz0w3bd71hu 2437764 2437763 2026-07-16T13:24:47Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1318-1327].eqiad.wmnet 2437764 wikitext text/x-wiki r66gmrcfcgbmzj5sb9ktlnne5dolfwi 2437765 2437764 2026-07-16T13:24:59Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet 2437765 wikitext text/x-wiki cmf0g1lrv7kbrjd9avimivngc7ua7kw 2437766 2437765 2026-07-16T13:25:26Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2437766 wikitext text/x-wiki cv5iozatxkrgrxl0yx8pph6b6pceo1m 2437767 2437766 2026-07-16T13:25:32Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet 2437767 wikitext text/x-wiki 6ksgm8435y1zkwrsyzostmx8evvt8w1 2437768 2437767 2026-07-16T13:25:42Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet 2437768 wikitext text/x-wiki 8n6qxbh8f83ad4ygnd783mji3vldgkx 2437769 2437768 2026-07-16T13:25:52Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet 2437769 wikitext text/x-wiki jruyrqeignhsfo8dygpuso2471z959t 2437770 2437769 2026-07-16T13:27:17Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2437770 wikitext text/x-wiki dcm2g8xfy2p4jaezvl3n6qir9yy7u6o 2437772 2437770 2026-07-16T13:33:08Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.dns.roll-reboot finished rebooting dns7001.wikimedia.org 2437772 wikitext text/x-wiki hqc72fx33iqxk60xlid8iaequnb3w1u 2437773 2437772 2026-07-16T13:33:10Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:dnsbox and (A:eqsin or A:drmrs or A:magru) and not (P{dns5003*} or P{dns7002*}) and (A:dnsbox) 2437773 wikitext text/x-wiki 6lol2j6sgel9rhrdv10n0qmo2i2lxg6 2437774 2437773 2026-07-16T13:34:54Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1318-1327].eqiad.wmnet 2437774 wikitext text/x-wiki rgb81p7cch3w6jq7wqwtvltezjd21mq 2437775 2437774 2026-07-16T13:34:57Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1318-1327].eqiad.wmnet 2437775 wikitext text/x-wiki nnd1kxc9mu2yftb6517aosy83prmjx4 2437776 2437775 2026-07-16T13:35:17Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1328-1337].eqiad.wmnet 2437776 wikitext text/x-wiki mq9cyzpsjj8gxsxonymjgbzi58dz4n1 2437777 2437776 2026-07-16T13:40:13Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad 2437777 wikitext text/x-wiki hlli03ysn6iimo87ylxnjtyozrwnp66 2437778 2437777 2026-07-16T13:40:58Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1328-1337].eqiad.wmnet 2437778 wikitext text/x-wiki hd1au949zn0tkadotdh1eh2yd2vq6xo 2437781 2437778 2026-07-16T13:45:52Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1263.eqiad.wmnet with reason: host reimage 2437781 wikitext text/x-wiki qo60etb38zo9zdoftidaynnqscyc40j 2437782 2437781 2026-07-16T13:49:21Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1328-1337].eqiad.wmnet 2437782 wikitext text/x-wiki 53o5d4adxw3a836uckwciic2qc75gut 2437783 2437782 2026-07-16T13:49:24Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1328-1337].eqiad.wmnet 2437783 wikitext text/x-wiki f2l5pfp0qiwl143w1v6fcsaqy5750h3 2437784 2437783 2026-07-16T13:49:33Z Stashbot 7414 sfaci@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2437784 wikitext text/x-wiki c9p2vhezzlzqb1h004pia7ic3b3g6ep 2437785 2437784 2026-07-16T13:49:38Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet 2437785 wikitext text/x-wiki grcv7gomogzeyexaqbqf9wenfo50yo9 2437786 2437785 2026-07-16T13:49:43Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1338-1347].eqiad.wmnet 2437786 wikitext text/x-wiki geh1brihpsnnlz5lyc95qrzao0esqrs 2437787 2437786 2026-07-16T13:50:15Z Stashbot 7414 sfaci@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2437787 wikitext text/x-wiki c42tbm79faemll9r9xigks9ht81foab 2437788 2437787 2026-07-16T13:50:25Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1263.eqiad.wmnet with reason: host reimage 2437788 wikitext text/x-wiki lyly8w6i57ig853we85jpc156vwnku9 2437790 2437788 2026-07-16T13:54:46Z Stashbot 7414 sfaci@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2437790 wikitext text/x-wiki 8pmu4xihsibkt7vo967b5r0xk0r73es 2437791 2437790 2026-07-16T13:54:55Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet 2437791 wikitext text/x-wiki dxkjzrczhal3k8lj4w16zh2jr5g06p5 2437792 2437791 2026-07-16T13:54:58Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet 2437792 wikitext text/x-wiki mtp332chnot2nwx3s5lnwlemioxmrx4 2437794 2437792 2026-07-16T13:55:49Z Stashbot 7414 sfaci@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2437794 wikitext text/x-wiki gtj6oi5q0rnhoqtvllpjsfzk2zvm2qh 2437796 2437794 2026-07-16T13:56:34Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-codfw 2437796 wikitext text/x-wiki bekbdc0lh7d6ute7b2mwpsg4qs8s50w 2437797 2437796 2026-07-16T13:59:18Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet 2437797 wikitext text/x-wiki 2blkzv0qj9ry63f9fm8bxn7b6d0rffq 2437798 2437797 2026-07-16T13:59:35Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1338-1347].eqiad.wmnet 2437798 wikitext text/x-wiki b16d3gjocfg4rxvhn98mxjmwxd2iag6 2437799 2437798 2026-07-16T14:00:02Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet 2437799 wikitext text/x-wiki df062pod80ix0nfmydpiusqgcb96w4t 2437800 2437799 2026-07-16T14:02:07Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet 2437800 wikitext text/x-wiki abppitm79b464gvmd61gmgrucoxjmic 2437801 2437800 2026-07-16T14:02:10Z Stashbot 7414 topranks: beginning depools for lsw1-b6-codfw maintenance T430922 2437801 wikitext text/x-wiki 6ungxi4jy0afltehd5ezyqo5dhjck13 2437802 2437801 2026-07-16T14:05:35Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet 2437802 wikitext text/x-wiki 3j5q2rilr1r36zi24gdf9mj5dhruywe 2437803 2437802 2026-07-16T14:06:55Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1338-1347].eqiad.wmnet 2437803 wikitext text/x-wiki 9ndzqp1769uflgesv98v4liodqng9fe 2437804 2437803 2026-07-16T14:06:58Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1338-1347].eqiad.wmnet 2437804 wikitext text/x-wiki fujmbckfvzjpj498i9sh8w5tf7wjw4v 2437805 2437804 2026-07-16T14:07:10Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet 2437805 wikitext text/x-wiki cmhuffvoch62fj0lqu82qxc4sg392rr 2437806 2437805 2026-07-16T14:07:17Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1348-1357].eqiad.wmnet 2437806 wikitext text/x-wiki eqqtbbs8wosh4ekfjwcg8xrky3rnspz 2437807 2437806 2026-07-16T14:07:19Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet 2437807 wikitext text/x-wiki 95szix0mz2oc5erfi758htwocd3oifo 2437808 2437807 2026-07-16T14:07:32Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster 2437808 wikitext text/x-wiki 0872efqb4pzl8q9qhm8wfpxswy2enrz 2437809 2437808 2026-07-16T14:07:57Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet 2437809 wikitext text/x-wiki 6258gozjzwla5f5h6qa57z7u7182e4s 2437810 2437809 2026-07-16T14:08:03Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2093-2094,2102-2106,2279-2283].codfw.wmnet 2437810 wikitext text/x-wiki 6t9z5wdv7ufi38rbay9bghl124kob37 2437811 2437810 2026-07-16T14:11:38Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.depool depool db2161: switch maintenance codfw rack b6 2437811 wikitext text/x-wiki ponep44j2ug7r5f7kur0yfac0r90cd1 2437813 2437811 2026-07-16T14:11:58Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: switch maintenance codfw rack b6 2437813 wikitext text/x-wiki 94gy4ng3ikdcirjb83eb175yij9zh98 2437814 2437813 2026-07-16T14:12:03Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.depool depool db2162: switch maintenance codfw rack b6 2437814 wikitext text/x-wiki gu105obvy0rt8ykev9w9psrlkhn581y 2437815 2437814 2026-07-16T14:12:19Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1263.eqiad.wmnet with OS trixie 2437815 wikitext text/x-wiki qpcwo3jeavqt235fyu8lwv2x98d5p9p 2437816 2437815 2026-07-16T14:12:34Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2162: switch maintenance codfw rack b6 2437816 wikitext text/x-wiki 1cd7m6wsi947i38c6cok91cfr3zn224 2437817 2437816 2026-07-16T14:12:40Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.depool depool db2251: switch maintenance codfw rack b6 2437817 wikitext text/x-wiki ct3nws1t41ij4jhianue40nb3lq5hq0 2437818 2437817 2026-07-16T14:12:41Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.parsercache 2437818 wikitext text/x-wiki fkdp76shwm2nlj7gy2p02i9cc0swfvs 2437819 2437818 2026-07-16T14:12:49Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2437819 wikitext text/x-wiki ac9v6qqp2c1h5ju01fa1p7693w5mwjm 2437820 2437819 2026-07-16T14:12:50Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2251: switch maintenance codfw rack b6 2437820 wikitext text/x-wiki 5dost9fg584rv8fvq38zmcgcxkerc8b 2437821 2437820 2026-07-16T14:12:55Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.depool depool pc2022: switch maintenance codfw rack b6 2437821 wikitext text/x-wiki cv0i04tbc9fjq0nz7au0au3295y9hkf 2437822 2437821 2026-07-16T14:12:56Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.parsercache 2437822 wikitext text/x-wiki n6trsvxb23y1jkg4u5a567hjks1lpie 2437823 2437822 2026-07-16T14:12:58Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1348-1357].eqiad.wmnet 2437823 wikitext text/x-wiki l2zzplvms56k6hnlri9gju3lbxuuehk 2437824 2437823 2026-07-16T14:13:05Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2437824 wikitext text/x-wiki 5ymn99gvwgrfeqq8gnihoyc8xgzrhzt 2437825 2437824 2026-07-16T14:13:06Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2022: switch maintenance codfw rack b6 2437825 wikitext text/x-wiki o2glafe8zp0kb7h0359iqxmlwasudcs 2437826 2437825 2026-07-16T14:13:10Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2006.codfw.wmnet 2437826 wikitext text/x-wiki kv6gkkvscy1ftgkuvyrymkeugp9mrle 2437827 2437826 2026-07-16T14:13:11Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2006.codfw.wmnet 2437827 wikitext text/x-wiki lay5o9s2ixhmm6b5l0nl5ptnjczyis6 2437828 2437827 2026-07-16T14:14:44Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2093-2094,2102-2106,2279-2283].codfw.wmnet 2437828 wikitext text/x-wiki 6y09em9xx8jfks8126pz51t0ctxilmv 2437831 2437828 2026-07-16T14:19:10Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: lsw1-b6-codfw JunOS upgrade 2437831 wikitext text/x-wiki bha4tg537xlgatge6po3m5yc9mbr0nb 2437832 2437831 2026-07-16T14:19:43Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet 2437832 wikitext text/x-wiki smvt5ody93kgh7pc5jd1rz4e85z020b 2437833 2437832 2026-07-16T14:20:10Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet 2437833 wikitext text/x-wiki 4t2rl58m2blf3h512ji0e06zbuk8z30 2437834 2437833 2026-07-16T14:20:29Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b6-codfw,lsw1-b6-codfw IPv6,lsw1-b6-codfw.mgmt,ssw1-a[1,8]-codfw with reason: lsw1-b6-codfw JunOS upgrade 2437834 wikitext text/x-wiki r1lrsq96rn6ialxsedy0padqib4f9dv 2437835 2437834 2026-07-16T14:21:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1001.eqiad.wmnet with OS bookworm 2437835 wikitext text/x-wiki haw8xeq290gcfmsfrirt8k8ul7sxcqc 2437836 2437835 2026-07-16T14:21:12Z Stashbot 7414 kamila@deploy2003: Started scap sync-world: Test deployment to check rsync is working - T432108 2437836 wikitext text/x-wiki ipcu73ghzlkaalhuilne1gs2v1kd0f9 2437837 2437836 2026-07-16T14:21:25Z Stashbot 7414 topranks: reboot lsw1-b6-codfw to upgrade JunOS T430922 2437837 wikitext text/x-wiki 4ctk6fu5dqb02ag6txdymu0hw9g43cv 2437838 2437837 2026-07-16T14:21:46Z Stashbot 7414 kamila@deploy2003: sync-world aborted: Test deployment to check rsync is working - T432108 (duration: 00m 36s) 2437838 wikitext text/x-wiki 7xcapu70h8zyaz72k17dzhmx8lice8e 2437839 2437838 2026-07-16T14:22:31Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1348-1357].eqiad.wmnet 2437839 wikitext text/x-wiki p5juo5inr87wfwlz8e545o5ugwnc6qh 2437840 2437839 2026-07-16T14:22:34Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1348-1357].eqiad.wmnet 2437840 wikitext text/x-wiki d7pjlx1bj0h72exlh4ji3kog3shoiaq 2437841 2437840 2026-07-16T14:22:52Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1358-1366].eqiad.wmnet 2437841 wikitext text/x-wiki 47k2pkbyl19fpnjsm6ihdyzzoht7k1c 2437842 2437841 2026-07-16T14:25:00Z Stashbot 7414 kamila@deploy2003: Started scap sync-world: Test deployment to check rsync is working - T432108 2437842 wikitext text/x-wiki 51euuo9866q4up251gguyquc5ygu1is 2437843 2437842 2026-07-16T14:25:35Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet 2437843 wikitext text/x-wiki eb3uxr0136k0kg3uv43ycuf0tv636al 2437844 2437843 2026-07-16T14:25:38Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet 2437844 wikitext text/x-wiki 2oiojkaq5do8ltyyjo971gavefjhhqk 2437845 2437844 2026-07-16T14:26:03Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet 2437845 wikitext text/x-wiki m9ecyotjzpif8t4vwko01lboio80q7b 2437846 2437845 2026-07-16T14:26:06Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet 2437846 wikitext text/x-wiki 4omyorcmmpsvjuh0stjndi48n7k2y45 2437848 2437846 2026-07-16T14:27:44Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-flink-eqiad cluster: Roll restart of jvm daemons. 2437848 wikitext text/x-wiki kdd580z6t9gjhelfh8wrlbdyxj1hmle 2437849 2437848 2026-07-16T14:27:58Z Stashbot 7414 kamila@deploy2003: Finished scap sync-world: Test deployment to check rsync is working - T432108 (duration: 02m 57s) 2437849 wikitext text/x-wiki n0zr6o2fr1io44h4h5n2c4w5k7dzs07 2437850 2437849 2026-07-16T14:28:00Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1358-1366].eqiad.wmnet 2437850 wikitext text/x-wiki 0ngbksvhwgcrngdcr8sr0t84qag016u 2437851 2437850 2026-07-16T14:29:17Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2006.codfw.wmnet 2437851 wikitext text/x-wiki 0qh86cqrfu5dfhhq3jfvfr3xq0e8id8 2437852 2437851 2026-07-16T14:31:24Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet 2437852 wikitext text/x-wiki oye8h7d0skwveagkfj7m60glmzmjnmw 2437853 2437852 2026-07-16T14:32:03Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet 2437853 wikitext text/x-wiki rv1vas68in7em9njv3wpxfcdnm7ln8v 2437854 2437853 2026-07-16T14:34:06Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-flink-eqiad cluster: Roll restart of jvm daemons. 2437854 wikitext text/x-wiki 6w0k8h7rke55wo30l7a5shfx6uobgzs 2437855 2437854 2026-07-16T14:34:25Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2006.codfw.wmnet 2437855 wikitext text/x-wiki dyje9nf7tjxeglqnky8zigeu34juwf8 2437856 2437855 2026-07-16T14:34:54Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1358-1366].eqiad.wmnet 2437856 wikitext text/x-wiki ojmaakmv4d1xwvf9bgg202i0v7jmrz5 2437857 2437856 2026-07-16T14:34:57Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1358-1366].eqiad.wmnet 2437857 wikitext text/x-wiki 8vsynmpp9dcop7cv480n2wojm7i8gns 2437858 2437857 2026-07-16T14:35:13Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1367-1375].eqiad.wmnet 2437858 wikitext text/x-wiki i9h15pbanvw7iafahxsxns6x72fwhok 2437860 2437858 2026-07-16T14:40:21Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1367-1375].eqiad.wmnet 2437860 wikitext text/x-wiki 5w3uszy2eiw1a493aw6w48wdr31kea8 2437861 2437860 2026-07-16T14:40:42Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet 2437861 wikitext text/x-wiki cgk180b0p7cramh1ko7tjfo7o66hqt7 2437862 2437861 2026-07-16T14:42:02Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet 2437862 wikitext text/x-wiki kpjvw9yiti53flwa2mnpisky4egklrh 2437863 2437862 2026-07-16T14:44:50Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1263.eqiad.wmnet 2437863 wikitext text/x-wiki dv30jyw86box5acxvnt6hdyysk75yrn 2437864 2437863 2026-07-16T14:44:51Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1263.eqiad.wmnet 2437864 wikitext text/x-wiki lmo9roz27i9ulpyn4x6van5974tw72w 2437865 2437864 2026-07-16T14:44:54Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1263.eqiad.wmnet 2437865 wikitext text/x-wiki etwu3okn59ur8p3pxk2180o3oih3q4z 2437866 2437865 2026-07-16T14:47:23Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet 2437866 wikitext text/x-wiki nylzfwineztifdutnrml07y09uvwlfl 2437867 2437866 2026-07-16T14:47:40Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1367-1375].eqiad.wmnet 2437867 wikitext text/x-wiki mfo4emc27xadbm7jnt6s2gotr6dsydj 2437868 2437867 2026-07-16T14:47:43Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1367-1375].eqiad.wmnet 2437868 wikitext text/x-wiki 9qdozl28b9u7llco74v9ywywurx0bnu 2437869 2437868 2026-07-16T14:47:48Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1264.eqiad.wmnet 2437869 wikitext text/x-wiki gqbf1y3vd7ml8441op3hsmqp30ldry7 2437870 2437869 2026-07-16T14:47:53Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1264.eqiad.wmnet 2437870 wikitext text/x-wiki ied0exvf33eboamnlq0cccde9k69458 2437871 2437870 2026-07-16T14:47:57Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1265.eqiad.wmnet 2437871 wikitext text/x-wiki f5buc1id15wukx3hdih49efys68xb9z 2437872 2437871 2026-07-16T14:48:00Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1265.eqiad.wmnet 2437872 wikitext text/x-wiki nlsiixa44sm31p2h3udlig467eb1020 2437873 2437872 2026-07-16T14:48:02Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1376-1384].eqiad.wmnet 2437873 wikitext text/x-wiki 5n8adaxfal4ehemts78n85njo1xn592 2437875 2437873 2026-07-16T14:48:19Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1001.eqiad.wmnet with reason: host reimage 2437875 wikitext text/x-wiki 22rtyuvcqjdfbmxbl0d6y7tssnss6v0 2437876 2437875 2026-07-16T14:48:25Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1264.eqiad.wmnet 2437876 wikitext text/x-wiki s2gmevi1zzffc0j6ms0rtzndb5jt3tx 2437877 2437876 2026-07-16T14:49:06Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet 2437877 wikitext text/x-wiki 0z4s0u67bidgp3v68d73ymtd39s872b 2437878 2437877 2026-07-16T14:49:08Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1265.eqiad.wmnet 2437878 wikitext text/x-wiki soweddh4ed0gn3c0ysk698gco0zzteh 2437879 2437878 2026-07-16T14:49:30Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet 2437879 wikitext text/x-wiki a9zik9g6xeylcfxg90wuq610mlayj2b 2437880 2437879 2026-07-16T14:49:40Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2093-2094,2102-2106,2279-2283].codfw.wmnet 2437880 wikitext text/x-wiki otl6deb13qu651xajotcbx44jegeql4 2437881 2437880 2026-07-16T14:49:42Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1265.eqiad.wmnet with OS trixie 2437881 wikitext text/x-wiki 549wow8iy4yt57x6i7au3p5iswjl0rt 2437882 2437881 2026-07-16T14:49:50Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1264.eqiad.wmnet with OS trixie 2437882 wikitext text/x-wiki levyalczqhmts13pbvyhzb3cfphfc8v 2437883 2437882 2026-07-16T14:49:56Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2093-2094,2102-2106,2279-2283].codfw.wmnet 2437883 wikitext text/x-wiki kddoleyxen6c0m2udexprxash5akmq7 2437884 2437883 2026-07-16T14:50:11Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1265 2437884 wikitext text/x-wiki rgb1hkjmzqq4sb61rn0kthubtmr639y 2437885 2437884 2026-07-16T14:50:19Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet 2437885 wikitext text/x-wiki 4ca3a93rt7el9xblthf2xj82hb0n6n3 2437886 2437885 2026-07-16T14:50:32Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2093-2094,2102-2106,2279-2283].codfw.wmnet 2437886 wikitext text/x-wiki d59poldcn90h7fdo9y1zud6zdz35odl 2437887 2437886 2026-07-16T14:50:38Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.pool pool db2161: switch maintenance completed codfw rack b6 2437887 wikitext text/x-wiki nlsbsu3oau9owzg4x1xmfwd5e4emqle 2437888 2437887 2026-07-16T14:50:40Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2093-2094,2102-2106,2279-2283].codfw.wmnet 2437888 wikitext text/x-wiki 96pu68rv9jueyimty13ilwmjwu5mzy5 2437889 2437888 2026-07-16T14:50:53Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2006.codfw.wmnet 2437889 wikitext text/x-wiki 828swx6jwqjsn6olinr8unrvyrokc01 2437890 2437889 2026-07-16T14:50:54Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2006.codfw.wmnet 2437890 wikitext text/x-wiki 4dfen9qsv22w1ba2qheksx9iwcj5cgs 2437891 2437890 2026-07-16T14:51:33Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.netbox 2437891 wikitext text/x-wiki jdyx8jicaurm99z1s7h4pxj8u9jmxi6 2437892 2437891 2026-07-16T14:53:25Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1001.eqiad.wmnet with reason: host reimage 2437892 wikitext text/x-wiki 7gv0k0otyow4iaqv43acpswyec8lyhb 2437893 2437892 2026-07-16T14:53:34Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1376-1384].eqiad.wmnet 2437893 wikitext text/x-wiki g8iwwora3bn0vxkzd209gjckp1l7wk8 2437894 2437893 2026-07-16T14:53:42Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet 2437894 wikitext text/x-wiki 5eb8lym1u1k4rglrehbr0oeie46p65w 2437895 2437894 2026-07-16T14:56:52Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1265 - kamila@cumin1003" 2437895 wikitext text/x-wiki 27xqeho8vnwh6f2i5zqofgr19wy9v4c 2437896 2437895 2026-07-16T14:56:56Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1265 - kamila@cumin1003" 2437896 wikitext text/x-wiki t3fm38oouc2n77aud6uw90lknqy55q4 2437897 2437896 2026-07-16T14:56:58Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2437897 wikitext text/x-wiki nftc9ta3n8ttnb5k2ch2j3cn0bre113 2437898 2437897 2026-07-16T14:56:59Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1265.eqiad.wmnet 75.32.64.10.in-addr.arpa 5.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437898 wikitext text/x-wiki jwcfnl2dttigncimts56t6h28yw1vt6 2437899 2437898 2026-07-16T14:57:00Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1265.eqiad.wmnet 75.32.64.10.in-addr.arpa 5.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437899 wikitext text/x-wiki 6tapyt7sl00mnlk04cp2ykydgr4evky 2437900 2437899 2026-07-16T14:57:01Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1265 2437900 wikitext text/x-wiki ax5b5dt8c1wjnnr537kxle4u57259nj 2437901 2437900 2026-07-16T14:57:13Z Stashbot 7414 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1310596|Set $wgMathInternalRestbaseURL explicitly (T349582)]] 2437901 wikitext text/x-wiki ci3j1tex3a3fv1h4xqnikgbwsnbxqk3 2437902 2437901 2026-07-16T14:57:28Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1265 2437902 wikitext text/x-wiki qu104yjyxinx5ourzo410tpyb4og8mp 2437903 2437902 2026-07-16T14:57:29Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1265 2437903 wikitext text/x-wiki f613hkdmed4hihw3ta2um12xry3n051 2437904 2437903 2026-07-16T14:57:51Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1264 2437904 wikitext text/x-wiki 8nlyv5tzo34hrukf42iu3u9ya5braci 2437905 2437904 2026-07-16T14:58:07Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.netbox 2437905 wikitext text/x-wiki cmftn6rjpkbfzh3guz6ih3r2albxu4h 2437906 2437905 2026-07-16T14:58:29Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1266.eqiad.wmnet 2437906 wikitext text/x-wiki no989y22y9g4xblqsdeaidaxziwvt23 2437907 2437906 2026-07-16T14:58:32Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1266.eqiad.wmnet 2437907 wikitext text/x-wiki 280gqf3zovp3pvj8d6cqyj0ymnn21tv 2437908 2437907 2026-07-16T14:59:05Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1266.eqiad.wmnet 2437908 wikitext text/x-wiki m3950dre0z78tis3a3berihs4diderz 2437909 2437908 2026-07-16T14:59:50Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet 2437909 wikitext text/x-wiki 9j9m7hu04uq8g788fmyoaqoshmz0hb7 2437910 2437909 2026-07-16T15:00:28Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1376-1384].eqiad.wmnet 2437910 wikitext text/x-wiki eppe9zj42293dzoukid18hnp6xnht1h 2437911 2437910 2026-07-16T15:00:30Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1376-1384].eqiad.wmnet 2437911 wikitext text/x-wiki ojlk2z7avrlaaqu2ofdkut2w20blxql 2437912 2437911 2026-07-16T15:00:34Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp1001.eqiad.wmnet 2437912 wikitext text/x-wiki 5r41fu7kcdg73uw8d7vh8asounarfc2 2437913 2437912 2026-07-16T15:01:05Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp1001.eqiad.wmnet 2437913 wikitext text/x-wiki i9fn66drp351m349v1eax9op5mdy7n4 2437914 2437913 2026-07-16T15:02:02Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp1001.eqiad.wmnet 2437914 wikitext text/x-wiki 3658wbilubupe0waq918qgb96jplvn4 2437915 2437914 2026-07-16T15:02:03Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp1001.eqiad.wmnet 2437915 wikitext text/x-wiki 2qk46sabb89c404pf0j5zpjn6iilsyu 2437916 2437915 2026-07-16T15:02:05Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-eqiad 2437916 wikitext text/x-wiki 5v5hv9oypev8xrqsn0jm4siw0vo2ab5 2437918 2437916 2026-07-16T15:03:44Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1264 - kamila@cumin1003" 2437918 wikitext text/x-wiki 2yf3vqq76z0qpeoj1cwyln87k36s7sz 2437919 2437918 2026-07-16T15:03:49Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1264 - kamila@cumin1003" 2437919 wikitext text/x-wiki rae3q9extz3hcdo0iu11pleio1g8bzm 2437920 2437919 2026-07-16T15:03:50Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2437920 wikitext text/x-wiki armh9uaf6g7nxquxd52844rte7b67rl 2437921 2437920 2026-07-16T15:03:51Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1264.eqiad.wmnet 74.32.64.10.in-addr.arpa 4.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437921 wikitext text/x-wiki cg57pv0t77uwucegmzubb5ec75ulm3m 2437922 2437921 2026-07-16T15:03:52Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1264.eqiad.wmnet 74.32.64.10.in-addr.arpa 4.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437922 wikitext text/x-wiki gntxjxtsbl0kil12plfnco7zhychmwa 2437923 2437922 2026-07-16T15:03:54Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1264 2437923 wikitext text/x-wiki rp8tgz3uz43x58qif3duvmkj7jkxytd 2437924 2437923 2026-07-16T15:04:10Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1266.eqiad.wmnet with OS trixie 2437924 wikitext text/x-wiki 2j5lkru0ddbbu12psi536urlclt275i 2437925 2437924 2026-07-16T15:04:13Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1264 2437925 wikitext text/x-wiki b2dvxyvtqm5v7thnqa6kb296uxcula3 2437926 2437925 2026-07-16T15:04:14Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1264 2437926 wikitext text/x-wiki 76cmif5jfrzysm0u7ljteitx49vdyx9 2437927 2437926 2026-07-16T15:04:39Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1266 2437927 wikitext text/x-wiki im6dhowi89vlrcqpjuw6h3elakp4fdb 2437928 2437927 2026-07-16T15:07:59Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.netbox 2437928 wikitext text/x-wiki 8w67oya7jduymn8z714kq1facjbuyvb 2437930 2437928 2026-07-16T15:15:06Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1266 - kamila@cumin1003" 2437930 wikitext text/x-wiki t345jhg6ha6gwg4jqwa56ki5mazo935 2437931 2437930 2026-07-16T15:15:11Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1266 - kamila@cumin1003" 2437931 wikitext text/x-wiki q64exx1rg8fytdial2pnm522bpcbm5j 2437932 2437931 2026-07-16T15:15:12Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2437932 wikitext text/x-wiki t9bw9hu5k3jfes5g3wddfpcolgd77yh 2437933 2437932 2026-07-16T15:15:13Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1266.eqiad.wmnet 76.32.64.10.in-addr.arpa 6.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437933 wikitext text/x-wiki evtfbad0xngbvkbzwxznbrmxsor1t20 2437934 2437933 2026-07-16T15:15:15Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1266.eqiad.wmnet 76.32.64.10.in-addr.arpa 6.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437934 wikitext text/x-wiki 0kvn7xpg4je7i3kxunu3ygqpn3jeb42 2437935 2437934 2026-07-16T15:15:16Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1266 2437935 wikitext text/x-wiki 1asbcf1zibgu0i39eouxbhxkyi9y74u 2437936 2437935 2026-07-16T15:16:24Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs1001.eqiad.wmnet with OS bookworm 2437936 wikitext text/x-wiki 5s7f5m27x50h4gx3ewt4gxnubprmnbv 2437937 2437936 2026-07-16T15:16:47Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1265.eqiad.wmnet with reason: host reimage 2437937 wikitext text/x-wiki qeyxc1de8sufc1i6x48es5lh0t0n1t6 2437938 2437937 2026-07-16T15:20:11Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet 2437938 wikitext text/x-wiki 3e5fadqmf104ibp69e0dmkj97xsqiyl 2437939 2437938 2026-07-16T15:21:55Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1110.eqiad.wmnet 2437939 wikitext text/x-wiki ac32ctwnshi50niq8ysomaza1zm4iz0 2437940 2437939 2026-07-16T15:24:00Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1266 2437940 wikitext text/x-wiki gfv7bqykg8rkcprfqcebai463mv6wfv 2437941 2437940 2026-07-16T15:24:01Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1266 2437941 wikitext text/x-wiki 442yicqpcogt0fp8da27320yjghj6ha 2437942 2437941 2026-07-16T15:25:05Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1265.eqiad.wmnet with reason: host reimage 2437942 wikitext text/x-wiki g6xtmdpibt8t0c1dzm49oki5w78l10y 2437943 2437942 2026-07-16T15:27:08Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet 2437943 wikitext text/x-wiki 43snvema2fq1k22i07hofezageg91q0 2437944 2437943 2026-07-16T15:31:07Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet 2437944 wikitext text/x-wiki qdnrb8zpwztiujx2adoiiz9ff86wfqg 2437945 2437944 2026-07-16T15:31:44Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet 2437945 wikitext text/x-wiki qfpv1a94r0oj30eq8tca26nmc7q2j6a 2437946 2437945 2026-07-16T15:32:16Z Stashbot 7414 jiji@deploy2003: Started scap sync-world: Backport for [[gerrit:1311464|ProductionServices: reboot poolcounter1006 (#1/4) (T431705)]] 2437946 wikitext text/x-wiki 3lrtcyuc4ap9nrtenhz5qlog34wbr72 2437947 2437946 2026-07-16T15:32:52Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet 2437947 wikitext text/x-wiki 5qlc79hd78fq0jf5ebo27cip1jqlegu 2437948 2437947 2026-07-16T15:34:13Z Stashbot 7414 jiji@deploy2003: jiji: Backport for [[gerrit:1311464|ProductionServices: reboot poolcounter1006 (#1/4) (T431705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2437948 wikitext text/x-wiki abx5ohm2aqmzmylnbb8q0mejzllqa2c 2437949 2437948 2026-07-16T15:36:13Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: switch maintenance completed codfw rack b6 2437949 wikitext text/x-wiki m3qu31khta3yb6zrv7ym34ddutugu02 2437950 2437949 2026-07-16T15:36:18Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.pool pool db2162: switch maintenance completed codfw rack b6 2437950 wikitext text/x-wiki ou3gkfl7qu6ut4zrawdnksvcqk3i353 2437951 2437950 2026-07-16T15:37:42Z Stashbot 7414 jiji@deploy2003: jiji: Continuing with deployment 2437951 wikitext text/x-wiki 74by83jk74vmrfmt2gwilqylg5jp1v4 2437952 2437951 2026-07-16T15:42:02Z Stashbot 7414 jiji@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311464|ProductionServices: reboot poolcounter1006 (#1/4) (T431705)]] (duration: 09m 46s) 2437952 wikitext text/x-wiki o328o36f44wp1khtrwtlo6s3kfha8q8 2437953 2437952 2026-07-16T15:44:00Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet 2437953 wikitext text/x-wiki rf79rf73lzjpu52r8bk939n4tfqrp03 2437954 2437953 2026-07-16T15:44:53Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1266.eqiad.wmnet with reason: host reimage 2437954 wikitext text/x-wiki 9ksr0ku0gbgyruskoiak8a53n3nnm3x 2437955 2437954 2026-07-16T15:45:34Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1265.eqiad.wmnet with OS trixie 2437955 wikitext text/x-wiki aqw81dly0uhhje4ee7hum622b8ro6vv 2437956 2437955 2026-07-16T15:47:45Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet 2437956 wikitext text/x-wiki t8ow2qumwp9peuq49rios0auxjzq772 2437959 2437956 2026-07-16T15:49:15Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1266.eqiad.wmnet with reason: host reimage 2437959 wikitext text/x-wiki podf04nascc5evdhat51oapwkupdaxi 2437962 2437959 2026-07-16T16:01:52Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet 2437962 wikitext text/x-wiki sd7iia6izd3ugxmpnwf7xlq3m69ssbx 2437963 2437962 2026-07-16T16:01:53Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet 2437963 wikitext text/x-wiki 6s1v139wmaht5lydjdvvtksv1u0rn2o 2437964 2437963 2026-07-16T16:01:54Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1270.eqiad.wmnet 2437964 wikitext text/x-wiki ridcwzrtubzsih1mc4b2ppd14x2yk73 2437965 2437964 2026-07-16T16:01:57Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1270.eqiad.wmnet 2437965 wikitext text/x-wiki cj4sn2ywedeu6iihf62umvxldgpsmfb 2437966 2437965 2026-07-16T16:02:13Z Stashbot 7414 jiji@deploy2003: Started scap sync-world: Backport for [[gerrit:1311465|ProductionServices: reboot poolcounter1007 (#2/4) (T431705)]] 2437966 wikitext text/x-wiki 54t4qh1k4224voxdnzrf2hg0kk0g70x 2437967 2437966 2026-07-16T16:02:28Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1270.eqiad.wmnet 2437967 wikitext text/x-wiki 4f2ijq93ebhe035zf60d14dcr9xq58u 2437968 2437967 2026-07-16T16:03:12Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1270.eqiad.wmnet with OS trixie 2437968 wikitext text/x-wiki dyy3pg29q0flegwwqesznuwbl55jeao 2437969 2437968 2026-07-16T16:03:41Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1270 2437969 wikitext text/x-wiki e1kysj9tf071y676fp8p18xm635bh3q 2437970 2437969 2026-07-16T16:03:58Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1265.eqiad.wmnet 2437970 wikitext text/x-wiki 1h2cjywystb2zx179m0awoq8iwymudz 2437971 2437970 2026-07-16T16:03:59Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1265.eqiad.wmnet 2437971 wikitext text/x-wiki kkcjydw9jm2rltlhk30cww40v3k69qf 2437972 2437971 2026-07-16T16:04:01Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1265.eqiad.wmnet 2437972 wikitext text/x-wiki co26zcosfpddtrxkaohnug3r3xoqlwe 2437973 2437972 2026-07-16T16:04:22Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.netbox 2437973 wikitext text/x-wiki kdx9ydv33os97v5v9iggnjl94g3plw0 2437974 2437973 2026-07-16T16:06:21Z Stashbot 7414 jiji@deploy2003: jiji: Backport for [[gerrit:1311465|ProductionServices: reboot poolcounter1007 (#2/4) (T431705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2437974 wikitext text/x-wiki pjb1cadosdru664t1oab3dgeftecz37 2437975 2437974 2026-07-16T16:07:44Z Stashbot 7414 jiji@deploy2003: jiji: Continuing with deployment 2437975 wikitext text/x-wiki 0q3t1gtaiecfobjc2o9a0je8caa1zsc 2437976 2437975 2026-07-16T16:08:40Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet 2437976 wikitext text/x-wiki 0rg0mb5mjwncid8fd7uf46tnxhcjh01 2437978 2437976 2026-07-16T16:09:23Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1270 - swfrench@cumin1003" 2437978 wikitext text/x-wiki 3g7vw4pz8sggm4h916w5z66cs4toyio 2437979 2437978 2026-07-16T16:09:28Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1270 - swfrench@cumin1003" 2437979 wikitext text/x-wiki n157r0kiavxs8k556jb2zzqtgdr0m9a 2437980 2437979 2026-07-16T16:09:29Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2437980 wikitext text/x-wiki it3zakyj0i9x95vg5jltwj1xj1zcwfs 2437981 2437980 2026-07-16T16:09:30Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1270.eqiad.wmnet 125.48.64.10.in-addr.arpa 5.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437981 wikitext text/x-wiki 286hf8n1wnkvroa953uipc6zpxl0vqv 2437982 2437981 2026-07-16T16:09:32Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1270.eqiad.wmnet 125.48.64.10.in-addr.arpa 5.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2437982 wikitext text/x-wiki 5ufxbtosbikmgm64e6qeah6r497a36f 2437983 2437982 2026-07-16T16:09:33Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1270 2437983 wikitext text/x-wiki lq82wki5e9deir7p1moptwkehcv91e1 2437984 2437983 2026-07-16T16:10:56Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1270 2437984 wikitext text/x-wiki 3dwwabwu4tvy7aclyb7zk08fgfm6yt2 2437985 2437984 2026-07-16T16:10:58Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1270 2437985 wikitext text/x-wiki 610msqjq8rt9pbwvzexk2lrxf7rxoza 2437986 2437985 2026-07-16T16:10:59Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1266.eqiad.wmnet with OS trixie 2437986 wikitext text/x-wiki mxa81eikgpkxe5clqv8b8ch4rmrvu8d 2437987 2437986 2026-07-16T16:12:00Z Stashbot 7414 jiji@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311465|ProductionServices: reboot poolcounter1007 (#2/4) (T431705)]] (duration: 09m 47s) 2437987 wikitext text/x-wiki 93yxxr7q50jt1kw3mg02zv38kkirqe4 2437988 2437987 2026-07-16T16:12:30Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet 2437988 wikitext text/x-wiki d7p3po9b5iivm1k24iqlduaedzi51s8 2437989 2437988 2026-07-16T16:13:05Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet 2437989 wikitext text/x-wiki q2i0ap9u5nu6kb5kckzzbqilvtzyppd 2437990 2437989 2026-07-16T16:13:57Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet 2437990 wikitext text/x-wiki 84v805jx8mnrks81f4ldse85mui1tum 2437991 2437990 2026-07-16T16:14:59Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet 2437991 wikitext text/x-wiki i0r6sye5cvx8w7niqq9nbre09rqhbsw 2437992 2437991 2026-07-16T16:15:01Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_eqsin 2437992 wikitext text/x-wiki lp5nrv6barju6qxrluynhx3nhvwkkzj 2437993 2437992 2026-07-16T16:16:15Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet 2437993 wikitext text/x-wiki fiyxsz7bmw90839ek62548fag1z89rd 2437994 2437993 2026-07-16T16:19:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2437994 wikitext text/x-wiki q7f439le61yr3p8x91e29hyqjj6ppk5 2437995 2437994 2026-07-16T16:20:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2185.codfw.wmnet with OS trixie 2437995 wikitext text/x-wiki 0y462ui8j92d3ig5l5sp9qsqz8p1o12 2437996 2437995 2026-07-16T16:21:51Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2162: switch maintenance completed codfw rack b6 2437996 wikitext text/x-wiki 8w8agt784vyi7m790xbs108059jqlme 2437997 2437996 2026-07-16T16:23:33Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2437997 wikitext text/x-wiki cjdnhtpagf3f76nhrvj46ml2htw1tn2 2437998 2437997 2026-07-16T16:24:29Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2437998 wikitext text/x-wiki rjmo7m1mtwf8arc7md9pptqws59hh3d 2437999 2437998 2026-07-16T16:24:55Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1264.eqiad.wmnet with OS trixie 2437999 wikitext text/x-wiki 777c3p7phxz7hiu3gmrinq7sry1pcad 2438000 2437999 2026-07-16T16:24:56Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1264.eqiad.wmnet 2438000 wikitext text/x-wiki tdguk7linzf1t5keo8jwrp2ze5ug9n7 2438001 2438000 2026-07-16T16:31:44Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1270.eqiad.wmnet with reason: host reimage 2438001 wikitext text/x-wiki lzsjnthb2fa3bdfvv2uedu2wsjnbxmx 2438005 2438001 2026-07-16T16:37:42Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1267.eqiad.wmnet 2438005 wikitext text/x-wiki d0156sus8y63d02daxnwdgom1iqjc7k 2438006 2438005 2026-07-16T16:37:45Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1267.eqiad.wmnet 2438006 wikitext text/x-wiki jbfbwm8ddln35fw0j1rpe8fgy4dxzjq 2438007 2438006 2026-07-16T16:38:04Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1270.eqiad.wmnet with reason: host reimage 2438007 wikitext text/x-wiki apr8gf89vch0vyl2ns6wm17ufukp46r 2438008 2438007 2026-07-16T16:38:17Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1267.eqiad.wmnet 2438008 wikitext text/x-wiki 5324eujy0amayaqi3lz0hdlfmgjaqpb 2438009 2438008 2026-07-16T16:38:56Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1267.eqiad.wmnet with OS trixie 2438009 wikitext text/x-wiki 65gtrz9g3llw9v8vp8d05gda55kybz4 2438010 2438009 2026-07-16T16:39:10Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2185.codfw.wmnet with reason: host reimage 2438010 wikitext text/x-wiki grw8y7ssovz9xnwdm3mfuuffp2mc2nr 2438011 2438010 2026-07-16T16:39:25Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1267 2438011 wikitext text/x-wiki hu7vhgtg0rs53i6ewxoeuzqacyfkv4e 2438012 2438011 2026-07-16T16:39:37Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.netbox 2438012 wikitext text/x-wiki ag4vkbn3xfvdnj4wodrhhz7j28yz717 2438014 2438012 2026-07-16T16:41:50Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet 2438014 wikitext text/x-wiki 698akcf224w3z6d5eympsp2g622tlls 2438015 2438014 2026-07-16T16:41:52Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad 2438015 wikitext text/x-wiki tnaccezuf767s7g261jw3tw299np9zg 2438016 2438015 2026-07-16T16:41:53Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet 2438016 wikitext text/x-wiki br99scldcxnrcf8pyz3ckmmxffevrka 2438017 2438016 2026-07-16T16:41:54Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_eqiad 2438017 wikitext text/x-wiki 9vj5cbsa7t69v3vnmgherwwrrhjxw77 2438018 2438017 2026-07-16T16:43:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2185.codfw.wmnet with reason: host reimage 2438018 wikitext text/x-wiki 363nes55nwv2nlmzlp9fu675rh80kp1 2438019 2438018 2026-07-16T16:45:48Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1266.eqiad.wmnet 2438019 wikitext text/x-wiki snbuib2ao4hwydjreclgxjae5a4cd8w 2438020 2438019 2026-07-16T16:45:50Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1266.eqiad.wmnet 2438020 wikitext text/x-wiki mngwqsalev63ikryoi5f7dxhifgrxzw 2438021 2438020 2026-07-16T16:45:51Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1266.eqiad.wmnet 2438021 wikitext text/x-wiki fkyz86od369gt212turh1ksllpyoo6w 2438024 2438021 2026-07-16T16:50:42Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet 2438024 wikitext text/x-wiki qr2j5hl1zmq7rqi1voorml4e3rlja55 2438025 2438024 2026-07-16T16:50:43Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_esams 2438025 wikitext text/x-wiki exu2oqa4xdz8914sx9kvpdxitgh3d8t 2438026 2438025 2026-07-16T16:52:52Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet 2438026 wikitext text/x-wiki n2cx5wuuoe95ixks4h1byfc2f0z24be 2438027 2438026 2026-07-16T16:52:54Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_esams 2438027 wikitext text/x-wiki 2v54qh0robdea8erz7g5xbw3qhcrwoo 2438028 2438027 2026-07-16T16:56:05Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet 2438028 wikitext text/x-wiki 0mvcbzod1dck8yudos1qg59f2p1j1nx 2438029 2438028 2026-07-16T16:56:06Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqsin 2438029 wikitext text/x-wiki q4a0c2ysdxbxr20tspj7qu5vldgvhah 2438030 2438029 2026-07-16T16:57:00Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1267 - kamila@cumin1003" 2438030 wikitext text/x-wiki dn0ip8f793djlalt11zy4e8uqxd1ht7 2438031 2438030 2026-07-16T16:57:04Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1267 - kamila@cumin1003" 2438031 wikitext text/x-wiki emcyibz2rdk8jwuuj62yxyuwf0nco46 2438032 2438031 2026-07-16T16:57:06Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438032 wikitext text/x-wiki nuqfr3iv3muexbgjzrz1p7fi53vahpo 2438033 2438032 2026-07-16T16:57:07Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1267.eqiad.wmnet 77.32.64.10.in-addr.arpa 7.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438033 wikitext text/x-wiki 0eolbqqlh5ps679f1stdjaqwkgvtahb 2438034 2438033 2026-07-16T16:57:08Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1267.eqiad.wmnet 77.32.64.10.in-addr.arpa 7.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438034 wikitext text/x-wiki r7pvoiq9lsuljij5mec4vb8cm8tazld 2438035 2438034 2026-07-16T16:57:09Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1267 2438035 wikitext text/x-wiki f2n9dntpnrksq49o8k2rfek87lmhru5 2438036 2438035 2026-07-16T16:57:45Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1267 2438036 wikitext text/x-wiki o19mut0ib7aii6ccrypolgw26j4iqdf 2438037 2438036 2026-07-16T16:57:46Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1267 2438037 wikitext text/x-wiki 21jawog85t8l23a39511j9kf4hxg5c0 2438038 2438037 2026-07-16T16:58:09Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1270.eqiad.wmnet with OS trixie 2438038 wikitext text/x-wiki afik9h5eyqnu432rcmjmzzbzhl9x5ha 2438039 2438038 2026-07-16T16:59:49Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2438039 wikitext text/x-wiki 8lstiioa6mp5bn5qtrh0c8ud7dhz8da 2438040 2438039 2026-07-16T17:00:45Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2185.codfw.wmnet with OS trixie 2438040 wikitext text/x-wiki 4xuhd1b889mc3140f7lymy2psvxcrbu 2438041 2438040 2026-07-16T17:00:51Z Stashbot 7414 jiji@deploy2003: Started scap sync-world: Backport for [[gerrit:1311466|ProductionServices: reboot poolcounter2005 (#3/4) (T431705)]] 2438041 wikitext text/x-wiki d953odmlmb2tgij1kg86uhdkmyolm6i 2438042 2438041 2026-07-16T17:01:41Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2438042 wikitext text/x-wiki kdpmfefrftrlcoj4mndwt6pi9xwylxl 2438043 2438042 2026-07-16T17:03:11Z Stashbot 7414 jiji@deploy2003: jiji: Backport for [[gerrit:1311466|ProductionServices: reboot poolcounter2005 (#3/4) (T431705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2438043 wikitext text/x-wiki gtuegkdftbpij4la9rryizuzohqp0nk 2438044 2438043 2026-07-16T17:04:11Z Stashbot 7414 jiji@deploy2003: jiji: Continuing with deployment 2438044 wikitext text/x-wiki i4h9qkme5asxc985j3vh433kesuch54 2438046 2438044 2026-07-16T17:08:25Z Stashbot 7414 jiji@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311466|ProductionServices: reboot poolcounter2005 (#3/4) (T431705)]] (duration: 07m 34s) 2438046 wikitext text/x-wiki nye8xgmtoirjsgdkyji8h4csblhvqpv 2438047 2438046 2026-07-16T17:08:40Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for moved arelion cct cr2-eqiad - cmooney@cumin1003" 2438047 wikitext text/x-wiki t1yxlv4xdhvgxcc1hadjnc63kh2vatp 2438048 2438047 2026-07-16T17:08:44Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for moved arelion cct cr2-eqiad - cmooney@cumin1003" 2438048 wikitext text/x-wiki gkuch88q6wgrzfdf2uzy2pfqppt5kbg 2438049 2438048 2026-07-16T17:08:46Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438049 wikitext text/x-wiki 92s2qouwj6w4rc6m8a879v9qyzmrieg 2438050 2438049 2026-07-16T17:09:08Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet 2438050 wikitext text/x-wiki sa2uuy1hkzhmqq3tlncwgsvtl8n7j6p 2438051 2438050 2026-07-16T17:11:56Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1270.eqiad.wmnet 2438051 wikitext text/x-wiki 037ojxs48oidxw81fhmguldepw5yvzt 2438052 2438051 2026-07-16T17:11:57Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1270.eqiad.wmnet 2438052 wikitext text/x-wiki 52f7dwoot38i2w0u4p9fezh5hwtp2o9 2438053 2438052 2026-07-16T17:11:59Z Stashbot 7414 swfrench@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1270.eqiad.wmnet 2438053 wikitext text/x-wiki ndgip3o4cdgx87gmcg7m1gubmos9uc6 2438056 2438053 2026-07-16T17:12:57Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet 2438056 wikitext text/x-wiki ax2pum79cyxr1jvr8uj23nq8729ixnj 2438057 2438056 2026-07-16T17:17:50Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1268.eqiad.wmnet 2438057 wikitext text/x-wiki 96rxjxfismcr9tnls63c4u472259p7g 2438058 2438057 2026-07-16T17:17:54Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1268.eqiad.wmnet 2438058 wikitext text/x-wiki rkm9d00d8a1touuhjkh1cdkgphjxkl2 2438059 2438058 2026-07-16T17:18:26Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1268.eqiad.wmnet 2438059 wikitext text/x-wiki nny1w9pf0l70e2o7npl368e0ktxexdo 2438060 2438059 2026-07-16T17:18:30Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1267.eqiad.wmnet with reason: host reimage 2438060 wikitext text/x-wiki mby8ugh8vl144wgxhfcclb444v0kcbe 2438062 2438060 2026-07-16T17:23:50Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1267.eqiad.wmnet with reason: host reimage 2438062 wikitext text/x-wiki gaudv7jz144bm095ar5ptcvy2nuf4e7 2438068 2438062 2026-07-16T17:27:52Z Stashbot 7414 swfrench@deploy2003: Started scap sync-world: Backport for [[gerrit:1311467|ProductionServices: reboot poolcounter2006 (#4/4) (T431705)]] 2438068 wikitext text/x-wiki swv9pu4166315pqin9hlwzxy1zhn57k 2438069 2438068 2026-07-16T17:28:12Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1268.eqiad.wmnet with OS trixie 2438069 wikitext text/x-wiki qyo6u5m20l9ph9gbavakjaqhyckzbp4 2438070 2438069 2026-07-16T17:28:39Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1268 2438070 wikitext text/x-wiki oe8m9isrcx0q77n8nmg0l0cxlt6onpp 2438071 2438070 2026-07-16T17:28:47Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.netbox 2438071 wikitext text/x-wiki 4322rpcueb4kqiy1n9qprokv6w79j5z 2438072 2438071 2026-07-16T17:29:51Z Stashbot 7414 swfrench@deploy2003: jiji, swfrench: Backport for [[gerrit:1311467|ProductionServices: reboot poolcounter2006 (#4/4) (T431705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2438072 wikitext text/x-wiki tci3gn8v6vc76j7mqbivqsmn7j89ng8 2438077 2438072 2026-07-16T17:31:05Z Stashbot 7414 swfrench@deploy2003: jiji, swfrench: Continuing with deployment 2438077 wikitext text/x-wiki ja7e6x6z7wz99jitqtxob3xiww8e189 2438078 2438077 2026-07-16T17:34:11Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1268 - kamila@cumin1003" 2438078 wikitext text/x-wiki jl4rzauwjd06kq3z6861hxltgmd16e3 2438079 2438078 2026-07-16T17:34:15Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1268 - kamila@cumin1003" 2438079 wikitext text/x-wiki t7nhufyvxncoo2eof2t51lyltymt37d 2438080 2438079 2026-07-16T17:34:17Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438080 wikitext text/x-wiki g59wdc3ndneayyz7ij6m4a40818jh0b 2438081 2438080 2026-07-16T17:34:18Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1268.eqiad.wmnet 78.32.64.10.in-addr.arpa 8.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438081 wikitext text/x-wiki 6xufyn3sfokskumdevv0udrd23zoem8 2438082 2438081 2026-07-16T17:34:19Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1268.eqiad.wmnet 78.32.64.10.in-addr.arpa 8.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438082 wikitext text/x-wiki 8j60oejelunpeg8tug9s1k3q9c3kijo 2438083 2438082 2026-07-16T17:34:21Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1268 2438083 wikitext text/x-wiki 84lfjx7glrbnx5conimlrr08zktwtxg 2438084 2438083 2026-07-16T17:35:20Z Stashbot 7414 swfrench@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311467|ProductionServices: reboot poolcounter2006 (#4/4) (T431705)]] (duration: 07m 27s) 2438084 wikitext text/x-wiki s4wrjrtkirgnbmakr72fh9p1p6xphnp 2438086 2438084 2026-07-16T17:39:51Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet 2438086 wikitext text/x-wiki r22yu74cjf7mgdpezr5ve3uq438aez2 2438087 2438086 2026-07-16T17:43:39Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet 2438087 wikitext text/x-wiki 5rmpkkkniijsnpb490p4a5vu9mg9bf1 2438088 2438087 2026-07-16T17:44:40Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1267.eqiad.wmnet with OS trixie 2438088 wikitext text/x-wiki j6suwaduw0yjmtipdbifs0t5os8zihn 2438089 2438088 2026-07-16T17:45:02Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1268 2438089 wikitext text/x-wiki exwpqr86p6yp5ewm7kiwx0wr9ysxama 2438090 2438089 2026-07-16T17:45:03Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1268 2438090 wikitext text/x-wiki radbgs18y3oa8rzc9uk428ops6xqvsg 2438091 2438090 2026-07-16T17:59:19Z Stashbot 7414 swfrench@deploy2003: Started scap sync-world: Backport for [[gerrit:1311468|ProductionServices: repool poolcounter2006 after reboot (T431705)]] 2438091 wikitext text/x-wiki 794j30adnrm3hkffcandhaoquppy6tv 2438092 2438091 2026-07-16T18:01:22Z Stashbot 7414 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply 2438092 wikitext text/x-wiki 3hz2z19ppsklpb4llvbl6k05vhh2jv5 2438093 2438092 2026-07-16T18:01:25Z Stashbot 7414 swfrench@deploy2003: jiji, swfrench: Backport for [[gerrit:1311468|ProductionServices: repool poolcounter2006 after reboot (T431705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2438093 wikitext text/x-wiki b5j6deh4kvbnq6b1oe8r9blqfr7u6i0 2438094 2438093 2026-07-16T18:01:41Z Stashbot 7414 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply 2438094 wikitext text/x-wiki 9g2uiogrr8e1gqvyy3pooii41506rpi 2438095 2438094 2026-07-16T18:02:09Z Stashbot 7414 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply 2438095 wikitext text/x-wiki he0otv3wjxuutkc0oua8hnpw6ti5j0e 2438096 2438095 2026-07-16T18:02:27Z Stashbot 7414 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply 2438096 wikitext text/x-wiki rhhyyombly9c22orvzp9zryheega0qu 2438097 2438096 2026-07-16T18:02:28Z Stashbot 7414 swfrench@deploy2003: jiji, swfrench: Continuing with deployment 2438097 wikitext text/x-wiki 5c33yash3j0xbshjtc7ifurie81jlqx 2438098 2438097 2026-07-16T18:02:49Z Stashbot 7414 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply 2438098 wikitext text/x-wiki b8pgo3db4653bsufytc2qqh3tkia102 2438099 2438098 2026-07-16T18:03:18Z Stashbot 7414 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply 2438099 wikitext text/x-wiki tinuf5mecrld1zq5azvu9b3kvl3u3nc 2438100 2438099 2026-07-16T18:06:08Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1268.eqiad.wmnet with reason: host reimage 2438100 wikitext text/x-wiki 5dunukmp65hp2q1xf83uaqkqczbx323 2438101 2438100 2026-07-16T18:06:26Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 2438101 wikitext text/x-wiki ofjf7ny6ay999fxou7ioucym1eo34dk 2438102 2438101 2026-07-16T18:06:35Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 (duration: 00m 23s) 2438102 wikitext text/x-wiki q3k5g5xy68kbcweisxwx9frp4nh2xof 2438103 2438102 2026-07-16T18:06:54Z Stashbot 7414 swfrench@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311468|ProductionServices: repool poolcounter2006 after reboot (T431705)]] (duration: 07m 34s) 2438103 wikitext text/x-wiki krd0s15s9biazyow6fpoxao0a748g0v 2438105 2438103 2026-07-16T18:08:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430879, restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -> wcqs1001.eqiad.wmnet, repooling source-only afterwards 2438105 wikitext text/x-wiki 6cssjd5lcyohsp04vobvzxb2x58nxi7 2438106 2438105 2026-07-16T18:09:46Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1268.eqiad.wmnet with reason: host reimage 2438106 wikitext text/x-wiki aqy44mifn4tn85zbwh50979nv5tkaa1 2438107 2438106 2026-07-16T18:16:03Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1267.eqiad.wmnet 2438107 wikitext text/x-wiki 32n5gi9ghe9au268ah48n3bw0ktszyh 2438108 2438107 2026-07-16T18:16:04Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1267.eqiad.wmnet 2438108 wikitext text/x-wiki ozpbr4qta5ojkt6lw1e7cgtzf4ry0x9 2438109 2438108 2026-07-16T18:16:06Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1267.eqiad.wmnet 2438109 wikitext text/x-wiki m1k3ocghcbdlwvbya3yvg2gz51ip46l 2438110 2438109 2026-07-16T18:18:49Z Stashbot 7414 jhuneidi@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.11 refs T430830 2438110 wikitext text/x-wiki hyi8r6wqhqoqvm4b53y60c2dzmczlsm 2438111 2438110 2026-07-16T18:31:49Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1268.eqiad.wmnet with OS trixie 2438111 wikitext text/x-wiki 4f535n624lewcokomz3qgswifyrbdf4 2438115 2438111 2026-07-16T18:53:03Z Stashbot 7414 dancy@deploy2003: Started scap sync-world: testing T428971 2438115 wikitext text/x-wiki nhrch72scolgsoia7t9du3ynqo1znwu 2438116 2438115 2026-07-16T18:55:44Z Stashbot 7414 dancy@deploy2003: Finished scap sync-world: testing T428971 (duration: 02m 41s) 2438116 wikitext text/x-wiki dxmla1ynthdi9e2ibhna7hn1x1b35ao 2438117 2438116 2026-07-16T19:03:01Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1269.eqiad.wmnet 2438117 wikitext text/x-wiki ouwdiojsl67x42ai56k0692qsitzq36 2438118 2438117 2026-07-16T19:03:04Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1269.eqiad.wmnet 2438118 wikitext text/x-wiki qgwev0mmr6j5cclckzr62kk43tfcmry 2438119 2438118 2026-07-16T19:03:36Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1269.eqiad.wmnet 2438119 wikitext text/x-wiki ney8hd8dw2lour86ylf08l635zbxs6c 2438120 2438119 2026-07-16T19:05:51Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1269.eqiad.wmnet with OS trixie 2438120 wikitext text/x-wiki ktgembulgfvmpl1u3d1ovsg55q0gijm 2438121 2438120 2026-07-16T19:06:21Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1269 2438121 wikitext text/x-wiki g7zfntyzsjs3cud2oe3w2xqbuooan0v 2438122 2438121 2026-07-16T19:12:24Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1029.eqiad.wmnet 2438122 wikitext text/x-wiki n3svbag36uoo8rchrc94q2k1tbxu9wz 2438124 2438122 2026-07-16T19:17:19Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.netbox 2438124 wikitext text/x-wiki q4s949t1rqpbivhiwlbdgsb8tmh178u 2438125 2438124 2026-07-16T19:19:47Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1029.eqiad.wmnet 2438125 wikitext text/x-wiki 55x0y4s0d1e6xn5e9l3w9s1774m774i 2438126 2438125 2026-07-16T19:19:54Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet 2438126 wikitext text/x-wiki bhk858cbpqle5piie64g1wlm9tpczay 2438127 2438126 2026-07-16T19:23:28Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430879, restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -> wcqs1001.eqiad.wmnet, repooling source-only afterwards 2438127 wikitext text/x-wiki gfx9ssi2dz98ooywuarcve17glio8qx 2438128 2438127 2026-07-16T19:27:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet 2438128 wikitext text/x-wiki d8spw6ofypp9ol6opecpr4uqluw417w 2438129 2438128 2026-07-16T19:27:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet 2438129 wikitext text/x-wiki ko70k8fb0rmcrks4cc94z99zqadho13 2438130 2438129 2026-07-16T19:31:49Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply 2438130 wikitext text/x-wiki snwat00zeahzztqplp79d8jffodgfwp 2438131 2438130 2026-07-16T19:32:20Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2438131 wikitext text/x-wiki 1sc3j312w2m2xsc0fz3xldkp8lrltqn 2438132 2438131 2026-07-16T19:32:22Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2438132 wikitext text/x-wiki 249m9uhfk44mwtu62d7izh8tkce0avu 2438133 2438132 2026-07-16T19:32:37Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2438133 wikitext text/x-wiki 1vmz5cp207aixmtwxf282bk7fpcfcoi 2438134 2438133 2026-07-16T19:32:38Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2438134 wikitext text/x-wiki f6hn77s7dyqry5og6ekkhhdccevtl2l 2438135 2438134 2026-07-16T19:32:53Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2438135 wikitext text/x-wiki oqw7r9afo2rkhx7tklmaidsmf2evkkt 2438136 2438135 2026-07-16T19:32:54Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2438136 wikitext text/x-wiki fbdscel1hls0ucfxsh0bc9rc0hvdid5 2438137 2438136 2026-07-16T19:33:09Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2438137 wikitext text/x-wiki qyrj5emnp3naq05lze1ivxzlsrfchd0 2438138 2438137 2026-07-16T19:33:11Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2438138 wikitext text/x-wiki rfubv7ks0e3dzsdb4on84r35bkd2hry 2438139 2438138 2026-07-16T19:33:31Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2438139 wikitext text/x-wiki 6u4ii69ep9ydwz8hj2h3kec34n8onyf 2438140 2438139 2026-07-16T19:33:33Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2438140 wikitext text/x-wiki km7gct044jq35hdm97aasv4s30l0z4e 2438141 2438140 2026-07-16T19:34:00Z Stashbot 7414 swfrench@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2438141 wikitext text/x-wiki qocaif07kzwd8ue9vb97aqbxudf0z89 2438142 2438141 2026-07-16T19:35:34Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2438142 wikitext text/x-wiki ql7o1pmobcdbjylrvfdz8v8k40wwgf3 2438143 2438142 2026-07-16T19:35:50Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet 2438143 wikitext text/x-wiki 3ys1ne05ktmwkqe3juthoaob4aui58l 2438144 2438143 2026-07-16T19:35:55Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet 2438144 wikitext text/x-wiki hjrd7bt7y0ilrvonhf28ilqo2raay44 2438145 2438144 2026-07-16T19:36:12Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2438145 wikitext text/x-wiki pew0etypj05xttp8iz4s5s5bk3mmp96 2438146 2438145 2026-07-16T19:36:43Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2438146 wikitext text/x-wiki 0moyf3w46mzzuflrft1m6c4qnkqug2w 2438147 2438146 2026-07-16T19:39:46Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2438147 wikitext text/x-wiki gqayv54qa12ewecica38fj6ux7o404k 2438148 2438147 2026-07-16T19:40:22Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2438148 wikitext text/x-wiki 5z06ynixeodu6ufm4ety40hl7von0p9 2438149 2438148 2026-07-16T19:40:31Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2438149 wikitext text/x-wiki ieee93vdrfx0mhbn49vw4lhgtihm1co 2438150 2438149 2026-07-16T19:41:02Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2438150 wikitext text/x-wiki akqmb5jsgoqu56c3ddyz9c3aza3eucs 2438151 2438150 2026-07-16T19:41:18Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2438151 wikitext text/x-wiki 15icoxevi8qokwsi952vlrpopu0rf03 2438152 2438151 2026-07-16T19:41:49Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2438152 wikitext text/x-wiki sssb469dqwnptqivx4jnlefzgm7anyt 2438153 2438152 2026-07-16T19:42:11Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2438153 wikitext text/x-wiki reqx7km50jr9nrmnak9hq86ymug9ckw 2438154 2438153 2026-07-16T19:42:42Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2438154 wikitext text/x-wiki a5tt3tevyj58ipblsw6zwhrp23mfk2e 2438155 2438154 2026-07-16T19:43:31Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet 2438155 wikitext text/x-wiki syry79dljib6z5lxgs2l0n26s8r888u 2438156 2438155 2026-07-16T19:43:35Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1033.eqiad.wmnet 2438156 wikitext text/x-wiki 9ssdn1l9y3qtonnrr14geae3xi7fk5h 2438157 2438156 2026-07-16T19:43:37Z Stashbot 7414 swfrench@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2438157 wikitext text/x-wiki a53jtb38qha8emi5no3eeck30vjrhp7 2438158 2438157 2026-07-16T19:48:55Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1033.eqiad.wmnet 2438158 wikitext text/x-wiki 6a4aqoibaxg2388jm3rw6d61dts9xa6 2438159 2438158 2026-07-16T19:48:59Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1034.eqiad.wmnet 2438159 wikitext text/x-wiki 5l3e4nafpv2gr2egl3sa8bqi1yj81bk 2438160 2438159 2026-07-16T19:54:27Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1034.eqiad.wmnet 2438160 wikitext text/x-wiki 1kj425mtuve53lpuvvrksg7no1yode5 2438161 2438160 2026-07-16T19:54:32Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1035.eqiad.wmnet 2438161 wikitext text/x-wiki 9jto2rlp1ipcvf5ph5vo0uom27b8fk6 2438162 2438161 2026-07-16T19:55:56Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T432396 2438162 wikitext text/x-wiki ajp1k42xpk65brjjpxs00en9j4kak1n 2438163 2438162 2026-07-16T19:56:31Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2204 with weight 0 T432396', diff saved to https://phabricator.wikimedia.org/P94891 and previous config saved to /var/cache/conftool/dbconfig/20260716-195628-marostegui.json 2438163 wikitext text/x-wiki 3xh00wmpvkhfl003mso7tl2xo6a90wg 2438165 2438163 2026-07-16T19:59:34Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1035.eqiad.wmnet 2438165 wikitext text/x-wiki khpzgdma14przwhbmmm4fb3hqjnrjev 2438166 2438165 2026-07-16T20:00:52Z Stashbot 7414 marostegui: Starting emergency s2 codfw failover from db2207 to db2204 - T432396 2438166 wikitext text/x-wiki od8ps98tt836wicnh7xz0slizg05389 2438167 2438166 2026-07-16T20:01:59Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2204 to s2 primary T432396', diff saved to https://phabricator.wikimedia.org/P94892 and previous config saved to /var/cache/conftool/dbconfig/20260716-200157-marostegui.json 2438167 wikitext text/x-wiki fpmre9cqukl23euucfcilequ2rvi7el 2438168 2438167 2026-07-16T20:03:00Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2207 T432396', diff saved to https://phabricator.wikimedia.org/P94893 and previous config saved to /var/cache/conftool/dbconfig/20260716-200257-marostegui.json 2438168 wikitext text/x-wiki esgf9itfspr8r7jrccdxnndr6ydxxmn 2438169 2438168 2026-07-16T20:03:18Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-flink-codfw cluster: Roll restart of jvm daemons. 2438169 wikitext text/x-wiki ph2v8xlatkafschj3o745ksuecg3rkv 2438172 2438169 2026-07-16T20:05:27Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2438172 wikitext text/x-wiki 7ll3wfjkrj3w5nmusn7fni4eqa7p591 2438173 2438172 2026-07-16T20:07:02Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2438173 wikitext text/x-wiki g4zjaqei4fit8vzzgou0d175zw7c6qo 2438174 2438173 2026-07-16T20:09:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-flink-codfw cluster: Roll restart of jvm daemons. 2438174 wikitext text/x-wiki ggzdq3s2rlherolb29a98swo2hdxnsp 2438175 2438174 2026-07-16T20:13:10Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1269 - kamila@cumin1003" 2438175 wikitext text/x-wiki 46d9xzv8xrchynb8lqq91bzunxq8qn9 2438176 2438175 2026-07-16T20:13:14Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1269 - kamila@cumin1003" 2438176 wikitext text/x-wiki q9uqhw4q6hpmhgk85m0fqu3puzsygea 2438177 2438176 2026-07-16T20:13:16Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438177 wikitext text/x-wiki tpkpcgtvsdcs1rcxoc0133lw3y7cogx 2438178 2438177 2026-07-16T20:13:17Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1269.eqiad.wmnet 80.32.64.10.in-addr.arpa 0.8.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438178 wikitext text/x-wiki 7wslrgmn0aar8exflie99g5238f5vkw 2438179 2438178 2026-07-16T20:13:18Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1269.eqiad.wmnet 80.32.64.10.in-addr.arpa 0.8.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438179 wikitext text/x-wiki 58nwpx44qn19oah91s9w3qwtdjxkoa5 2438180 2438179 2026-07-16T20:13:20Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1269 2438180 wikitext text/x-wiki paljdoqlnvrm6ix4irshs3m1oszk7ee 2438181 2438180 2026-07-16T20:16:41Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply 2438181 wikitext text/x-wiki s8ww8720ecpg4fygj2zqs1l5vcnxttv 2438182 2438181 2026-07-16T20:17:20Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply 2438182 wikitext text/x-wiki 7fqomxj7q8zrn8bmgoa2pwob344pqb5 2438183 2438182 2026-07-16T20:17:51Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply 2438183 wikitext text/x-wiki qldc9owy03zxedgo1kg252ponuk7dli 2438184 2438183 2026-07-16T20:18:19Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply 2438184 wikitext text/x-wiki bh6zdtbx30fvtn4j45r4fc6zqze9gmu 2438185 2438184 2026-07-16T20:18:50Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply 2438185 wikitext text/x-wiki cd25dbaijn7tc1yz3gjb35qdgz20yyx 2438186 2438185 2026-07-16T20:19:13Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply 2438186 wikitext text/x-wiki s6j6h9byiy74r47ski4oe74cu118ix3 2438187 2438186 2026-07-16T20:19:44Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply 2438187 wikitext text/x-wiki skgvtl95prhumgr4q0bzdcwhvjxuyu3 2438188 2438187 2026-07-16T20:20:04Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2438188 wikitext text/x-wiki 0h4o0rgjgc1y6xafdt2wpzskrnxysnu 2438189 2438188 2026-07-16T20:20:35Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2438189 wikitext text/x-wiki 0rt37odcsqbv3kwtf98tbhny9pxw8vh 2438190 2438189 2026-07-16T20:21:00Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2438190 wikitext text/x-wiki g0wkhwrqnkjfyr6a19jfwmmdsxy9axr 2438191 2438190 2026-07-16T20:21:31Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2438191 wikitext text/x-wiki qflohnd3d9uhedx39h53t1i3djoixsl 2438192 2438191 2026-07-16T20:22:37Z Stashbot 7414 swfrench@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2438192 wikitext text/x-wiki 5jcz76gk1oxr5kbemvnry8a6tib2qd7 2438193 2438192 2026-07-16T20:26:13Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db2207.codfw.wmnet with reason: Host down 2438193 wikitext text/x-wiki dzmt4ys82cl4kfbgbx5bld2bd0jjy6t 2438194 2438193 2026-07-16T20:33:39Z Stashbot 7414 aude@deploy2003: Started scap sync-world: Backport for [[gerrit:1311472|Preserve menus after toolbox (e.g. print/export) in page tools (T432316)]] 2438194 wikitext text/x-wiki t3l9fg4sh6ix470rkh6r481wbak2ck7 2438195 2438194 2026-07-16T20:35:18Z Stashbot 7414 aude@deploy2003: aude: Backport for [[gerrit:1311472|Preserve menus after toolbox (e.g. print/export) in page tools (T432316)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2438195 wikitext text/x-wiki 7hbj6p4j2q53no8q3u157j2h77neepb 2438196 2438195 2026-07-16T20:36:50Z Stashbot 7414 aude@deploy2003: aude: Continuing with deployment 2438196 wikitext text/x-wiki 6gdwfpzzb0xe420lid5ocie3r42m47t 2438197 2438196 2026-07-16T20:39:52Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1269 2438197 wikitext text/x-wiki r9m0y2ggyvcxn9z7b6u1rhba9lbx6jr 2438198 2438197 2026-07-16T20:39:54Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1269 2438198 wikitext text/x-wiki q80aq0a7e8gkggw7t4o5p12sfd0ufty 2438199 2438198 2026-07-16T20:41:13Z Stashbot 7414 aude@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311472|Preserve menus after toolbox (e.g. print/export) in page tools (T432316)]] (duration: 07m 34s) 2438199 wikitext text/x-wiki mpwbp4rlubq9njtgtwloqh2pan68086 2438201 2438199 2026-07-16T20:46:57Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1271.eqiad.wmnet 2438201 wikitext text/x-wiki npmbqkfjy326vd4i9znv5du35gx8x03 2438202 2438201 2026-07-16T20:47:00Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1271.eqiad.wmnet 2438202 wikitext text/x-wiki i3j6avov1ejsxnug4mg0fv5grgmcfko 2438203 2438202 2026-07-16T20:47:37Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1271.eqiad.wmnet 2438203 wikitext text/x-wiki mfmqizf29t4ru4pl9xgkmani5o31bfe 2438204 2438203 2026-07-16T20:48:04Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1271.eqiad.wmnet with OS trixie 2438204 wikitext text/x-wiki 9uafrbq62jooqioyttna4youcv6rsss 2438205 2438204 2026-07-16T20:48:34Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1271 2438205 wikitext text/x-wiki tu11gmlg3p9sviy484670m2gq9j6n5j 2438206 2438205 2026-07-16T20:49:01Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1268.eqiad.wmnet 2438206 wikitext text/x-wiki atky6rhro27h86zt2xlykez53atyr06 2438207 2438206 2026-07-16T20:49:03Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1268.eqiad.wmnet 2438207 wikitext text/x-wiki db7y8pqmrwvzfhgcw960j2q4kmwdd6u 2438208 2438207 2026-07-16T20:49:05Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1268.eqiad.wmnet 2438208 wikitext text/x-wiki 81l3f9xvyyx0jlm3wuf093po6crp47v 2438209 2438208 2026-07-16T20:49:34Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.netbox 2438209 wikitext text/x-wiki l49qoqnoe4bxrfe5ftw3a9gs19d942a 2438210 2438209 2026-07-16T20:50:55Z Stashbot 7414 arlolra@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2438210 wikitext text/x-wiki mkuifxvmplg2oid5gl6j8pq6nvecug8 2438211 2438210 2026-07-16T20:51:26Z Stashbot 7414 arlolra@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2438211 wikitext text/x-wiki qqjbe9w32vip5z4qtfa0x16djt8mdsv 2438212 2438211 2026-07-16T20:51:27Z Stashbot 7414 arlolra@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2438212 wikitext text/x-wiki 51btws8qb6tjkihhx91h3d06ig76x0j 2438213 2438212 2026-07-16T20:51:54Z Stashbot 7414 arlolra@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2438213 wikitext text/x-wiki hlbldi2vh0lo9g32rfah40pqs4x7t4y 2438214 2438213 2026-07-16T20:54:23Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1271 - swfrench@cumin1003" 2438214 wikitext text/x-wiki ii6qyqoa2hdq02cp4ttr8u1npewxivf 2438215 2438214 2026-07-16T20:54:27Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1271 - swfrench@cumin1003" 2438215 wikitext text/x-wiki jw5l5qhphlz5k4k886ceummxv3jnjwp 2438216 2438215 2026-07-16T20:54:28Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438216 wikitext text/x-wiki n8wgzmu4zm4mzk92cubr5pgnze06ddi 2438217 2438216 2026-07-16T20:54:30Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1271.eqiad.wmnet 126.48.64.10.in-addr.arpa 6.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438217 wikitext text/x-wiki 7lbcycsdjfw0o34mo329oxu4ta68gmn 2438218 2438217 2026-07-16T20:54:31Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1271.eqiad.wmnet 126.48.64.10.in-addr.arpa 6.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438218 wikitext text/x-wiki jivftmf8wmvwpg6gj6iv7iljlcxhjhn 2438219 2438218 2026-07-16T20:54:32Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1271 2438219 wikitext text/x-wiki 0k7y1xmank809cmndd4f23eraqdlsyi 2438220 2438219 2026-07-16T20:56:00Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1271 2438220 wikitext text/x-wiki 3wmbi48tf8sulu5ihitrltqr0zoc687 2438221 2438220 2026-07-16T20:56:01Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1271 2438221 wikitext text/x-wiki 5c6ue5g51sadiw2w42yd6o3ratowkxb 2438222 2438221 2026-07-16T21:00:35Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1269.eqiad.wmnet with reason: host reimage 2438222 wikitext text/x-wiki mufdix6vossan6q0vtvxeya4ypj66w3 2438223 2438222 2026-07-16T21:04:25Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1269.eqiad.wmnet with reason: host reimage 2438223 wikitext text/x-wiki iip1ctyd69mlglbr0rzu6ulf0odtsmu 2438224 2438223 2026-07-16T21:17:16Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1271.eqiad.wmnet with reason: host reimage 2438224 wikitext text/x-wiki 4nlz2nnwetguwfodn3z54iseb18r556 2438225 2438224 2026-07-16T21:20:00Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs2025.codfw.wmnet with OS bookworm 2438225 wikitext text/x-wiki l8rdpxpdeu1anbw3scz2i86v7mmkbvy 2438226 2438225 2026-07-16T21:20:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs2025 2438226 wikitext text/x-wiki 1gexquuasfkaji07m02oglvx1x9tg03 2438227 2438226 2026-07-16T21:22:10Z Stashbot 7414 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1311481|Cleanup: remove reauth indicator from log message (T432042)]] 2438227 wikitext text/x-wiki 40m6z4ah8vv87p6wwqqi27qgxo1buyi 2438228 2438227 2026-07-16T21:22:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2438228 wikitext text/x-wiki tghatjj5td5yvlfcn3abbuv25ajamv9 2438229 2438228 2026-07-16T21:23:53Z Stashbot 7414 sbassett@deploy2003: sbassett: Backport for [[gerrit:1311481|Cleanup: remove reauth indicator from log message (T432042)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2438229 wikitext text/x-wiki d16n5xoza0ff3q930ez5enzamkagerk 2438230 2438229 2026-07-16T21:24:38Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1271.eqiad.wmnet with reason: host reimage 2438230 wikitext text/x-wiki qg46po4ey7h1kd9cq4k2apkpp558j7v 2438231 2438230 2026-07-16T21:24:44Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1269.eqiad.wmnet with OS trixie 2438231 wikitext text/x-wiki dtp9ay9abwcax222aw5i4u605gw1hgf 2438232 2438231 2026-07-16T21:26:12Z Stashbot 7414 sbassett@deploy2003: sbassett: Continuing with deployment 2438232 wikitext text/x-wiki qp7spux16lx8w0bxp0bswtxg1nge7cj 2438233 2438232 2026-07-16T21:30:30Z Stashbot 7414 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311481|Cleanup: remove reauth indicator from log message (T432042)]] (duration: 08m 19s) 2438233 wikitext text/x-wiki sb63qxlt9ru7wl67oxwf8cqqwljhn6t 2438234 2438233 2026-07-16T21:37:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2025 - bking@cumin2003" 2438234 wikitext text/x-wiki mujymv308z8x5jn0exfjls74riz36vv 2438235 2438234 2026-07-16T21:37:11Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2025 - bking@cumin2003" 2438235 wikitext text/x-wiki sbh0a0mfq5fys9qp8itvel833bknwdg 2438236 2438235 2026-07-16T21:37:12Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438236 wikitext text/x-wiki 1yxnpq7kborcpevo6qwivquxpi6pvdu 2438237 2438236 2026-07-16T21:37:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs2025.codfw.wmnet 220.48.192.10.in-addr.arpa 0.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438237 wikitext text/x-wiki ipealqvgtj931vdkymh9f0ge2i34f9z 2438238 2438237 2026-07-16T21:37:15Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2025.codfw.wmnet 220.48.192.10.in-addr.arpa 0.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438238 wikitext text/x-wiki 6czq79s2z1qhc6s68a9li1cdsfu1voe 2438239 2438238 2026-07-16T21:37:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2025 2438239 wikitext text/x-wiki orha9csj3wmn80q7zlvhrw7hy9iov1w 2438240 2438239 2026-07-16T21:38:32Z Stashbot 7414 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1311482|Use non-sampled authentication log channel instead of authevents (T432042)]] 2438240 wikitext text/x-wiki dfpfc86j04qci3t7bxyrb2n8prrxg32 2438241 2438240 2026-07-16T21:40:17Z Stashbot 7414 sbassett@deploy2003: sbassett: Backport for [[gerrit:1311482|Use non-sampled authentication log channel instead of authevents (T432042)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2438241 wikitext text/x-wiki 6kh6nrn72ilqjfzz7zq4rhm1eoqgdqn 2438242 2438241 2026-07-16T21:40:32Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2025 2438242 wikitext text/x-wiki n6jqxb18eghkcevjf6d9rxdqvngqm6d 2438243 2438242 2026-07-16T21:40:33Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2025 2438243 wikitext text/x-wiki obsq8p0z4xtql1eta538kstvh5votjl 2438244 2438243 2026-07-16T21:40:49Z Stashbot 7414 sbassett@deploy2003: sbassett: Continuing with deployment 2438244 wikitext text/x-wiki kx23r5r336c0tl8ondyjm5fx1ntx43z 2438245 2438244 2026-07-16T21:45:03Z Stashbot 7414 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311482|Use non-sampled authentication log channel instead of authevents (T432042)]] (duration: 06m 31s) 2438245 wikitext text/x-wiki nsl1hyt4djbwe9ghj33ozje0kg68jen 2438246 2438245 2026-07-16T21:46:20Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1271.eqiad.wmnet with OS trixie 2438246 wikitext text/x-wiki g76i09i6mv95wu6k2qs7t8qmc69dr7r 2438247 2438246 2026-07-16T21:55:42Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1271.eqiad.wmnet 2438247 wikitext text/x-wiki ls2qizeroi5159ogtli3w6e69s0ln76 2438248 2438247 2026-07-16T21:55:44Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1271.eqiad.wmnet 2438248 wikitext text/x-wiki 4pzaqtvbdmdapedkhlma1xhkqf6sr25 2438249 2438248 2026-07-16T21:55:46Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1271.eqiad.wmnet 2438249 wikitext text/x-wiki mq9gywhice4i2tq03fpwvylzn03x80t 2438250 2438249 2026-07-16T21:59:29Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1272.eqiad.wmnet 2438250 wikitext text/x-wiki q78pjwo1lemwxuxga4g16e7k3ukvg4y 2438251 2438250 2026-07-16T21:59:33Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1272.eqiad.wmnet 2438251 wikitext text/x-wiki 907co2n3utjheayn5837ly5ov3xhlvp 2438252 2438251 2026-07-16T22:00:04Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1272.eqiad.wmnet 2438252 wikitext text/x-wiki tfsmx3q3k8qgxsst46jxvmyygbml9i4 2438253 2438252 2026-07-16T22:00:26Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1272.eqiad.wmnet with OS trixie 2438253 wikitext text/x-wiki q75waxhrd94tl1o98l8qvtschjgfah2 2438254 2438253 2026-07-16T22:00:55Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1272 2438254 wikitext text/x-wiki 04dfxtruegsn0opblinhnx24s94pk4m 2438255 2438254 2026-07-16T22:01:16Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.netbox 2438255 wikitext text/x-wiki i0zgz5r7c02glq0n4ha6ugakitu92l3 2438256 2438255 2026-07-16T22:03:34Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2025.codfw.wmnet with reason: host reimage 2438256 wikitext text/x-wiki nkc7ajbloevdnfu8o60ump5efz2zwnm 2438257 2438256 2026-07-16T22:05:42Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1272 - swfrench@cumin1003" 2438257 wikitext text/x-wiki gqsabxn0vowegdi7r8x6vlp5oyfvp58 2438258 2438257 2026-07-16T22:05:47Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1272 - swfrench@cumin1003" 2438258 wikitext text/x-wiki acx9l0ybvw45zyilbgu5oqst6nv8sya 2438259 2438258 2026-07-16T22:05:48Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438259 wikitext text/x-wiki 99l34zeygkd2tzbs9spn8vvwb58xlqc 2438260 2438259 2026-07-16T22:05:49Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1272.eqiad.wmnet 127.48.64.10.in-addr.arpa 7.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438260 wikitext text/x-wiki 7l0lgf4bxmsit3p6uxpwx65gel0a8wg 2438261 2438260 2026-07-16T22:05:51Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1272.eqiad.wmnet 127.48.64.10.in-addr.arpa 7.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438261 wikitext text/x-wiki nc1plqvr3dftx3adyz7fknifnowmb5l 2438262 2438261 2026-07-16T22:05:53Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1272 2438262 wikitext text/x-wiki katt0t35k10u2jv9f3wszwp1q3s3qt4 2438263 2438262 2026-07-16T22:06:43Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1272 2438263 wikitext text/x-wiki e6z913f20er1jluydcrkrqoxbqtu3fv 2438264 2438263 2026-07-16T22:06:45Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1272 2438264 wikitext text/x-wiki 491unkwlc0pz7x0znf07ybotnf21rw1 2438265 2438264 2026-07-16T22:09:43Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2025.codfw.wmnet with reason: host reimage 2438265 wikitext text/x-wiki fv41yatves8u6e1rxvoampahfhknj61 2438266 2438265 2026-07-16T22:17:36Z Stashbot 7414 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] 2438266 wikitext text/x-wiki jtwet00j0wxsz7j9eolz0v2ev8qtnpi 2438267 2438266 2026-07-16T22:19:23Z Stashbot 7414 ladsgroup@deploy2003: ladsgroup, urbanecm: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2438267 wikitext text/x-wiki pobg6ovdy8g6cmrgz63o02redzvv3t7 2438268 2438267 2026-07-16T22:22:07Z Stashbot 7414 ladsgroup@deploy2003: ladsgroup, urbanecm: Continuing with deployment 2438268 wikitext text/x-wiki 9kpubu9zi8mzpu31ytiegz27y0eb4dd 2438269 2438268 2026-07-16T22:26:27Z Stashbot 7414 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311420|Upload: Do not throw for failure to save a chunk file (T430986)]], [[gerrit:1311529|AddImage: Request only standard thumbnail sizes (T428797)]] (duration: 08m 51s) 2438269 wikitext text/x-wiki ft8hy49w04zfupymk3bobgqlrlpns6v 2438270 2438269 2026-07-16T22:27:30Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1272.eqiad.wmnet with reason: host reimage 2438270 wikitext text/x-wiki 7ldf87q8zx1lujybdipdudia4nle160 2438271 2438270 2026-07-16T22:28:02Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1269.eqiad.wmnet 2438271 wikitext text/x-wiki 2bxj5fizj07wzrqz32jbpw32c8dw451 2438272 2438271 2026-07-16T22:28:03Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1269.eqiad.wmnet 2438272 wikitext text/x-wiki ncocds8bmqwhm7l5c88baunszu74bwc 2438273 2438272 2026-07-16T22:28:05Z Stashbot 7414 kamila@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1269.eqiad.wmnet 2438273 wikitext text/x-wiki arvq5ybdghv1s2khrs8fnyndfaj5yej 2438274 2438273 2026-07-16T22:32:05Z Stashbot 7414 bking@deploy2003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2438274 wikitext text/x-wiki olxo0yo5bv03q0qab7bp1xwy9dgynia 2438275 2438274 2026-07-16T22:32:10Z Stashbot 7414 bking@deploy2003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 27s) 2438275 wikitext text/x-wiki ogjb4tnaog5k69op1dys4w9p08uayt3 2438276 2438275 2026-07-16T22:35:09Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1272.eqiad.wmnet with reason: host reimage 2438276 wikitext text/x-wiki jtut59k4mp9g8pirhk9ohe8ntt4rkjl 2438277 2438276 2026-07-16T22:39:53Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2025.codfw.wmnet with OS bookworm 2438277 wikitext text/x-wiki 4p3p6n2026u702e2zyqht8jfiuhnru0 2438281 2438277 2026-07-16T22:56:52Z Stashbot 7414 Amir1: deleting echo notifications from 2015 in group0 2438281 wikitext text/x-wiki mz917jtt0tnt1vmtt2acrocfikm287p 2438282 2438281 2026-07-16T22:57:29Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1272.eqiad.wmnet with OS trixie 2438282 wikitext text/x-wiki o3i78py0jme2w3jx5yaxlr2s8yoomrq 2438283 2438282 2026-07-16T23:01:10Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, xfer to freshly reimaged/scap-deployed wdqs2025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling source-only afterwards 2438283 wikitext text/x-wiki tiitmzsv933rlr0qkoq61kpzqh4rtip 2438286 2438283 2026-07-16T23:08:41Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1272.eqiad.wmnet 2438286 wikitext text/x-wiki 248nnrz0lkxgz1pdic0b45chn0c5xgi 2438287 2438286 2026-07-16T23:08:43Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1272.eqiad.wmnet 2438287 wikitext text/x-wiki q71vyfe2flqxod6cw75tiv5agzqbdkd 2438288 2438287 2026-07-16T23:08:44Z Stashbot 7414 swfrench@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1272.eqiad.wmnet 2438288 wikitext text/x-wiki svf3v5dy6x8c1973aou2h16d7z9e7in 2438290 2438288 2026-07-16T23:09:13Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=wdqs-internal-scholarly,name=eqiad 2438290 wikitext text/x-wiki niwrmhamh0vvsd2f1t7olic9ahay9b4 2438291 2438290 2026-07-16T23:12:19Z Stashbot 7414 ryankemper: T430880 depooled dnsdisc of wdqs-internal-scholarly-eqiad bc we only have 1 host there 2438291 wikitext text/x-wiki 1nqjqhzaj4hln4o73xz3kar8kztzgv8 2438292 2438291 2026-07-16T23:12:51Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1274.eqiad.wmnet 2438292 wikitext text/x-wiki ji6nqpizmjpvzpawv2aazyrwxnh5xnq 2438293 2438292 2026-07-16T23:12:55Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1274.eqiad.wmnet 2438293 wikitext text/x-wiki ep5183o1rtiihgfq8ym605qxwv1woip 2438295 2438293 2026-07-16T23:13:27Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1274.eqiad.wmnet 2438295 wikitext text/x-wiki 28t3ngx1j049noixko7uib9wdx2kw82 2438296 2438295 2026-07-16T23:13:58Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1274.eqiad.wmnet with OS trixie 2438296 wikitext text/x-wiki kgb1c2xe5cycz1gni3bnpw7giqacvl7 2438297 2438296 2026-07-16T23:14:27Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1274 2438297 wikitext text/x-wiki bqf2qq6qdo62vt5cy4sjgqxouu0zf6i 2438298 2438297 2026-07-16T23:14:36Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.netbox 2438298 wikitext text/x-wiki 1n4yjjieqn6be47r7qtxasp5ejypix0 2438299 2438298 2026-07-16T23:19:01Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1027.eqiad.wmnet with OS bookworm 2438299 wikitext text/x-wiki 4ikaf3inggc0neb0gkmom8uoaho2bt6 2438300 2438299 2026-07-16T23:19:02Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1274 - swfrench@cumin1003" 2438300 wikitext text/x-wiki moh8du7f739cudw8lydyfj4r2p5dhfs 2438301 2438300 2026-07-16T23:19:06Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1274 - swfrench@cumin1003" 2438301 wikitext text/x-wiki hwcw8ylz4sci1hr6z49fgr21pre3npw 2438302 2438301 2026-07-16T23:19:07Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438302 wikitext text/x-wiki c3a4djjfx9bey9mbhpn1enpjywbmbz9 2438303 2438302 2026-07-16T23:19:09Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1274.eqiad.wmnet 145.48.64.10.in-addr.arpa 5.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438303 wikitext text/x-wiki prkcvk2xabzjs6vnc142uly6kqyuwcg 2438304 2438303 2026-07-16T23:19:10Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1274.eqiad.wmnet 145.48.64.10.in-addr.arpa 5.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438304 wikitext text/x-wiki 26th8iakpjdushcue43absm3hq7g0bj 2438305 2438304 2026-07-16T23:19:11Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1274 2438305 wikitext text/x-wiki sy42e1g1wka26hb67pzmn7g2b7dtyq9 2438306 2438305 2026-07-16T23:19:37Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1025.eqiad.wmnet with OS bookworm 2438306 wikitext text/x-wiki 49ph66w17p6ywygfmxwi7r8468i2lbm 2438307 2438306 2026-07-16T23:19:46Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1274 2438307 wikitext text/x-wiki m0qzsldn11ni24w0rh9sdq2qjhn1kqt 2438308 2438307 2026-07-16T23:19:47Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1274 2438308 wikitext text/x-wiki tfhult5rcicy0zq2sfohtu5wkg1398u 2438309 2438308 2026-07-16T23:22:23Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1027 2438309 wikitext text/x-wiki srsxh0zm95q11o69bx91vwgkwmchx6b 2438310 2438309 2026-07-16T23:22:25Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1027 2438310 wikitext text/x-wiki 5lbos1d1ef4iy4n1yd4guml60p5wpp2 2438311 2438310 2026-07-16T23:23:01Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1025 2438311 wikitext text/x-wiki 3fu35m9wvgjfnivm3agxi93sj0eloit 2438312 2438311 2026-07-16T23:23:03Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1025 2438312 wikitext text/x-wiki 8e8wwxnmjxm4fkuaotqahpykq9zxrya 2438313 2438312 2026-07-16T23:38:37Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1274.eqiad.wmnet with reason: host reimage 2438313 wikitext text/x-wiki jr50q6c5nsi2stnfqqbwn94vhbm5zhg 2438314 2438313 2026-07-16T23:39:42Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1027.eqiad.wmnet with reason: host reimage 2438314 wikitext text/x-wiki a9lyrt20n3o67ktukm1bi81b3oretg0 2438315 2438314 2026-07-16T23:41:48Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1025.eqiad.wmnet with reason: host reimage 2438315 wikitext text/x-wiki okevucre6pbip6lyjesw36u8qf0ctz6 2438316 2438315 2026-07-16T23:43:55Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1274.eqiad.wmnet with reason: host reimage 2438316 wikitext text/x-wiki p9f225hgqxez274nmiw4zutavyvsa9r 2438317 2438316 2026-07-16T23:47:33Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1025.eqiad.wmnet with reason: host reimage 2438317 wikitext text/x-wiki ki5u9xcl9tassqzcpj0pcps3opeih86 2438318 2438317 2026-07-16T23:50:08Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1027.eqiad.wmnet with reason: host reimage 2438318 wikitext text/x-wiki 5wzme3mnf270a59ybxnlp469akl5956 2438319 2438318 2026-07-16T23:50:17Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, xfer to freshly reimaged/scap-deployed wdqs2025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling source-only afterwards 2438319 wikitext text/x-wiki d3d5dpnouxpkvu0s40xydbrefj4g9xr 2438321 2438319 2026-07-17T00:04:57Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1274.eqiad.wmnet with OS trixie 2438321 wikitext text/x-wiki lr9h42pgdcrwgq11mcj3y8z6i4sogdc 2438322 2438321 2026-07-17T00:13:36Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1025.eqiad.wmnet with OS bookworm 2438322 wikitext text/x-wiki q26m5ht8urebyvv8wzenc53x656s0zo 2438323 2438322 2026-07-17T00:13:51Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1027.eqiad.wmnet with OS bookworm 2438323 wikitext text/x-wiki s4pns3w3x5yjwdz9jz3v9gzz6puki9i 2438324 2438323 2026-07-17T00:14:57Z Stashbot 7414 swfrench@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1274.eqiad.wmnet 2438324 wikitext text/x-wiki 7vunqur9erv6cgyy15qcg25cmfjgkbl 2438325 2438324 2026-07-17T00:14:58Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1274.eqiad.wmnet 2438325 wikitext text/x-wiki pq0uousu1ypfo6zvq72h23nihdbhvba 2438326 2438325 2026-07-17T00:14:59Z Stashbot 7414 swfrench@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1274.eqiad.wmnet 2438326 wikitext text/x-wiki 55hnlhnh90pvevla7oeptdrgbaskwt2 2438327 2438326 2026-07-17T00:23:15Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, Restore wdqs1027 after Bookworm reimage) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs1027.eqiad.wmnet, repooling both afterwards 2438327 wikitext text/x-wiki ek428r28e50xguximvjwp1wg0e2joy7 2438328 2438327 2026-07-17T00:23:48Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, Restore wdqs1025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling source-only afterwards 2438328 wikitext text/x-wiki p69e33i9q60brr8yq9eukiancbryp2g 2438329 2438328 2026-07-17T00:52:50Z Stashbot 7414 urbanecm@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply 2438329 wikitext text/x-wiki 4rl3a9t1ca50v0sl6vspp11vg2tea5u 2438330 2438329 2026-07-17T00:53:30Z Stashbot 7414 urbanecm@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply 2438330 wikitext text/x-wiki ooyhbx70mex3k8dwzir50kmqhq8rx1a 2438331 2438330 2026-07-17T00:54:00Z Stashbot 7414 urbanecm@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply 2438331 wikitext text/x-wiki 14qqf29gjbmp3q0hj56y2plt7r4xfky 2438332 2438331 2026-07-17T00:54:23Z Stashbot 7414 urbanecm@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply 2438332 wikitext text/x-wiki hducpdiyi3t25dgsf14t21d06zeirdu 2438333 2438332 2026-07-17T00:54:28Z Stashbot 7414 urbanecm@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply 2438333 wikitext text/x-wiki gvnmbxrxmfn1d2nbm42s31fy8g5u2vq 2438334 2438333 2026-07-17T00:55:35Z Stashbot 7414 urbanecm@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply 2438334 wikitext text/x-wiki gfch1hlte0pfvr3zih6jz5ijvo81tbi 2438335 2438334 2026-07-17T01:07:59Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore wdqs1027 after Bookworm reimage) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs1027.eqiad.wmnet, repooling both afterwards 2438335 wikitext text/x-wiki d50d1m9h1frln7y2a42ssjiclavftde 2438336 2438335 2026-07-17T01:13:01Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore wdqs1025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling source-only afterwards 2438336 wikitext text/x-wiki 3xlalogj48mnqxhcc8bynjgulkxl7dd 2438338 2438336 2026-07-17T02:00:23Z Stashbot 7414 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image 2438338 wikitext text/x-wiki j4dcyq165ojov7aijcnvb5o3mcw4tl8 2438339 2438338 2026-07-17T02:07:13Z Stashbot 7414 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 49s) 2438339 wikitext text/x-wiki fehxhj2mdcg7r0mqm8it76vnddz47p8 2438340 2438339 2026-07-17T02:37:03Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=wdqs-internal-scholarly,name=eqiad 2438340 wikitext text/x-wiki 62hkbtmpn7n0cxxf4q7rwyml32rvqzf 2438341 2438340 2026-07-17T02:37:45Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards 2438341 wikitext text/x-wiki tbjzvlvdi9qahqqkg7kfvmvgpdfnkbk 2438342 2438341 2026-07-17T02:44:48Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards 2438342 wikitext text/x-wiki qwhhvz3bjjrdi41y0gnq9nuykmbeye6 2438343 2438342 2026-07-17T02:46:14Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards 2438343 wikitext text/x-wiki 3vzy3oqx8ezk271qizvi6q8vo9j9bdp 2438344 2438343 2026-07-17T02:46:17Z Stashbot 7414 ryankemper@cumin2003: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards 2438344 wikitext text/x-wiki k8c5mud1m6ihkfmhraovcb7fmlu61ul 2438348 2438344 2026-07-17T06:48:32Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netbox1003.eqiad.wmnet 2438348 wikitext text/x-wiki hnq4s8xtxpddjnoyk88yafwey27jyyr 2438349 2438348 2026-07-17T06:52:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox1003.eqiad.wmnet 2438349 wikitext text/x-wiki 05hgc68dpned6nz9sic8l7vxl45umhz 2438350 2438349 2026-07-17T07:04:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.disable-merges 2438350 wikitext text/x-wiki tuifaufxcsng43vfef9xh0g1qiblklt 2438351 2438350 2026-07-17T07:04:12Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) 2438351 wikitext text/x-wiki fj6yyub13kiva57wftmy4551gt8bzo5 2438352 2438351 2026-07-17T07:04:16Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster 2438352 wikitext text/x-wiki 1ekul827cqw4sbjs5mivzonzh4ba07o 2438353 2438352 2026-07-17T07:05:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet 2438353 wikitext text/x-wiki 4w87ujxlzyg53f8pqg8u5f4dakr3t3q 2438354 2438353 2026-07-17T07:11:39Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet 2438354 wikitext text/x-wiki ji5ecb5ikr2v37kgmtrojryjjqdocfb 2438355 2438354 2026-07-17T07:12:04Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet 2438355 wikitext text/x-wiki cs188urc6lo00s8bc1dx9jqpif880eg 2438356 2438355 2026-07-17T07:18:57Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet 2438356 wikitext text/x-wiki oh3mjwdoks4lcbe3pj3tc8jnfq5x95u 2438357 2438356 2026-07-17T07:19:08Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet 2438357 wikitext text/x-wiki 3y9gqbu33tsefxbmh79tlcnkpzs67bp 2438358 2438357 2026-07-17T07:28:11Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet 2438358 wikitext text/x-wiki 6wegtm8tefizk1z3bkr9e96p4uzd347 2438359 2438358 2026-07-17T07:28:22Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetserver1001.eqiad.wmnet 2438359 wikitext text/x-wiki 2mwq4nogtf07iulc6aafz43syr26yvx 2438360 2438359 2026-07-17T07:37:24Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1001.eqiad.wmnet 2438360 wikitext text/x-wiki tgdrexj457frl4ywz8u3jfe9ktqa9ni 2438361 2438360 2026-07-17T07:37:54Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetdb2003.codfw.wmnet 2438361 wikitext text/x-wiki 6jx24w222cegipgrmv61x1k3za7wqz4 2438362 2438361 2026-07-17T07:46:11Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetdb2003.codfw.wmnet 2438362 wikitext text/x-wiki 8cw0uiwkjzrv6pwsvexbehskmxqzu0a 2438363 2438362 2026-07-17T07:46:25Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetdb1003.eqiad.wmnet 2438363 wikitext text/x-wiki itgd4mlinnorpeb24mmskrhc4swf3ez 2438364 2438363 2026-07-17T07:52:25Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetdb1003.eqiad.wmnet 2438364 wikitext text/x-wiki svyfjx0np7ja4fea28o8q2txu8bj980 2438365 2438364 2026-07-17T07:57:44Z Stashbot 7414 elukey@cumin2003: START - Cookbook sre.hosts.reboot-single for host cumin1003.eqiad.wmnet 2438365 wikitext text/x-wiki mb1l7s7b6gf1348q27s04daeppsh6pv 2438366 2438365 2026-07-17T08:03:35Z Stashbot 7414 elukey@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin1003.eqiad.wmnet 2438366 wikitext text/x-wiki 1n9u63d9upvewk3s92hpciac8n0yhie 2438367 2438366 2026-07-17T08:05:39Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.disable-merges 2438367 wikitext text/x-wiki 31zlfiugy2engqzwe0mp3m09s52l54s 2438368 2438367 2026-07-17T08:06:07Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.puppet.disable-merges (exit_code=99) 2438368 wikitext text/x-wiki cra94k4uev5ocu4dm1l8tz5u226tyxo 2438369 2438368 2026-07-17T08:09:06Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1335.eqiad.wmnet 2438369 wikitext text/x-wiki antpjl1ontxv73q48bnjmod8q87a8xx 2438370 2438369 2026-07-17T08:09:10Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1335.eqiad.wmnet 2438370 wikitext text/x-wiki 3q6hm0biiqheol7dzs69rz69qzz014y 2438371 2438370 2026-07-17T08:09:44Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1335.eqiad.wmnet 2438371 wikitext text/x-wiki irdz7tsoa184ujwa9tw107lpzyxqfxd 2438372 2438371 2026-07-17T08:10:28Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host wikikube-worker1335.eqiad.wmnet with OS trixie 2438372 wikitext text/x-wiki izaqayq0vhfygh0dqx589f0vjfm21oa 2438373 2438372 2026-07-17T08:10:58Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1335 2438373 wikitext text/x-wiki ge9wjemf4csjxqbg5hvkfvzjogd1hht 2438374 2438373 2026-07-17T08:13:18Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2438374 wikitext text/x-wiki 5xmkq7drqq1nqiawzrwz4i9jbcbns63 2438375 2438374 2026-07-17T08:14:27Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.disable-merges 2438375 wikitext text/x-wiki rv280hyd5xkz4zjc1mjklu5lxradpfp 2438376 2438375 2026-07-17T08:14:29Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) 2438376 wikitext text/x-wiki bsrm5lkb4wr79w0et5rit39ooc4txp4 2438377 2438376 2026-07-17T08:18:07Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1335 - cgoubert@cumin2003" 2438377 wikitext text/x-wiki 0ytxab2t5yuhe7dn0i6k04ksaxty7jm 2438378 2438377 2026-07-17T08:18:11Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1335 - cgoubert@cumin2003" 2438378 wikitext text/x-wiki pe9n0s1c90q10ydd2pxohg7iqa6mfel 2438379 2438378 2026-07-17T08:18:13Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438379 wikitext text/x-wiki r5awhpwbxnyqh56h5hp7wb856pnqqo1 2438380 2438379 2026-07-17T08:18:14Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache wikikube-worker1335.eqiad.wmnet 150.32.64.10.in-addr.arpa 0.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438380 wikitext text/x-wiki 0kdf8y4is6knl18xgj9elu02b8se25y 2438381 2438380 2026-07-17T08:18:16Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1335.eqiad.wmnet 150.32.64.10.in-addr.arpa 0.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438381 wikitext text/x-wiki eftc3zlpy5p3f67bgdapyf32zccncfj 2438382 2438381 2026-07-17T08:18:17Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1335 2438382 wikitext text/x-wiki dnf2c0eai1vvuytnyk34zd3vjiketg9 2438383 2438382 2026-07-17T08:18:46Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1335 2438383 wikitext text/x-wiki 6n2jttjy15r1v18s6dawaxzeg6moz0r 2438384 2438383 2026-07-17T08:18:47Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1335 2438384 wikitext text/x-wiki kmu8spcodre9vc8sbc85v29a8386rz9 2438385 2438384 2026-07-17T08:30:48Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1335.eqiad.wmnet with reason: host reimage 2438385 wikitext text/x-wiki diym6ft7qlqf6whtm2yq3fkah4mdm1o 2438386 2438385 2026-07-17T08:34:29Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1335.eqiad.wmnet with reason: host reimage 2438386 wikitext text/x-wiki qirrn1fcl0xae4n5rl9bltngiz41xst 2438387 2438386 2026-07-17T08:45:06Z Stashbot 7414 jiji@cumin2003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet 2438387 wikitext text/x-wiki 6zuw7qqeapd5nsgg29nyzamzzrq4hme 2438388 2438387 2026-07-17T08:50:11Z Stashbot 7414 jiji@cumin2003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet 2438388 wikitext text/x-wiki dug3gsllpth73mrtnjzau3jn1otfein 2438389 2438388 2026-07-17T08:51:14Z Stashbot 7414 jiji@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet 2438389 wikitext text/x-wiki 10pufy278qdcnofzw9awc2j6wks7kr9 2438390 2438389 2026-07-17T08:51:18Z Stashbot 7414 jiji@cumin2003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet 2438390 wikitext text/x-wiki g8165hune7i1efpcvtnuibkjxl49686 2438391 2438390 2026-07-17T08:52:15Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1335.eqiad.wmnet with OS trixie 2438391 wikitext text/x-wiki gn6g2aly2rzrabbu7q2rty3x2onkt2h 2438392 2438391 2026-07-17T08:54:10Z Stashbot 7414 jiji@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet 2438392 wikitext text/x-wiki i5wt2nx8gdw5s4kyqsjgaa5rjvvt0ub 2438393 2438392 2026-07-17T08:55:20Z Stashbot 7414 jiji@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet 2438393 wikitext text/x-wiki i0r4flypzcxg58rv18uuaa78w8xtvs5 2438394 2438393 2026-07-17T09:04:50Z Stashbot 7414 elukey: uploaded spicerack_13.1.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia 2438394 wikitext text/x-wiki q9r7ntwr7gjg8wxlhw7maot59xbe8xa 2438395 2438394 2026-07-17T09:06:47Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1335.eqiad.wmnet 2438395 wikitext text/x-wiki qbdhn0rr5lih3atlkvufjy0pqm7ebq1 2438396 2438395 2026-07-17T09:06:49Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1335.eqiad.wmnet 2438396 wikitext text/x-wiki eakprevhxyyb9kvm3au44lp0ucj8z1l 2438397 2438396 2026-07-17T09:06:51Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1335.eqiad.wmnet 2438397 wikitext text/x-wiki m8jmzj8alnrquophhpd8n0hxvaujfak 2438398 2438397 2026-07-17T09:08:27Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1336.eqiad.wmnet 2438398 wikitext text/x-wiki 4rhgzimahzzyl9k4fr1lze6il3w2yh8 2438399 2438398 2026-07-17T09:08:31Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1336.eqiad.wmnet 2438399 wikitext text/x-wiki 58xthddhabr074xzp9dnrbl64qu9hx2 2438400 2438399 2026-07-17T09:09:06Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1336.eqiad.wmnet 2438400 wikitext text/x-wiki g6o2ydxahxbvv7j2p2qdqo5gvb7jv3z 2438401 2438400 2026-07-17T09:09:18Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host wikikube-worker1336.eqiad.wmnet with OS trixie 2438401 wikitext text/x-wiki pl8qqikw5xp3zevrqlci5j2hvxmir6i 2438402 2438401 2026-07-17T09:09:48Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1336 2438402 wikitext text/x-wiki cisv54mcrxmqxev9k4677rtoxtofzwd 2438403 2438402 2026-07-17T09:09:57Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2438403 wikitext text/x-wiki 9owdflv33wuojskkmezol60wqcq1jz4 2438404 2438403 2026-07-17T09:10:17Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2438404 wikitext text/x-wiki mmxz4s493xxhmh6wksyux7mostgqdhc 2438405 2438404 2026-07-17T09:11:31Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2438405 wikitext text/x-wiki o29v6j7rcd8pp21sm0bu5eybvu805no 2438406 2438405 2026-07-17T09:14:37Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1336 - cgoubert@cumin2003" 2438406 wikitext text/x-wiki fu1lt7jrcs8mw76r1vqgzplp212826m 2438407 2438406 2026-07-17T09:14:42Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1336 - cgoubert@cumin2003" 2438407 wikitext text/x-wiki 2q272gw8zcl1qy9duabecdkwyxejf2d 2438408 2438407 2026-07-17T09:14:43Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438408 wikitext text/x-wiki bc1pk6pupogmp57ml3ahxxnr6mkbj6t 2438409 2438408 2026-07-17T09:14:44Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache wikikube-worker1336.eqiad.wmnet 152.32.64.10.in-addr.arpa 2.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438409 wikitext text/x-wiki 0m2eve3gsa9kckkvfxklliyxsl6rvu8 2438410 2438409 2026-07-17T09:14:46Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1336.eqiad.wmnet 152.32.64.10.in-addr.arpa 2.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438410 wikitext text/x-wiki rnbdo6sy47u60fvj1syfwn05szzwmiv 2438411 2438410 2026-07-17T09:14:47Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1336 2438411 wikitext text/x-wiki s3asbemuckp2vnpnpxgugyisx62f0x8 2438412 2438411 2026-07-17T09:19:44Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet 2438412 wikitext text/x-wiki 6c1r12ai1loeqr1bgd0cl454qpv2sd1 2438413 2438412 2026-07-17T09:20:07Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1336 2438413 wikitext text/x-wiki 9qawsy2riwfji6eek856u85b7q77cdw 2438414 2438413 2026-07-17T09:20:08Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1336 2438414 wikitext text/x-wiki 6oyh058i4c08ttzdfcjh2cu5ultt3ic 2438415 2438414 2026-07-17T09:21:09Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet 2438415 wikitext text/x-wiki 3rmn284c1h35g3cue2vvfaolimlcjhr 2438416 2438415 2026-07-17T09:26:52Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2438416 wikitext text/x-wiki s58u3gasucttv3ayick35tfgqwpoaxm 2438417 2438416 2026-07-17T09:28:29Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2438417 wikitext text/x-wiki pv462fqcrz6c1m29b758q7s93d9vjr2 2438418 2438417 2026-07-17T09:29:17Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1002.eqiad.wmnet 2438418 wikitext text/x-wiki ocmf6cynld9t7qnuz8ok0wqtjumcp5g 2438419 2438418 2026-07-17T09:29:23Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1002.eqiad.wmnet 2438419 wikitext text/x-wiki twb6kd6on0ut842minoxfwtcumfule8 2438420 2438419 2026-07-17T09:33:21Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1336.eqiad.wmnet with reason: host reimage 2438420 wikitext text/x-wiki c9jn37s13wn9aav91nh8i5aoxaghtxo 2438421 2438420 2026-07-17T09:39:25Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1336.eqiad.wmnet with reason: host reimage 2438421 wikitext text/x-wiki coi96xcyhnjp3vnig7u4sqw24zhhmsh 2438422 2438421 2026-07-17T09:40:07Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.bmc-user-mgmt for host acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet,an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet,db-test[2001-2002].codfw.wmnet,db-test[1001-1003].eqiad.wmn 2438422 wikitext text/x-wiki gwx0vb0jtgw2us94cu70gadxkrohwrn 2438423 2438422 2026-07-17T09:45:10Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet,an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet,db-test[2001-2002].codfw.wmnet,db-test[1 2438423 wikitext text/x-wiki ozoenn1k6ld70dsnla5socpicjmr64y 2438424 2438423 2026-07-17T09:47:46Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts (check the cookbook's logs for more details.) 2438424 wikitext text/x-wiki 2e3cdfh836w40i689qphlg4zie469o2 2438425 2438424 2026-07-17T09:47:51Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts (check the cookbook's logs for more details.) 2438425 wikitext text/x-wiki miuwjwpiix1z3npt5qwz0h9f35f9cpi 2438426 2438425 2026-07-17T09:58:13Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1336.eqiad.wmnet with OS trixie 2438426 wikitext text/x-wiki k96dhczfif5s6ikb2qb9ddr74ivbgag 2438427 2438426 2026-07-17T10:03:40Z Stashbot 7414 elukey@cumin2002: START - Cookbook sre.hosts.bmc-user-mgmt for 13 hosts (check the cookbook's logs for more details.) 2438427 wikitext text/x-wiki rhmji013ox04lxspxf5tvfpxmpwmbsj 2438428 2438427 2026-07-17T10:05:16Z Stashbot 7414 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 13 hosts (check the cookbook's logs for more details.) 2438428 wikitext text/x-wiki 7qzmglk78cjy7hmvbeqe5m3o6hn8z6e 2438429 2438428 2026-07-17T10:09:40Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db1176.eqiad.wmnet 2438429 wikitext text/x-wiki d9luerlsg6k1hadciqu1iaxeoulx18n 2438430 2438429 2026-07-17T10:10:09Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db1176.eqiad.wmnet 2438430 wikitext text/x-wiki 0kxq2tz8prh9npwwih11cltr4lpwjfu 2438431 2438430 2026-07-17T10:11:29Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db1176.eqiad.wmnet 2438431 wikitext text/x-wiki nutojlahdkw2bytl57jnu6shjktgkza 2438432 2438431 2026-07-17T10:15:09Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1336.eqiad.wmnet 2438432 wikitext text/x-wiki pli421ng0sj408ydj5wcxfd1zje6947 2438433 2438432 2026-07-17T10:15:11Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1336.eqiad.wmnet 2438433 wikitext text/x-wiki 8uudav6mcr4cstel4a1q0mundpn06u2 2438434 2438433 2026-07-17T10:15:12Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1336.eqiad.wmnet 2438434 wikitext text/x-wiki 84hcb0njs62w0zq7gxdntcqnlpjono3 2438435 2438434 2026-07-17T10:16:43Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1337.eqiad.wmnet 2438435 wikitext text/x-wiki jmmptj4pdqp6i081p7vyngk1bhu36n3 2438436 2438435 2026-07-17T10:16:46Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1337.eqiad.wmnet 2438436 wikitext text/x-wiki gvwn54vzxfv0tgyvq8pblakuprsd0d1 2438437 2438436 2026-07-17T10:17:00Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db1176.eqiad.wmnet 2438437 wikitext text/x-wiki eoyo2nlj315vvhhizf8wnfhbrq6xvml 2438438 2438437 2026-07-17T10:17:21Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1337.eqiad.wmnet 2438438 wikitext text/x-wiki 7fwlzzhbh8tcjnlyw6or7tn925xua2c 2438439 2438438 2026-07-17T10:17:57Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host wikikube-worker1337.eqiad.wmnet with OS trixie 2438439 wikitext text/x-wiki 38bdtynksv3pn1je04pgpmzgzoy1rjd 2438440 2438439 2026-07-17T10:18:27Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1337 2438440 wikitext text/x-wiki c3e5xqkdh3rx96mjmdiiomld7zmsc3j 2438441 2438440 2026-07-17T10:21:48Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2438441 wikitext text/x-wiki 2d9l1pp5am1biea4d7vg7btmhwmixze 2438443 2438441 2026-07-17T10:26:23Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1337 - cgoubert@cumin2003" 2438443 wikitext text/x-wiki tk2ytie7gjh3elyxjwq7b8mfrwar059 2438444 2438443 2026-07-17T10:26:27Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1337 - cgoubert@cumin2003" 2438444 wikitext text/x-wiki laci5wmcqrcwt8ztpfjkj2dhmzd9jko 2438445 2438444 2026-07-17T10:26:29Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438445 wikitext text/x-wiki 824uf5cirvhmmu2pt8jqfwha0vvjgdd 2438446 2438445 2026-07-17T10:26:30Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache wikikube-worker1337.eqiad.wmnet 154.32.64.10.in-addr.arpa 4.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438446 wikitext text/x-wiki shbszkd564wdmjj726xkr2o46vtroez 2438447 2438446 2026-07-17T10:26:31Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1337.eqiad.wmnet 154.32.64.10.in-addr.arpa 4.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438447 wikitext text/x-wiki n2ulpv979xqgmentz2cjt7nedquhhhs 2438448 2438447 2026-07-17T10:26:33Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1337 2438448 wikitext text/x-wiki 7olz2xa7mq8ot27vkycv67cfamud17w 2438449 2438448 2026-07-17T10:27:03Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1337 2438449 wikitext text/x-wiki 9xkcnx4yzs064sipgoqvp8a3hy3t0r7 2438450 2438449 2026-07-17T10:27:04Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1337 2438450 wikitext text/x-wiki 9m00phgyywedist5u0sqbkrlk4mb00n 2438451 2438450 2026-07-17T10:27:45Z Stashbot 7414 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2438451 wikitext text/x-wiki bozxn2ppwr9zsruy7kwwlhdac8ci4qv 2438452 2438451 2026-07-17T10:28:03Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1001-1003].eqiad.wmnet 2438452 wikitext text/x-wiki q52t8hjlwjmx4p4v1cx1catfpkvpsdc 2438453 2438452 2026-07-17T10:30:31Z Stashbot 7414 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2438453 wikitext text/x-wiki 46g8q68zf7z94vn04gjxd8cagcdmenh 2438454 2438453 2026-07-17T10:34:30Z Stashbot 7414 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2438454 wikitext text/x-wiki q8eek3odowyow0h2ivd70eufixuyvyr 2438455 2438454 2026-07-17T10:34:33Z Stashbot 7414 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2438455 wikitext text/x-wiki p40d9mq6iy0em523e7157dnut1g5g57 2438456 2438455 2026-07-17T10:39:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1001-1003].eqiad.wmnet 2438456 wikitext text/x-wiki 15tz4qgroj07gpgoptc0xl42nzqrm0b 2438457 2438456 2026-07-17T10:39:21Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1337.eqiad.wmnet with reason: host reimage 2438457 wikitext text/x-wiki q1u7xsk0240fbp53j4bmecflaefxla8 2438458 2438457 2026-07-17T10:39:26Z Stashbot 7414 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2438458 wikitext text/x-wiki 5rvsjidut7i5vm0toycxy094xu03w83 2438459 2438458 2026-07-17T10:40:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet 2438459 wikitext text/x-wiki k3u3qwf672es7dgvb2jq67g6zqlcnl9 2438460 2438459 2026-07-17T10:43:45Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1337.eqiad.wmnet with reason: host reimage 2438460 wikitext text/x-wiki jifrj3p8s3qrj9oaxtccaf9gl1yjv2f 2438461 2438460 2026-07-17T10:50:04Z Stashbot 7414 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2438461 wikitext text/x-wiki qxztnq6yczps66xx7db5cuws07nrf1x 2438462 2438461 2026-07-17T10:51:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[2001-2002].codfw.wmnet 2438462 wikitext text/x-wiki 1ev0j88l0hbmy0kb09mmsszk4p1tvie 2438463 2438462 2026-07-17T11:02:39Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1337.eqiad.wmnet with OS trixie 2438463 wikitext text/x-wiki p3ewlr2zn539zwqbuli751xj0knndtg 2438464 2438463 2026-07-17T11:17:59Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1337.eqiad.wmnet 2438464 wikitext text/x-wiki 66li4ylrenpnlziejr2ud4fdja61qsm 2438465 2438464 2026-07-17T11:18:01Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1337.eqiad.wmnet 2438465 wikitext text/x-wiki 81jp7jxgkdldu9n2nxzwbogq06euqwh 2438466 2438465 2026-07-17T11:18:03Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1337.eqiad.wmnet 2438466 wikitext text/x-wiki me6tnx4uki519hcx1nynsijb04y0x5h 2438467 2438466 2026-07-17T11:19:05Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1338.eqiad.wmnet 2438467 wikitext text/x-wiki gbg9f0ntxqheid8hiv2jpywy6iy4rgg 2438468 2438467 2026-07-17T11:19:09Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1338.eqiad.wmnet 2438468 wikitext text/x-wiki p05hc3nex5o5ko9hn1t72fdth3l3m09 2438469 2438468 2026-07-17T11:19:44Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1338.eqiad.wmnet 2438469 wikitext text/x-wiki o4hjmrsnf46xnh0nr7bh9kn5l63unbo 2438470 2438469 2026-07-17T11:20:25Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host wikikube-worker1338.eqiad.wmnet with OS trixie 2438470 wikitext text/x-wiki luiwo146uns0qci84xfps1dvhivuabq 2438471 2438470 2026-07-17T11:20:43Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1338 2438471 wikitext text/x-wiki k21wltxkkftapu2krcc7dgyxopqkj5b 2438472 2438471 2026-07-17T11:20:58Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.netbox 2438472 wikitext text/x-wiki d5gfeu6ttbxzpedssienb9ri2g19tkv 2438475 2438472 2026-07-17T11:24:00Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db2230.codfw.wmnet 2438475 wikitext text/x-wiki mmnzh0152otl6q1lrpra5l7bfo5ghul 2438477 2438475 2026-07-17T11:25:32Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1338 - cgoubert@cumin2003" 2438477 wikitext text/x-wiki 3ma0zu2ok3g5qp12tixmch1vtr44u9z 2438478 2438477 2026-07-17T11:25:36Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1338 - cgoubert@cumin2003" 2438478 wikitext text/x-wiki ej6wge76e635mn68h2h8ziuv0gnttaz 2438479 2438478 2026-07-17T11:25:38Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2438479 wikitext text/x-wiki 2hkdv0fyj4sh9qr86bjvr2op0nmn7oq 2438480 2438479 2026-07-17T11:25:39Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache wikikube-worker1338.eqiad.wmnet 155.32.64.10.in-addr.arpa 5.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438480 wikitext text/x-wiki e1k0bk3v7nkpnjqlktgott7ivyb6qld 2438481 2438480 2026-07-17T11:25:41Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1338.eqiad.wmnet 155.32.64.10.in-addr.arpa 5.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2438481 wikitext text/x-wiki tr3np6snpqwgo7enceap8ykxw82lphf 2438482 2438481 2026-07-17T11:25:42Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1338 2438482 wikitext text/x-wiki m931honp8i19xk4si7i46s3syg8yvi4 2438483 2438482 2026-07-17T11:26:07Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1338 2438483 wikitext text/x-wiki do9rmh64bv1ssxu5hm3qbegvhx1wht7 2438484 2438483 2026-07-17T11:26:08Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1338 2438484 wikitext text/x-wiki dw28eun7hnae8ycs1xi3c4qre423uq5 2438486 2438484 2026-07-17T11:31:53Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db2230.codfw.wmnet 2438486 wikitext text/x-wiki f3skt238bl9t188i393es0tpbhdi78q 2438487 2438486 2026-07-17T11:38:35Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1338.eqiad.wmnet with reason: host reimage 2438487 wikitext text/x-wiki p3cn148941l0f6et9c7k35xun48u0wr 2438488 2438487 2026-07-17T11:42:26Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1338.eqiad.wmnet with reason: host reimage 2438488 wikitext text/x-wiki b45wyoog2qpdspyz6efpc2vjm4bbvsf 2438489 2438488 2026-07-17T11:51:37Z Stashbot 7414 klausman@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2438489 wikitext text/x-wiki imj7k38zq1x8onsxiir9e4k5a1ca1jc 2438490 2438489 2026-07-17T11:53:04Z Stashbot 7414 klausman@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2438490 wikitext text/x-wiki tvb00exgagnr0n175jb9izeo551ynwk 2438491 2438490 2026-07-17T11:54:59Z Stashbot 7414 klausman@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2438491 wikitext text/x-wiki a0phurjpvj2qz3boiusvdnz3noihy9n 2438492 2438491 2026-07-17T11:56:14Z Stashbot 7414 klausman@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2438492 wikitext text/x-wiki t5mumuk5q2re6gak9hjflucxo7wc7xl 3 10603 2438473 95174 2026-07-17T11:22:21Z StrikerBot 8475 Welcome to Toolforge! 2438473 wikitext text/x-wiki dqnnd64yix67zrjeis36tne5mp6diky 3 16287 2438476 121479 2026-07-17T11:24:34Z StrikerBot 8475 Welcome to Toolforge! 2438476 wikitext text/x-wiki ixp6on3x8pva16g9jxfw0w1kqwsnr38 0 17290 2437686 2437289 2026-07-16T12:10:50Z Stashbot 7414 Krinkle: krinkle@contint1002: zuul restart 2437686 wikitext text/x-wiki jzhr5ylm7hxqpqov38xsoyvt5u2jgga 2437698 2437686 2026-07-16T12:31:32Z Stashbot 7414 Krinkle: krinkle@contint1003:~$ sudo /usr/sbin/service jenkins restart 2437698 wikitext text/x-wiki oe1axna1tspo1qlcmbxvgm0ag7btwnv 2438002 2437698 2026-07-16T16:32:46Z Stashbot 7414 dancy: Restarted docker on contint1003 2438002 wikitext text/x-wiki 1x7w0nozl6to9krbs8ykju3tu3v1jd7 2438003 2438002 2026-07-16T16:33:04Z Stashbot 7414 dancy: Restarted docker on contint1003 (T432326) 2438003 wikitext text/x-wiki 57b0q4lb1xz8nvd47gln7652rt2k7zs 2438004 2438003 2026-07-16T16:34:19Z Stashbot 7414 dancy: Restarted docker on contint2003 (T432326) 2438004 wikitext text/x-wiki drlqcpi7ze69uh2z5u76cs7tx9c9ho5 2438013 2438004 2026-07-16T16:40:40Z Stashbot 7414 brennen: contint1002: chown and chmod on /srv/zuul/git/mediawiki/extensions/CampaignEvents/.git per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions 2438013 wikitext text/x-wiki fuomfxas77jk5m2r39803s1j122my6l 2438045 2438013 2026-07-16T17:07:06Z Stashbot 7414 bd808: Hard reboot of deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud via Horizon; console shows OOM (T432374) 2438045 wikitext text/x-wiki gig2pvg5m874pl0rvnp162ithh3e28g 2438112 2438045 2026-07-16T18:33:38Z Stashbot 7414 dancy: Jenkins placed in shutdown mode in preparation for a restart 2438112 wikitext text/x-wiki ilhgmf8rollqone55atigj1des3th59 2438113 2438112 2026-07-16T18:37:04Z Stashbot 7414 dancy: Restarted Jenkins to unstick jobs. 2438113 wikitext text/x-wiki c75y1xo8d78xiehfh8p53yht2j78whj 2438114 2438113 2026-07-16T18:51:30Z Stashbot 7414 dancy: Updated buildkit to v0.31.2 gitlab-cloud-runners (staging and production) (T432360) 2438114 wikitext text/x-wiki qx7iyfch1allbl5lef4ig3xy9pvrv4a 2438285 2438114 2026-07-16T23:07:00Z Stashbot 7414 mutante: gerrit1003/2002/2003: rm /srv/gerrit/.ssh/config and revert gerrit:1311577 which fixed T398401 but caused T432413 - replication is working again 2438285 wikitext text/x-wiki fp2myqyyjv5vlb72kk9w359geb49e35 498 30942 2437829 2436936 2026-07-16T14:17:13Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2437829 wikitext text/x-wiki j2ackbx3pgdxov26wabfcygbck8lwk1 2437830 2437829 2026-07-16T14:17:45Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2437830 wikitext text/x-wiki jm3n9mqfwtncgh4ongd5w3le7nnhn0p 0 32278 2438170 2436172 2026-07-16T20:03:35Z BCornwall-WMF 31521 /* Emergency failover */ Change note to use the warn template 2438170 wikitext text/x-wiki 19r2bfjgp8odlmi9pfhdwjz2brq7pwf 0 442261 2438280 2427073 2026-07-16T22:55:12Z BryanDavis 1604 /* Updating the demo build */ update docs for changes that happened a loooong time ago 2438280 wikitext text/x-wiki 2c6wmql09gn8xszgf28kcqtnha323sy 0 442273 2438054 2361737 2026-07-16T17:12:22Z Taavi 13997 /* Metricsinfra Prometheus */ controller-3 2438054 wikitext text/x-wiki is2wkrwdwvz5jgvxfrvy89rxnmyaa0l 498 443946 2438289 2433173 2026-07-16T23:08:54Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 418e934c5a (update Manbhumi item ID after merge) [actually deployed 30878c6831, temporary T431146 branch rebased on top of that) 2438289 wikitext text/x-wiki 6y9xitz3w5fpez25hxbcnepui4ae6t4 0 446030 2437812 2430919 2026-07-16T14:11:58Z BKing (WMF) 29017 /* Clean up object storage */ 2437812 wikitext text/x-wiki shqmhtfbt9fxr3bausohe6z1dyklpb5 498 447152 2438278 2427576 2026-07-16T22:44:32Z Stashbot 7414 bd808: Hared reboot of toolhub-demo02.toolhub.eqiad1.wikimedia.cloud via Horizon; instance seemed to have dropped off of network following a live migration on 2026-07-06. 2438278 wikitext text/x-wiki cbjdd19fux3i8epvzn5glg8rkuehhvb 0 449160 2438164 2437516 2026-07-16T19:58:23Z Dexbot 30554 Bot: Updating the report 2438164 wikitext text/x-wiki dp1eiufkkythvehf1ooasu4iq2ksan6 2438320 2438164 2026-07-17T00:01:53Z Dexbot 30554 Bot: Updating the report 2438320 wikitext text/x-wiki pe2geb9g0c2hz2c8jmfp5nc1hdhohue 0 450684 2437717 2424014 2026-07-16T12:56:09Z Adam Wight (WMDE) 3064 /* How to provision a development environment */ fix minor typos 2437717 wikitext text/x-wiki f1fvdcbf5aa99xsnyaa5m5254w1neuy 0 451719 2438123 2382009 2026-07-16T19:13:42Z SFrench-WMF 37362 /* to scap */ Update guidance on how to add a new deployment to scap 2438123 wikitext text/x-wiki cdc90zj4dp5m9o9oslylbki946k9fu7 0 455989 2438104 2433397 2026-07-16T18:07:08Z JVanderhoop-WMF 41069 Removed reference to experiment scorecard 2438104 wikitext text/x-wiki sm9g3aw2f2dr0or6xut1fhqnhzpo9c2 498 458879 2437795 2432904 2026-07-16T13:55:59Z Stashbot 7414 Raymond_Ndibe: increased the object count quota to 500k T431248 2437795 wikitext text/x-wiki 003xazobj2k6rtebl0ngcwiw1mnzzzk 0 459699 2438337 2437637 2026-07-17T01:25:45Z Stashbot 7414 eileen: civicrm upgraded from 1c94b489 to dd65e1b6 2438337 wikitext text/x-wiki cp07esu10e1lzn3y2r6c3lukhq19ykq 2438345 2438337 2026-07-17T03:44:20Z Stashbot 7414 eileen: civicrm upgraded from dd65e1b6 to 2eba4374 2438345 wikitext text/x-wiki fez61vc771fqu7njbm9uhzpob99f96c 0 460233 2437789 2432971 2026-07-16T13:54:31Z MPopov (WMF) 5891 Update references to (Web) Actions fact table 2437789 wikitext text/x-wiki i17q0b39bq6p0ead3s8unkd857gkn0h 2438294 2437789 2026-07-16T23:13:15Z MPopov (WMF) 5891 /* Metric definitions should be agnostic of group/variation */ 2438294 wikitext text/x-wiki mg553mhcbdsrpmydoq0i97xkrjmm2qf 0 460234 2438076 2432383 2026-07-16T17:30:48Z KReid-WMF 49588 /* Create/schedule an experiment */ Update STATUS note to reflect that GB is now the prefered configuration source. 2438076 wikitext text/x-wiki rv4kc9w0s4xa9turfulh2l13gmhc39g 0 460336 2438200 2430960 2026-07-16T20:45:16Z ADancy (WMF) 17694 Add GerritLab notes 2438200 wikitext text/x-wiki 2qu89s3qoiemo5wpc74ifx661jx3xlf 0 460366 2437957 2433185 2026-07-16T15:48:37Z TBurmeister (WMF) 27417 Add category 2437957 wikitext text/x-wiki o97wo6rpd6ys8kfyxzztvrfabb2o8g5 0 460375 2437958 2435978 2026-07-16T15:49:02Z TBurmeister (WMF) 27417 Add category 2437958 wikitext text/x-wiki o3tvxjertb39thx2ggckl1fjkb93kwx 0 460376 2437960 2435513 2026-07-16T15:49:22Z TBurmeister (WMF) 27417 Add category 2437960 wikitext text/x-wiki dxm9nkbmi1zpa4pdzrmc7anq5no0g49 0 460377 2437961 2435963 2026-07-16T15:49:42Z TBurmeister (WMF) 27417 Add category 2437961 wikitext text/x-wiki 9bv69c8zweg3yqpgr8lm8hkvdaclt19 2 460448 2438085 2026-07-16T17:36:49Z Taavi 13997 Created page with "There are about two primary user account systems in the Wikimedia circles: == Global accounts == '''aka:''' Wikimedia account, SUL account, Single User Login account, Central account, Wikipedia account, Unified login/account This is the account system used to log in to ~all public Wikimedia wikis. MediaWiki, our wiki software, did not originally have a way to share accounts between individual wikis. This was sub-optimal,{{Citation needed}} so we wrote a MediaWiki ext..." 2438085 wikitext text/x-wiki amzyyo0qmm9ti1pg20csgq5bnnw52ix 2 460449 2438279 2026-07-16T22:46:39Z Mmmwdhub 57735 https://mmmwdhub.org 2438279 wikitext text/x-wiki 5lkdblnoj46bvxqp0es26o9l19c6diz 2438284 2438279 2026-07-16T23:04:43Z NDG 39502 Requesting deletion 2438284 wikitext text/x-wiki 0r34wb0t28e2fqpurp9qhberv6omo1j 3 460450 2438346 2026-07-17T04:41:24Z Mmmwdhub 57735 /* Mushi Master Mind World Digital Hub - MMMWD Hub: MMMWD Hub Category: Global Multi-Sector Innovation Company . Clean Energy . Electric Mobility . Smart Cities . Rural Development */ new section 2438346 wikitext text/x-wiki 8z5fv5vy8fu287c1lljdhqyqsjyz8fr 3 460451 2438474 2026-07-17T11:22:41Z StrikerBot 8475 Welcome to Toolforge! 2438474 wikitext text/x-wiki 976owb76dciyxru664c8x6q60w203e7 3 460452 2438485 2026-07-17T11:26:32Z StrikerBot 8475 Welcome to Toolforge! 2438485 wikitext text/x-wiki 6enhx47jytzt67615jyyu1niqmkbwrb