Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.10 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 739 2433921 2349339 2026-07-07T17:38:54Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2433921 wikitext text/x-wiki 3t4as1njghsmydf1hkn8gj1l27t6xe2 2433927 2433921 2026-07-07T17:47:57Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) 2433927 wikitext text/x-wiki q6kfn58u8vipvt4hzmj4g611achlsai 2433929 2433927 2026-07-07T17:52:11Z Stashbot 7414 andrewbogott: replacing dumps-nfs-1 server with dumps-nfs-2 2433929 wikitext text/x-wiki o4epk8lkdx3njsdbyv2c4o5ndyjtq0a 2433930 2433929 2026-07-07T17:52:20Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service 2433930 wikitext text/x-wiki 7wyzgu22vfpr6r60vu00ice2r7pzfbx 2433931 2433930 2026-07-07T17:52:29Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) 2433931 wikitext text/x-wiki 9pe8uebbdar012fmk9ikd40ok4gp1tg 2433933 2433931 2026-07-07T17:54:38Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service 2433933 wikitext text/x-wiki gee5m4kv5goo95mjiqp16ux7ikchwjt 2433934 2433933 2026-07-07T17:57:07Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) 2433934 wikitext text/x-wiki sg5uawdat9c01gjr3482cv74n2hs7m5 498 3086 2433870 2433392 2026-07-07T15:37:25Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2433870 wikitext text/x-wiki ofwgn0ozx7p5a9y48md2jwne4e7ywgz 2433875 2433870 2026-07-07T15:51:22Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2433875 wikitext text/x-wiki i9yxq78605laajbe8fq3j2osfvomf67 2433888 2433875 2026-07-07T16:36:12Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2433888 wikitext text/x-wiki jcjtrba1c5xbj23us3jjx6vyjho76eg 2433892 2433888 2026-07-07T16:51:36Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2433892 wikitext text/x-wiki ot0g96367bknndrz43sg8d9av36b9yp 2433911 2433892 2026-07-07T17:08:16Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission 2433911 wikitext text/x-wiki p4txswz8jcrl274ycc30d9zuw05868m 2433915 2433911 2026-07-07T17:17:24Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission 2433915 wikitext text/x-wiki 3hoerie3fssf58h4v6mo94z9i44yl01 2434103 2433915 2026-07-08T08:55:40Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2434103 wikitext text/x-wiki lbh6rhr9tpircombxfdkl9jao1pvg2u 2434108 2434103 2026-07-08T09:08:31Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2434108 wikitext text/x-wiki mo74q7qu1zeqepvsbpnymjt95y85amx 0 3205 2433893 2431988 2026-07-07T16:53:52Z BCornwall-WMF 31521 /* Please read before we get started... */ move heading to warn template in attempt to clean up heading spam 2433893 wikitext text/x-wiki deg39ruvr8ovas98nkzphjruq41fh8x 2433896 2433893 2026-07-07T16:55:20Z BCornwall-WMF 31521 /* Remove the service from the load-balancers and the backend servers */ Fix mistaken double-mention of low-traffic in example 2433896 wikitext text/x-wiki ksioilgwmbqh5z3p22x1ex7ivmcp2jw 2433913 2433896 2026-07-07T17:15:20Z BCornwall-WMF 31521 Add something along the lines of a cleanup template 2433913 wikitext text/x-wiki hbbiqr3arhp55w6ds4uuaes3008f2ky 2 3849 2434033 2117352 2026-07-08T05:15:21Z ArielGlenn 11 placeholder for drafts of docs 2434033 wikitext text/x-wiki e2wwa8wfdurxpix3salkxhsam23e7e4 0 4108 2433882 2433602 2026-07-07T16:19:59Z Dreamy Jazz 31583 Move IPoid decommissioning to Thursday due to conflicts with Jenkins being offline 2433882 wikitext text/x-wiki 9nlmo2xhyvnrp1731cfis6q3x7pyf6p 2433902 2433882 2026-07-07T16:57:03Z KSoučková-WMF 35993 2433902 wikitext text/x-wiki l34b5yxlgqy0ypkhbf83r6ku9lcucg3 2433938 2433902 2026-07-07T18:19:10Z Brennen 54136 /* {{Deployment_day|date=2026-07-07}} */ Update train conductor info. 2433938 wikitext text/x-wiki n98eeqrvf885rnar0c4y20nt13n40y8 2433939 2433938 2026-07-07T18:20:21Z Brennen 54136 /* {{Deployment_day|date=2026-07-08}} */ Update train conductors. 2433939 wikitext text/x-wiki a2kestslckp3i3ooiqzrpwuv7qett39 2433940 2433939 2026-07-07T18:21:13Z Brennen 54136 /* {{Deployment_day|date=2026-07-09}} */ Update train conductors. 2433940 wikitext text/x-wiki f1he00xjwgrqyfrbqad9blxslmcyyoy 2433968 2433940 2026-07-07T20:11:13Z ScheduleDeploymentBot 37566 Add [[gerrit:1308196]] to Tuesday, July 07 UTC late backport window 2433968 wikitext text/x-wiki 7tefhdpx6eqdt9w1ee2xqb1nwo8i1he 2434063 2433968 2026-07-08T08:07:14Z ScheduleDeploymentBot 37566 Add [[gerrit:1308553]] to Wednesday, July 08 UTC afternoon backport window 2434063 wikitext text/x-wiki k4w451cjv6ocw8onn5k2jo0cfv0w564 2434065 2434063 2026-07-08T08:08:46Z Hashar 32 /* Week of July 06 */ fix up train windows, they are run in the morning, blocker is T430829 2434065 wikitext text/x-wiki 0tils5333025igfq8vf2byi6yfizm83 498 6726 2433863 2433388 2026-07-07T15:25:01Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2433863 wikitext text/x-wiki sv4i1xj63mi8hvzz4mixz589p6ek2nf 2433869 2433863 2026-07-07T15:37:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2433869 wikitext text/x-wiki lif9sedsj99uhklf4zss9zk0pzfm4cq 2433884 2433869 2026-07-07T16:23:25Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2433884 wikitext text/x-wiki nvvz2trsywto38tvc0709sz434cm7bq 2433886 2433884 2026-07-07T16:35:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2433886 wikitext text/x-wiki nppahektwd2ke0jaxazv8g1xqhe0tdq 2433897 2433886 2026-07-07T16:55:33Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api 2433897 wikitext text/x-wiki ceobhybla196es18ds8e33dvfjj8uh4 2433898 2433897 2026-07-07T16:55:38Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api 2433898 wikitext text/x-wiki j2ibqgestso8atf6j56sd4e2vuto02w 2433899 2433898 2026-07-07T16:55:48Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api 2433899 wikitext text/x-wiki 7gim11696qp0vyvx32ysx5xrdnx89lp 2433900 2433899 2026-07-07T16:56:23Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api 2433900 wikitext text/x-wiki qdv20dmpnzyxtw90h3m49itmz6w24ss 2433901 2433900 2026-07-07T16:56:41Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission 2433901 wikitext text/x-wiki ccyyhmbk5wsg8tuhmfa7x88fv1q8iye 2433908 2433901 2026-07-07T17:05:11Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission 2433908 wikitext text/x-wiki n28c2drjpdq2kgeeuto1rio5bc1ophu 2434078 2433908 2026-07-08T08:24:13Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2434078 wikitext text/x-wiki 9nuil15k4uhn7f0xxt0idaovjovcvok 2434085 2434078 2026-07-08T08:36:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2434085 wikitext text/x-wiki oluwml2a0fcsk310dv96c83bqe0gkg2 0 7919 2433624 2433623 2026-07-07T11:59:34Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl1001.eqiad.wmnet 2433624 wikitext text/x-wiki jqmpv42xriodsrxn3npx5gzvj4wjoek 2433625 2433624 2026-07-07T11:59:36Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl1001.eqiad.wmnet 2433625 wikitext text/x-wiki 734yt6yeov2gtuzz177759rzj327o5q 2433626 2433625 2026-07-07T11:59:43Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl1002.eqiad.wmnet 2433626 wikitext text/x-wiki e5frrojqqxtihqjus6n6d1u7xuxqepn 2433627 2433626 2026-07-07T11:59:44Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl1002.eqiad.wmnet 2433627 wikitext text/x-wiki b8b8o05cty1tnr7uvgjtytpv8cmwdx9 2433628 2433627 2026-07-07T12:03:09Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433628 wikitext text/x-wiki 95bb7m8igzw0sap35gptgbhmam4lvk9 2433629 2433628 2026-07-07T12:03:10Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) 2433629 wikitext text/x-wiki 8fq9lq9q88cnorp9gs0i2fhyzeawf65 2433630 2433629 2026-07-07T12:03:39Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433630 wikitext text/x-wiki 7qva86kbhl1g8o5foc3h9bqcd8inac3 2433631 2433630 2026-07-07T12:03:40Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) 2433631 wikitext text/x-wiki pb8w0pybw73406o096dji52tb7tf5k4 2433632 2433631 2026-07-07T12:04:43Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433632 wikitext text/x-wiki dg1zqtwfdb3xb8039m2eqwaz0r9u40i 2433633 2433632 2026-07-07T12:04:44Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) 2433633 wikitext text/x-wiki k50r6zoiigcmrr0ku80t0lwzoqwzepz 2433634 2433633 2026-07-07T12:04:53Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl1002.eqiad.wmnet 2433634 wikitext text/x-wiki f18znj9uq5f1vbp9g49rqg029r2a6u8 2433635 2433634 2026-07-07T12:04:55Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl1002.eqiad.wmnet 2433635 wikitext text/x-wiki ga9pztdxwygfeqd9o9lvc8xpoh2e73w 2433636 2433635 2026-07-07T12:04:56Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-master-eqiad 2433636 wikitext text/x-wiki h1trfrj4nv47dujrttiwvn86sgh8gqh 2433637 2433636 2026-07-07T12:05:25Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433637 wikitext text/x-wiki 656u20nnq765lvj2t27zvk7kyfpj30g 2433638 2433637 2026-07-07T12:05:26Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) 2433638 wikitext text/x-wiki 9au9q98vqwrg0mig7bxid1kng4kle3n 2433639 2433638 2026-07-07T12:06:04Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433639 wikitext text/x-wiki 2we3s5nphctrpt0cpmhw9ofkimln4dw 2433640 2433639 2026-07-07T12:06:05Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) 2433640 wikitext text/x-wiki j4i8274bn89uhdyt21wd7y2bl00zpga 2433641 2433640 2026-07-07T12:06:10Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433641 wikitext text/x-wiki d8p3zu388v2yrrmz7af4ems60nft9xh 2433642 2433641 2026-07-07T12:07:29Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2005.codfw.wmnet with reason: host reimage 2433642 wikitext text/x-wiki 7xgr1fcpzj3qeh75301tytt7jmk9l43 2433643 2433642 2026-07-07T12:07:37Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) 2433643 wikitext text/x-wiki k7fwbn7wxrmb6a18ukxvb3cnq5rn1ao 2433644 2433643 2026-07-07T12:08:19Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433644 wikitext text/x-wiki 76dkw40xuqp7o5my47ukxh444j1tgi4 2433645 2433644 2026-07-07T12:14:18Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) 2433645 wikitext text/x-wiki s5aoveocvwsui1wpvklbfy92yb76rym 2433646 2433645 2026-07-07T12:14:29Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2005.codfw.wmnet with reason: host reimage 2433646 wikitext text/x-wiki rleddcgzba3tnta3c10myhs1yn5shud 2433647 2433646 2026-07-07T12:14:57Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433647 wikitext text/x-wiki fxmynu1mo1gs8r2ciyr1j8l1zufq48m 2433649 2433647 2026-07-07T12:19:39Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) 2433649 wikitext text/x-wiki b8fzaqm0g618mtzm2jzrfizmmedxkfi 2433650 2433649 2026-07-07T12:20:05Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433650 wikitext text/x-wiki d90dwaw0xxdof49mi21vdsjopropgvv 2433651 2433650 2026-07-07T12:21:07Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: restarting for replication filter 2433651 wikitext text/x-wiki 8x95xdrs06rj3s8l6tmnfl3knt6mlab 2433652 2433651 2026-07-07T12:21:15Z Stashbot 7414 marostegui: Restart mariadb@s7 on db1155 to pick up new filters - T431124 2433652 wikitext text/x-wiki etfcfcqghnvd4x432gcv7lh7vx8mcjn 2433653 2433652 2026-07-07T12:22:10Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1036.eqiad.wmnet 2433653 wikitext text/x-wiki jq8e4zodntnqakk9hlc05g7cdcw42ll 2433654 2433653 2026-07-07T12:22:14Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet 2433654 wikitext text/x-wiki spnig6y9ca3eo2p9dnauygykjs2b6aw 2433655 2433654 2026-07-07T12:22:20Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1082.eqiad.wmnet with OS trixie 2433655 wikitext text/x-wiki 0vran5ewmke5tsi9pct9djg3t74nw3v 2433656 2433655 2026-07-07T12:22:49Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet 2433656 wikitext text/x-wiki lzfk7f3k5ixfu1kico69w9w5oupgb03 2433657 2433656 2026-07-07T12:23:02Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1036.eqiad.wmnet with OS trixie 2433657 wikitext text/x-wiki gitmfmkjvyfxp83z3c0a0kfje6i9ykc 2433658 2433657 2026-07-07T12:23:31Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1036 2433658 wikitext text/x-wiki mrvl8u92nnxo47rmp399m0lqoajcxd8 2433659 2433658 2026-07-07T12:23:56Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.netbox 2433659 wikitext text/x-wiki 07h0b565dyb3p8wmyj13717x173yz6i 2433660 2433659 2026-07-07T12:26:07Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) 2433660 wikitext text/x-wiki h0ro1r5il9oki74iy3t6herbmcdc6sq 2433661 2433660 2026-07-07T12:26:31Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.wdqs.restart 2433661 wikitext text/x-wiki 3jh3ctjcsxc6sd731cdwog8rjgjjxsr 2433662 2433661 2026-07-07T12:28:26Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2433662 wikitext text/x-wiki a1uk9cf6u82571r8el7c0cth6wrlbg2 2433663 2433662 2026-07-07T12:29:50Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1036 - blake@cumin1003" 2433663 wikitext text/x-wiki bjpubrxm2qyyjxk0ypbi1oq3r9v8brh 2433664 2433663 2026-07-07T12:29:55Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1036 - blake@cumin1003" 2433664 wikitext text/x-wiki n7enha7y4693ernou43h5fu71jwj1zr 2433665 2433664 2026-07-07T12:29:56Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2433665 wikitext text/x-wiki rtcgy1cqs1opk24kz8u87xyygskurq4 2433666 2433665 2026-07-07T12:29:57Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1036.eqiad.wmnet 21.32.64.10.in-addr.arpa 1.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433666 wikitext text/x-wiki 4fnmzdz95aoumjnr9wzhp6c1wl7mr7o 2433667 2433666 2026-07-07T12:29:59Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1036.eqiad.wmnet 21.32.64.10.in-addr.arpa 1.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433667 wikitext text/x-wiki h73hef3wa4xsj2yvph4htvk90cwocfd 2433668 2433667 2026-07-07T12:29:59Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1036 2433668 wikitext text/x-wiki el4e04slhqogcrxecnrygu91yszhrtn 2433669 2433668 2026-07-07T12:30:05Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2433669 wikitext text/x-wiki gy9motbsht4gi1wih0mdki47i184nn4 2433670 2433669 2026-07-07T12:32:12Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1087.eqiad.wmnet with OS trixie 2433670 wikitext text/x-wiki lz9d6n7b3ggktplh15s9bzohm0wffmg 2433671 2433670 2026-07-07T12:32:28Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2005.codfw.wmnet with OS trixie 2433671 wikitext text/x-wiki r6gvkgfavolm7cxt4216fkcz31d1ifz 2433672 2433671 2026-07-07T12:32:33Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1036 2433672 wikitext text/x-wiki 80xr5a9j6ln68iwrdnfri4mh8bzvo4k 2433673 2433672 2026-07-07T12:32:33Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1036 2433673 wikitext text/x-wiki rr96xfpul8s0d5y3k29ch7ssfwpe7u0 2433675 2433673 2026-07-07T12:33:11Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.dns.netbox 2433675 wikitext text/x-wiki gp8m758dwyys02x103v97gacozaid0r 2433676 2433675 2026-07-07T12:36:00Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2433676 wikitext text/x-wiki pd74t2wqjcxsc5hcpgkcw97seov8vpu 2433677 2433676 2026-07-07T12:36:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2433677 wikitext text/x-wiki 7rhs4xqkrh6d1ild3occ4z71p9jvdw0 2433678 2433677 2026-07-07T12:38:08Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1082.eqiad.wmnet with reason: host reimage 2433678 wikitext text/x-wiki 2oxawefb28u1ngzh73830tpveacpkfh 2433679 2433678 2026-07-07T12:39:37Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2006.codfw.wmnet with OS trixie 2433679 wikitext text/x-wiki jcqlxyckhuvr0oa7s79dmbkd1e8ogx0 2433680 2433679 2026-07-07T12:39:50Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: move dumps-nfs IP to the shared one - filippo@cumin1003" 2433680 wikitext text/x-wiki 08insuxwz93d986j07s2aaqhpj4s3na 2433681 2433680 2026-07-07T12:39:55Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: move dumps-nfs IP to the shared one - filippo@cumin1003" 2433681 wikitext text/x-wiki 2r6uof5rvgwkr2glnhuzbhlvnj2hf7s 2433682 2433681 2026-07-07T12:39:56Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2433682 wikitext text/x-wiki ep4cntf8akx1yh6qqt5r6wlkkdgu9vh 2433683 2433682 2026-07-07T12:41:57Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply 2433683 wikitext text/x-wiki 9cytlwfd3f1d3wwwnb96jput97i6zff 2433684 2433683 2026-07-07T12:41:59Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply} 2433684 wikitext text/x-wiki hipfvabqz6gz674nksr4i7rrn6i87c5 2433685 2433684 2026-07-07T12:42:56Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges 2433685 wikitext text/x-wiki 4189wsj68m84c466acq1fp542f5cc4d 2433686 2433685 2026-07-07T12:42:58Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) 2433686 wikitext text/x-wiki s9s0g4fm1mgyc170hbl5pt0nss53sfm 2433687 2433686 2026-07-07T12:43:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver1002.eqiad.wmnet 2433687 wikitext text/x-wiki cdzkse9i6qpc8k1rj825rccz8srpprx 2433688 2433687 2026-07-07T12:44:35Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1082.eqiad.wmnet with reason: host reimage 2433688 wikitext text/x-wiki s3530fvnhh0cdxq0s9j3apcb318dqwi 2433689 2433688 2026-07-07T12:48:25Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1087.eqiad.wmnet with reason: host reimage 2433689 wikitext text/x-wiki l0vs4he68hnowc53fmhuqo4leeesfix 2433690 2433689 2026-07-07T12:48:44Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply 2433690 wikitext text/x-wiki 8q00tkittjwzl4oysxhcouvqwngzz9z 2433691 2433690 2026-07-07T12:49:21Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1036.eqiad.wmnet with reason: host reimage 2433691 wikitext text/x-wiki n45pyed5ipb2hhv5c0dqm0scp0ss9fh 2433692 2433691 2026-07-07T12:51:32Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet 2433692 wikitext text/x-wiki n46031xueb5bgmbjbmy63a3cal3b1nb 2433693 2433692 2026-07-07T12:52:11Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) 2433693 wikitext text/x-wiki 4w0k5vvmx89gj4qqgxfm2ablnpuvmdl 2433694 2433693 2026-07-07T12:52:14Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1002.eqiad.wmnet 2433694 wikitext text/x-wiki 2j4pblvwxpytkdfxl8s8nx4ph97f9in 2433695 2433694 2026-07-07T12:52:25Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1088.eqiad.wmnet with OS trixie 2433695 wikitext text/x-wiki 17w61llz89llcw4jl3b9st6f0r383sb 2433696 2433695 2026-07-07T12:53:25Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply} 2433696 wikitext text/x-wiki j7txto83t8nllvofajgd45317cflz8h 2433697 2433696 2026-07-07T12:53:36Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1087.eqiad.wmnet with reason: host reimage 2433697 wikitext text/x-wiki 22qeed9bwlzveda0u8fmfl868t6b6ro 2433698 2433697 2026-07-07T12:55:36Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges 2433698 wikitext text/x-wiki k9oo37024tk9tskohmnor99zzazqx87 2433699 2433698 2026-07-07T12:55:38Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) 2433699 wikitext text/x-wiki hny2031b7oeml8f5wh59a5b06gezfr3 2433700 2433699 2026-07-07T12:56:24Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs[5004-5006].eqsin.wmnet with reason: upgrade JunOS cr3-eqsin 2433700 wikitext text/x-wiki olao5thb0enfpwnf8vr1ertbckqd1bw 2433701 2433700 2026-07-07T12:57:05Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr1-codfw,cr[2-3]-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: upgrade JunOS cr3-eqsin 2433701 wikitext text/x-wiki 6crfocq77qypjzcfypmcpg5rbrlasxx 2433702 2433701 2026-07-07T12:57:34Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2006.codfw.wmnet with reason: host reimage 2433702 wikitext text/x-wiki bnu3t7wm57n6oauw0izi328n90g5pzz 2433703 2433702 2026-07-07T12:57:56Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet 2433703 wikitext text/x-wiki 6ko2c1rtnz4tqnwxqivnxu2m4nfpcfe 2433704 2433703 2026-07-07T12:58:06Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1036.eqiad.wmnet with reason: host reimage 2433704 wikitext text/x-wiki lt66kw96lyl0taccgplpunckbrbl536 2433705 2433704 2026-07-07T12:58:34Z Stashbot 7414 topranks: load updated JunOS on cr3-eqsin T429386 2433705 wikitext text/x-wiki 4r7mw3fvobtq81fnhpl0gm5s929tfib 2433706 2433705 2026-07-07T13:02:18Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2006.codfw.wmnet with reason: host reimage 2433706 wikitext text/x-wiki 0rpimp809dwa92jeqguwrtep702hjrp 2433707 2433706 2026-07-07T13:05:40Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2433707 wikitext text/x-wiki pz5io2owpxzocv4f4n0c01c43o6vuso 2433708 2433707 2026-07-07T13:06:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1004.eqiad.wmnet with OS bookworm 2433708 wikitext text/x-wiki 8mn1k8u4r9nh9gnsq16che3cv0kesxb 2433709 2433708 2026-07-07T13:07:21Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2433709 wikitext text/x-wiki lakyutyam8ecr8tprqhkxl6uw4vw84a 2433710 2433709 2026-07-07T13:07:40Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1082.eqiad.wmnet with OS trixie 2433710 wikitext text/x-wiki epxkcjglxj3864mxalfkctnbmg07s8v 2433711 2433710 2026-07-07T13:09:10Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1088.eqiad.wmnet with reason: host reimage 2433711 wikitext text/x-wiki 6c1s7scxf3pe80f9fpcq0hbyt946i9e 2433713 2433711 2026-07-07T13:12:04Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2433713 wikitext text/x-wiki rekqz9h0rblos6u7wx1tc9imnya21zh 2433714 2433713 2026-07-07T13:13:41Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2433714 wikitext text/x-wiki a0d8pwrwwwvn4a2jf94w571u6a44lb1 2433715 2433714 2026-07-07T13:14:14Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1088.eqiad.wmnet with reason: host reimage 2433715 wikitext text/x-wiki p1l56ybs8q2zxaps6q8wlz7p2i5gjq7 2433716 2433715 2026-07-07T13:14:45Z Stashbot 7414 topranks: reboot cr3-eqsin to install new JunOS and set PIC 0/0/0 to 100G 2433716 wikitext text/x-wiki ng2mkp82ea7gunoortip554o4v7a7uz 2433717 2433716 2026-07-07T13:14:55Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1087.eqiad.wmnet with OS trixie 2433717 wikitext text/x-wiki citq5vaebca61yk726m23hw4rqp422q 2433718 2433717 2026-07-07T13:15:38Z Stashbot 7414 jayme: Istio is being upgraded from 1.24.2 to 1.29.4 on wikikube staging eqiad and codfw - T427401 2433718 wikitext text/x-wiki n344c8hha951hpgh065zq86tu1rfcg2 2433719 2433718 2026-07-07T13:16:48Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply 2433719 wikitext text/x-wiki 2sqv2j9lveuunqbzr7nhbxhmceucwi6 2433720 2433719 2026-07-07T13:17:14Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply} 2433720 wikitext text/x-wiki 2se2mdm49jpru3jn85tuc92xw4izh5x 2433721 2433720 2026-07-07T13:18:00Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-etcd1004.eqiad.wmnet with reason: host reimage 2433721 wikitext text/x-wiki 40zdlw2sfc3ly6pivkqr37rr4ugqfpa 2433722 2433721 2026-07-07T13:18:40Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1036.eqiad.wmnet with OS trixie 2433722 wikitext text/x-wiki s1jmq075oesxd4ed4akeskg5pjse0wr 2433723 2433722 2026-07-07T13:20:35Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2006.codfw.wmnet with OS trixie 2433723 wikitext text/x-wiki gldy524ahqyibu3a8nlvlxs11qi5pa8 2433724 2433723 2026-07-07T13:23:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1004.eqiad.wmnet with reason: host reimage 2433724 wikitext text/x-wiki rn5czxsumxbime9e7p4j1isfzjv93nm 2433725 2433724 2026-07-07T13:29:49Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe2007.codfw.wmnet with OS trixie 2433725 wikitext text/x-wiki ods5i53k5y2qj2w3bjvutci9mghp9tj 2433726 2433725 2026-07-07T13:32:09Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=no; selector: name=dns7002.* 2433726 wikitext text/x-wiki p70jul7ygm08kf04u0vqwsltvx1pgbx 2433727 2433726 2026-07-07T13:32:48Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS trixie 2433727 wikitext text/x-wiki g4ua1xty9ts27xf2h5w21iex38bxucu 2433728 2433727 2026-07-07T13:32:49Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1088.eqiad.wmnet with OS trixie 2433728 wikitext text/x-wiki cwdu91ov7rtwmhtbuzmjngm2x41xyak 2433729 2433728 2026-07-07T13:33:23Z Stashbot 7414 topranks: reset cr3-eqsin configuration so traffic uses it again after upgrade 2433729 wikitext text/x-wiki bictgw1cn31aattrnoduymjnnezun7p 2433730 2433729 2026-07-07T13:33:45Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1120.eqiad.wmnet with OS trixie 2433730 wikitext text/x-wiki dd7x3elu383mi1vb7o4qjpabftwp1qo 2433731 2433730 2026-07-07T13:35:35Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1308106|fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105|fix(MentorListCleaner): Do not access property before inicialization (T430689)]] 2433731 wikitext text/x-wiki 7jpq7g82yf2mai0dnll583r4ss46mfy 2433732 2433731 2026-07-07T13:38:09Z Stashbot 7414 moritzm: installing e2fsprogs updates from Trixie point release 2433732 wikitext text/x-wiki 4cqmcxexa715v4tv007ks67f41kyi6d 2433733 2433732 2026-07-07T13:39:26Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2433733 wikitext text/x-wiki cpazhnw16ddz6bam3yufnruvoj702sq 2433734 2433733 2026-07-07T13:40:16Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2433734 wikitext text/x-wiki njcnzecgfkxjw6bd3jjgf11ddk2aow2 2433735 2433734 2026-07-07T13:44:30Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply 2433735 wikitext text/x-wiki sb19qn3trwx5mkf4xj7qy3t9rqllpn8 2433736 2433735 2026-07-07T13:44:32Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2433736 wikitext text/x-wiki 6hlgms2b35gc3g2vpoplv20b9lkvla9 2433737 2433736 2026-07-07T13:45:17Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2433737 wikitext text/x-wiki qwno8zhhqtzzp9wmydo3j332dkxgilx 2433738 2433737 2026-07-07T13:46:24Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2433738 wikitext text/x-wiki 26msjlxi0ghq4fwb6jfglj0x0hlvuuj 2433739 2433738 2026-07-07T13:46:28Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2433739 wikitext text/x-wiki gn72su2n9t137mfh471bsiowmyibshb 2433740 2433739 2026-07-07T13:46:38Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1120.eqiad.wmnet with reason: host reimage 2433740 wikitext text/x-wiki nftxehsswkyspw4d40u55vznquv8qzp 2433741 2433740 2026-07-07T13:46:51Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply 2433741 wikitext text/x-wiki f48y6hx2mw9bk1mf21sg7y6m43vp4eh 2433742 2433741 2026-07-07T13:46:53Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2007.codfw.wmnet with reason: host reimage 2433742 wikitext text/x-wiki fk3z546f74pi5nudlko9vbn2fkyzv1n 2433743 2433742 2026-07-07T13:47:08Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply 2433743 wikitext text/x-wiki fqsltn3ll7ehxqp29g3dgk8qh2mwhyw 2433744 2433743 2026-07-07T13:47:18Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply 2433744 wikitext text/x-wiki 7rv33o5ubvwuj6tznbu0xjj1zjnv2ln 2433745 2433744 2026-07-07T13:50:37Z Stashbot 7414 moritzm: installing Linux 5.10.259 on Bullseye hosts 2433745 wikitext text/x-wiki fbdxn7jhalwljb1zrfu3xzk02pfmwmf 2433746 2433745 2026-07-07T13:53:45Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1120.eqiad.wmnet with reason: host reimage 2433746 wikitext text/x-wiki cjp0a3su3pfk5uj0gpo7vd67n1pdc1w 2433747 2433746 2026-07-07T13:57:17Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage 2433747 wikitext text/x-wiki czpsgyogrcbv6fwt9ylhm7f5vjjdckd 2433748 2433747 2026-07-07T13:57:58Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2007.codfw.wmnet with reason: host reimage 2433748 wikitext text/x-wiki o8dzq69zxl8g8yblzy4o3yv3xg8i5jj 2433749 2433748 2026-07-07T13:58:07Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet with OS bookworm 2433749 wikitext text/x-wiki dj49779k6jk2b49bs1upkvxhpzq7wnd 2433750 2433749 2026-07-07T13:58:29Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1308106|fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105|fix(MentorListCleaner): Do not access property before inicialization (T430689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2433750 wikitext text/x-wiki f5bl2mn5b2r4hat07dqlfh964ml9ygt 2433751 2433750 2026-07-07T13:58:55Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Continuing with deployment 2433751 wikitext text/x-wiki 5ntnh63x1xm7fuxvwu68v65birk1qy3 2433752 2433751 2026-07-07T14:00:42Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1083.eqiad.wmnet with OS trixie 2433752 wikitext text/x-wiki ol6svc1s296u1bqc6v9s17towqihwsv 2433753 2433752 2026-07-07T14:00:57Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage 2433753 wikitext text/x-wiki 6pzc2xehq02kv141ucjtuf9mrjs83g5 2433754 2433753 2026-07-07T14:03:24Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308106|fix(MentorChangeLogFormatter): Remove unused XSS suppression (T430693)]], [[gerrit:1308105|fix(MentorListCleaner): Do not access property before inicialization (T430689)]] (duration: 27m 48s) 2433754 wikitext text/x-wiki 5tt33fjo5u82fpavm3zao7fwcpguwzz 2433755 2433754 2026-07-07T14:04:45Z Stashbot 7414 fabfur: disable puppet on A:cp-text to selectively apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308040 2433755 wikitext text/x-wiki d1ns6tnao08bqwxvtcb9qzvi2cndpk0 2433756 2433755 2026-07-07T14:05:13Z Stashbot 7414 moritzm: installing distro-info-data updates from trixie/bookworm point releases 2433756 wikitext text/x-wiki qcokcayqfe3j95cgfm2144r821wjc49 2433757 2433756 2026-07-07T14:11:14Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1120.eqiad.wmnet with OS trixie 2433757 wikitext text/x-wiki 6ux70d24iyxa3ngyyqin5c5s23iiv9u 2433758 2433757 2026-07-07T14:12:47Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-master@codfw 2433758 wikitext text/x-wiki odivq1dzhiwcth0chl6rfbihmyje6rl 2433759 2433758 2026-07-07T14:14:59Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet 2433759 wikitext text/x-wiki jj108v9xu3ur6e9a4n7lm8med1yb6i7 2433760 2433759 2026-07-07T14:15:01Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet 2433760 wikitext text/x-wiki ab1elqbquymqc7orw94ot3fb66vzti8 2433761 2433760 2026-07-07T14:15:03Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1036.eqiad.wmnet 2433761 wikitext text/x-wiki in5docuddm1wrvt5jhhmsr53fxphob8 2433762 2433761 2026-07-07T14:16:18Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1083.eqiad.wmnet with reason: host reimage 2433762 wikitext text/x-wiki r1xm0n8rgm56iln6blcp7p7sf2dzane 2433763 2433762 2026-07-07T14:16:56Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2007.codfw.wmnet with OS trixie 2433763 wikitext text/x-wiki j6wq94tyskfevz9bva89q33rjh4irhi 2433764 2433763 2026-07-07T14:18:08Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1037.eqiad.wmnet 2433764 wikitext text/x-wiki tbg0ouksfl6cmjs3wkgfikwu0oeb0d3 2433765 2433764 2026-07-07T14:18:12Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet 2433765 wikitext text/x-wiki dhc4semnc6n4ezqivjinzjykvsj418l 2433766 2433765 2026-07-07T14:18:37Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2433766 wikitext text/x-wiki nstr2c11dhaietajkgiehxshwfxu2yf 2433767 2433766 2026-07-07T14:18:47Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet 2433767 wikitext text/x-wiki 20imxq97lihoulivbqxj3m5qdf4dwga 2433768 2433767 2026-07-07T14:18:59Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1037.eqiad.wmnet with OS trixie 2433768 wikitext text/x-wiki mrb3pvv519ywncdct42h2w7him6du3y 2433769 2433768 2026-07-07T14:19:36Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1037 2433769 wikitext text/x-wiki liwcte02jz5qr1xqhefh9xva1owsgjf 2433770 2433769 2026-07-07T14:19:38Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2433770 wikitext text/x-wiki 7ungn305nij9e1gzv6qiraw8cg35pp5 2433771 2433770 2026-07-07T14:19:38Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-master@codfw 2433771 wikitext text/x-wiki 4s30q91c9tlphen6pcgcsh2cigdfryq 2433772 2433771 2026-07-07T14:19:52Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.netbox 2433772 wikitext text/x-wiki augogrleyvklz50a1bh6seexiwdkcsu 2433773 2433772 2026-07-07T14:20:30Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1083.eqiad.wmnet with reason: host reimage 2433773 wikitext text/x-wiki ftocbsd4c2l8rzzotecl50u860l1bq2 2433774 2433773 2026-07-07T14:24:04Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1037 - blake@cumin1003" 2433774 wikitext text/x-wiki ifrmlzz55sptqikzoewt3oq7q38n193 2433775 2433774 2026-07-07T14:24:09Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1037 - blake@cumin1003" 2433775 wikitext text/x-wiki qe3pwyu4m3e04mtmtopz7oyrfgdzfli 2433776 2433775 2026-07-07T14:24:09Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2433776 wikitext text/x-wiki riv9hudcl9em1w7vfegvyfyecvqor69 2433777 2433776 2026-07-07T14:24:10Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433777 wikitext text/x-wiki 38a4u6c9tiq5tb3x6drssqc8rlgwy6c 2433778 2433777 2026-07-07T14:24:13Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433778 wikitext text/x-wiki ad0pfkkp7s65z8uxlfoik13209ecugp 2433779 2433778 2026-07-07T14:24:27Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433779 wikitext text/x-wiki 0ol4qfil3vg5a9mywk6v6z46kg6h8ac 2433780 2433779 2026-07-07T14:24:31Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433780 wikitext text/x-wiki nl98ulm7wbmmjlhkxqxph8htplo7ixe 2433781 2433780 2026-07-07T14:25:04Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:cleanMentorList.php --wiki=frwiki # T427386 2433781 wikitext text/x-wiki n877yn4ebs5ucr3hitk98iny09c9gjv 2433782 2433781 2026-07-07T14:25:11Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433782 wikitext text/x-wiki njvzp9ufw5wybrxawngzrqq9t0i27pz 2433783 2433782 2026-07-07T14:25:14Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433783 wikitext text/x-wiki tj2w2c9sa26uhy90mlww2aiunzz3auw 2433784 2433783 2026-07-07T14:26:07Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2433784 wikitext text/x-wiki 75txao46q452sj7j3xs5vnvm9ol68x4 2433785 2433784 2026-07-07T14:26:30Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2433785 wikitext text/x-wiki m3lmaw51aikmyximkj9fqbp78p6kah6 2433786 2433785 2026-07-07T14:26:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2433786 wikitext text/x-wiki dl4va5mgs1wk4chzwpkx3lr9yfct5ae 2433787 2433786 2026-07-07T14:26:41Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1007.eqiad.wmnet with OS trixie 2433787 wikitext text/x-wiki tcrw1pjjnb6a2twyky1jdgcgllgev3o 2433788 2433787 2026-07-07T14:26:52Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2433788 wikitext text/x-wiki 3t72n877pkinigiixb2tk253e5h1zw8 2433789 2433788 2026-07-07T14:27:42Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1308112|Revert^2 "[Growth] frwiki: Deploy automated mentor list cleaner" (T427386)]] 2433789 wikitext text/x-wiki ayut7cd7pyxdceyktarjom47jikt5sr 2433790 2433789 2026-07-07T14:27:51Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2433790 wikitext text/x-wiki sx9yz6g7zlja9g4y854y67j3gkbd11x 2433791 2433790 2026-07-07T14:28:08Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2433791 wikitext text/x-wiki b1pwthm0wf0p3tuqnze0bueqpgv9rn7 2433792 2433791 2026-07-07T14:28:25Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2433792 wikitext text/x-wiki 0gvm4rf556v2soubsx5lxd7u7ujt40k 2433793 2433792 2026-07-07T14:28:43Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2433793 wikitext text/x-wiki bkuz2eo3y14kmngixq50p8ywfljffcm 2433794 2433793 2026-07-07T14:29:10Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433794 wikitext text/x-wiki p6uuqwqocoelbfvbrro8fe6xy2qx4ah 2433795 2433794 2026-07-07T14:29:14Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) wikikube-worker1037.eqiad.wmnet 143.48.64.10.in-addr.arpa 3.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433795 wikitext text/x-wiki 1zk0pxttf6zrctfn60rxrbm3gfg8gqj 2433797 2433795 2026-07-07T14:29:49Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2433797 wikitext text/x-wiki 7g5k46sjdsbe18y8vw1aphgq48vxo8u 2433798 2433797 2026-07-07T14:29:49Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-master@eqiad 2433798 wikitext text/x-wiki pfiiom4oq7lhk7cm0n4d7xglyspnpvr 2433799 2433798 2026-07-07T14:29:59Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2433799 wikitext text/x-wiki eto6mroz8w319adrjnu5mvurwm3aetz 2433800 2433799 2026-07-07T14:31:44Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2433800 wikitext text/x-wiki ke331y088h3vnquf8l21icj9fk4eqee 2433801 2433800 2026-07-07T14:31:58Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2433801 wikitext text/x-wiki 3najb49xgey6vlhp94129zzz2znv644 2433802 2433801 2026-07-07T14:32:51Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1037 2433802 wikitext text/x-wiki gdpmixz5yjo7bqrl7p0wdnxmii0u6tl 2433803 2433802 2026-07-07T14:33:21Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2433803 wikitext text/x-wiki 0k0orb0m3c3bnitk0r97lpwmyqpymma 2433804 2433803 2026-07-07T14:33:41Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2433804 wikitext text/x-wiki 54liuvva8p2wegdtw66wz8s3zocw99w 2433805 2433804 2026-07-07T14:34:04Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2433805 wikitext text/x-wiki mp7dksemxygaisgdi2b8dv4t5lkjolw 2433806 2433805 2026-07-07T14:34:38Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308112|Revert^2 "[Growth] frwiki: Deploy automated mentor list cleaner" (T427386)]] (duration: 06m 47s) 2433806 wikitext text/x-wiki kxnja1wdvkkqgcao7np0rcm7lslai0o 2433807 2433806 2026-07-07T14:34:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply 2433807 wikitext text/x-wiki rd1ra89jr1s7zrr0mw7668bgfmymh33 2433808 2433807 2026-07-07T14:34:42Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:cleanMentorList.php --wiki=frwiki # T427386 2433808 wikitext text/x-wiki 2158ane0oba5ljfc50dmbinsn3x3utj 2433809 2433808 2026-07-07T14:34:43Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1037 2433809 wikitext text/x-wiki j68oomzci3j5cfg9kpzgrwztyf7c93s 2433810 2433809 2026-07-07T14:34:44Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1037 2433810 wikitext text/x-wiki easeqgk86lnptfs2s5hafsrpjusz5gk 2433811 2433810 2026-07-07T14:35:05Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply} 2433811 wikitext text/x-wiki mbvpydmcln80gezu3vrvonllttuis04 2433812 2433811 2026-07-07T14:35:20Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2433812 wikitext text/x-wiki nbndeb0ljavpujpdjamwklszu2p55mq 2433813 2433812 2026-07-07T14:35:20Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-master@eqiad 2433813 wikitext text/x-wiki 5fjty1wapi7u4s0n62cxfxusn6975eb 2433814 2433813 2026-07-07T14:37:00Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2433814 wikitext text/x-wiki psshaiqjxxw576qp4cgwyialoe4fa6p 2433815 2433814 2026-07-07T14:37:11Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2433815 wikitext text/x-wiki 4kailt0kll9lyk0jkw4aq9u1ksjvsz0 2433816 2433815 2026-07-07T14:40:36Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1083.eqiad.wmnet with OS trixie 2433816 wikitext text/x-wiki rq9lhnznwm6yweit2howclmbdsdfndo 2433817 2433816 2026-07-07T14:42:54Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1007.eqiad.wmnet with reason: host reimage 2433817 wikitext text/x-wiki nxi5ojktvd9rg8azfh6y1fevq3uf6xz 2433818 2433817 2026-07-07T14:49:20Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1007.eqiad.wmnet with reason: host reimage 2433818 wikitext text/x-wiki f8zpkornd9icdqc0kilsbxvv8zf0sn8 2433819 2433818 2026-07-07T14:50:52Z Stashbot 7414 JavierMonton: Deploying Refinery at 7d8dc71f for change 1308087 / T431318 - update filerevision table sqoop and table 2433819 wikitext text/x-wiki jh8tt28l1srt5ej35hd8ytx0jl0kdr6 2433820 2433819 2026-07-07T14:51:26Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1037.eqiad.wmnet with reason: host reimage 2433820 wikitext text/x-wiki t7sawmle1vwbzumd2el9hrmj031vkpi 2433821 2433820 2026-07-07T14:51:31Z Stashbot 7414 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab2003.codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: maintenance 2433821 wikitext text/x-wiki gkvutwq1an77wcvy9ge8gehbpue4mvj 2433822 2433821 2026-07-07T14:51:47Z Stashbot 7414 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7d8dc71f] 2433822 wikitext text/x-wiki nu17cgg3cf0jihvverc65vrc0ow9rzx 2433823 2433822 2026-07-07T14:53:47Z Stashbot 7414 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7d8dc71f] (duration: 02m 00s) 2433823 wikitext text/x-wiki 030lhtzempivc14qqb1nbxd4j36g1o1 2433824 2433823 2026-07-07T14:53:57Z Stashbot 7414 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71]: Regular analytics weekly train [analytics/refinery@7d8dc71f] 2433824 wikitext text/x-wiki 3l3oba4lnvkonsavppgg6o8f2l1jzju 2433825 2433824 2026-07-07T14:55:00Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1037.eqiad.wmnet with reason: host reimage 2433825 wikitext text/x-wiki gji6uvkhbazzk6w1z2j07v5kyoe4sae 2433826 2433825 2026-07-07T14:58:12Z Stashbot 7414 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71]: Regular analytics weekly train [analytics/refinery@7d8dc71f] (duration: 04m 14s) 2433826 wikitext text/x-wiki 9dk9oo3taa0512sf0ncmcg8veggzxii 2433827 2433826 2026-07-07T14:58:30Z Stashbot 7414 javiermonton@deploy1003: Started deploy [analytics/refinery@7d8dc71] (thin): Regular analytics weekly train THIN [analytics/refinery@7d8dc71f] 2433827 wikitext text/x-wiki 9vk0k9siojhlv2ta96uoimvlhi7xdgs 2433828 2433827 2026-07-07T15:00:40Z Stashbot 7414 javiermonton@deploy1003: Finished deploy [analytics/refinery@7d8dc71] (thin): Regular analytics weekly train THIN [analytics/refinery@7d8dc71f] (duration: 02m 10s) 2433828 wikitext text/x-wiki ol0eqkbadm65fghh0qy56ibl440hmo0 2433829 2433828 2026-07-07T15:03:07Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@7e02037]: deploy phab2003 for T431440 2433829 wikitext text/x-wiki a0v8g10jbfhghe6kh5ughkjjk0amhe2 2433830 2433829 2026-07-07T15:03:59Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@7e02037]: deploy phab2003 for T431440 (duration: 00m 51s) 2433830 wikitext text/x-wiki 5tp6l8x802iqnxbjrgjfzrjdri8q1gh 2433831 2433830 2026-07-07T15:04:13Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@7e02037]: deploy phab1004 for T431440 2433831 wikitext text/x-wiki hz1ap0s48roc8d9jn8q2m1urfud49g1 2433832 2433831 2026-07-07T15:05:03Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@7e02037]: deploy phab1004 for T431440 (duration: 00m 47s) 2433832 wikitext text/x-wiki sj345zisjh96neveqqszbvbsaio1z31 2433833 2433832 2026-07-07T15:08:15Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org 2433833 wikitext text/x-wiki mgjmm9d4mp140tag66s5jp2yy3c4t4d 2433834 2433833 2026-07-07T15:08:32Z Stashbot 7414 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host clouddumps1002.wikimedia.org 2433834 wikitext text/x-wiki 22x5uxiam975ttjd9fzp5tll1ahlj5x 2433835 2433834 2026-07-07T15:08:42Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1007.eqiad.wmnet with OS trixie 2433835 wikitext text/x-wiki j7yrg2k7bsqqudd76pqziqt3dl1fpzu 2433836 2433835 2026-07-07T15:09:25Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1121.eqiad.wmnet with OS trixie 2433836 wikitext text/x-wiki k9pedau1opc6do5ye1llf7mf486j6az 2433837 2433836 2026-07-07T15:09:39Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org 2433837 wikitext text/x-wiki cmwzi82iovzen4qep7x3vf8sfeva1go 2433838 2433837 2026-07-07T15:09:43Z Stashbot 7414 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host clouddumps1002.wikimedia.org 2433838 wikitext text/x-wiki iks6bfgj1cyzo5jk1hqz5xagskodajb 2433839 2433838 2026-07-07T15:09:52Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1121 2433839 wikitext text/x-wiki jwoyj5recxnlkx7mf72shqg71haszpi 2433840 2433839 2026-07-07T15:09:58Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org 2433840 wikitext text/x-wiki c5vdth6ei8h4ae1c2qgyajgb5gvqlrw 2433841 2433840 2026-07-07T15:11:15Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.dns.netbox 2433841 wikitext text/x-wiki 1oqs1s3ka0gmyuwytti6jh465y72y00 2433843 2433841 2026-07-07T15:14:06Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1037.eqiad.wmnet with OS trixie 2433843 wikitext text/x-wiki 1cv535y96tly8srqhiv1k41e1f9bl9p 2433846 2433843 2026-07-07T15:16:21Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1121 - atsuko@cumin1003" 2433846 wikitext text/x-wiki hz0t470ka0i56238qocmv7duztcdt1x 2433847 2433846 2026-07-07T15:16:26Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1121 - atsuko@cumin1003" 2433847 wikitext text/x-wiki 0khfekpd6b6jzfk9f0trydzqsfc60uh 2433848 2433847 2026-07-07T15:16:27Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2433848 wikitext text/x-wiki 7go4frrur80uzebho0cen0auxxh1ycv 2433849 2433848 2026-07-07T15:16:28Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433849 wikitext text/x-wiki 29tsicos3g7fkh5zakgr8ummlahhcn8 2433850 2433849 2026-07-07T15:16:30Z Stashbot 7414 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433850 wikitext text/x-wiki 927b4zv7yomxhasjsydmuevn3dw1ibm 2433851 2433850 2026-07-07T15:16:48Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply 2433851 wikitext text/x-wiki npcrr3615fasuviccmfoubjrn0e8dbe 2433852 2433851 2026-07-07T15:16:49Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433852 wikitext text/x-wiki 6uc5j5a4y2dtevgql7bz6op3nba7fko 2433853 2433852 2026-07-07T15:16:52Z Stashbot 7414 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433853 wikitext text/x-wiki sw0mf3tuysvz8kjja3fhixhtej108y2 2433854 2433853 2026-07-07T15:16:57Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply 2433854 wikitext text/x-wiki nwn1hsy0gugx69iknoqeijnpqqr65w9 2433855 2433854 2026-07-07T15:17:41Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433855 wikitext text/x-wiki taq4otq710hh9drx6ayjw40v0qcvb42 2433856 2433855 2026-07-07T15:17:44Z Stashbot 7414 atsuko@cumin1003: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) cirrussearch1121.eqiad.wmnet 30.48.64.10.in-addr.arpa 0.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2433856 wikitext text/x-wiki brcdcqc0fh5ut1sjiq9e2zmgfibq9x3 2433857 2433856 2026-07-07T15:17:50Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1121 2433857 wikitext text/x-wiki 61p50qotw147pf03crmluuhlxy7us9p 2433858 2433857 2026-07-07T15:18:20Z Stashbot 7414 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1002.wikimedia.org 2433858 wikitext text/x-wiki 13l009wwh9y5sls2gvz146zqxh7pj54 2433859 2433858 2026-07-07T15:18:21Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1121 2433859 wikitext text/x-wiki q5jkc6f9j8tbmhej76bvsvtdtcygj6i 2433860 2433859 2026-07-07T15:18:22Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1121 2433860 wikitext text/x-wiki csdaarszrh2g8wx7wwij2okaviq3oe9 2433861 2433860 2026-07-07T15:20:48Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host clouddumps1001.wikimedia.org 2433861 wikitext text/x-wiki cc0bxhwhrukgkw1uvgs5so542jja8te 2433864 2433861 2026-07-07T15:29:21Z Stashbot 7414 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1001.wikimedia.org 2433864 wikitext text/x-wiki 5ehjqnkt9wopbqmipodlly12ng2u5kf 2433865 2433864 2026-07-07T15:30:42Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1121.eqiad.wmnet with reason: host reimage 2433865 wikitext text/x-wiki f4w9uy6962cq79j73ss4wnai8sprad0 2433866 2433865 2026-07-07T15:33:16Z Stashbot 7414 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7002.wikimedia.org with OS trixie 2433866 wikitext text/x-wiki odn1a7agmi3ykrukz845d3oye7vn1ay 2433867 2433866 2026-07-07T15:33:52Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm 2433867 wikitext text/x-wiki kh3f5no8eqw1wdia8618l5ehfedkk0j 2433868 2433867 2026-07-07T15:34:56Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1121.eqiad.wmnet with reason: host reimage 2433868 wikitext text/x-wiki gq7j5lqxbdytxjvzkslkwrewb0whre9 2433871 2433868 2026-07-07T15:41:09Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1006.eqiad.wmnet with OS trixie 2433871 wikitext text/x-wiki ko8fkac89h8adj69ytrt5w9bnv1l2bv 2433872 2433871 2026-07-07T15:42:51Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet 2433872 wikitext text/x-wiki smspq4cpu49oide3lbu5kchqqsiwiml 2433873 2433872 2026-07-07T15:42:52Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet 2433873 wikitext text/x-wiki jg4jshvut8liciswmzsgivr2z4go3cc 2433874 2433873 2026-07-07T15:42:54Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1037.eqiad.wmnet 2433874 wikitext text/x-wiki 42atzvo4oafq6hxn0d2hb0nv1lio612 2433876 2433874 2026-07-07T15:54:28Z Stashbot 7414 mutante: jenkins down in planned maintenance window 2433876 wikitext text/x-wiki ntus94gyinmmpjn79fwf9rcoxsjx05i 2433877 2433876 2026-07-07T15:56:29Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1006.eqiad.wmnet with reason: host reimage 2433877 wikitext text/x-wiki jrgpum303xb1s0o9yecrq726hn40nlk 2433878 2433877 2026-07-07T15:58:42Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage 2433878 wikitext text/x-wiki g0as610inyg5bu8zyr02p7c04h4d98h 2433879 2433878 2026-07-07T15:58:50Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1121.eqiad.wmnet with OS trixie 2433879 wikitext text/x-wiki s646lk24xr1rhrghnh4ytjlp6x298c2 2433880 2433879 2026-07-07T16:03:04Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1006.eqiad.wmnet with reason: host reimage 2433880 wikitext text/x-wiki 8fufryq7soolg6p4eeajcl7kuj62v0q 2433881 2433880 2026-07-07T16:06:25Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage 2433881 wikitext text/x-wiki dhsjfv01yh4ok35odyvvgma4tsc9e8s 2433883 2433881 2026-07-07T16:21:14Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1006.eqiad.wmnet with OS trixie 2433883 wikitext text/x-wiki tomr4u7hyg90qz8kaarlkdvpeppr2zm 2433885 2433883 2026-07-07T16:27:13Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1111.eqiad.wmnet with OS trixie 2433885 wikitext text/x-wiki 9ro8s4ubgnh8cdkm7ckwd7z48bhm4fg 2433887 2433885 2026-07-07T16:35:44Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 47794 2433887 wikitext text/x-wiki bsqwommgfp9g482lqug5ue15nln3udu 2433889 2433887 2026-07-07T16:38:35Z Stashbot 7414 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 47794 2433889 wikitext text/x-wiki 3wfdym02nf634s42ua1d16ejen26xxa 2433890 2433889 2026-07-07T16:40:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1111.eqiad.wmnet with reason: host reimage 2433890 wikitext text/x-wiki lp9p3nqvnycl0hnt74lrh3aav44cqo8 2433891 2433890 2026-07-07T16:43:33Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1111.eqiad.wmnet with reason: host reimage 2433891 wikitext text/x-wiki is07891rmu32hjbsb502nh4oo5s21qn 2433894 2433891 2026-07-07T16:53:59Z Stashbot 7414 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-secondary-eqiad' 'systemctl restart pybal.service' # lvs1020, T416623 2433894 wikitext text/x-wiki 8p2xdk5od6xrgwyrel8gf2is3eae8g0 2433895 2433894 2026-07-07T16:55:14Z Stashbot 7414 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' # lvs1019, T416623 2433895 wikitext text/x-wiki arc35vsp99grspl0a20gqorvnqf3s9h 2433903 2433895 2026-07-07T16:58:21Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7002.wikimedia.org with OS bookworm 2433903 wikitext text/x-wiki 8giy4v1obnufn4sglbgdj2ekgaw0rc4 2433904 2433903 2026-07-07T16:58:32Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1001.eqiad.wmnet 2433904 wikitext text/x-wiki 0rc4emv3chp5otqt1fuyteg74r6bg9w 2433905 2433904 2026-07-07T17:00:58Z Stashbot 7414 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' # lvs2014, T416623 2433905 wikitext text/x-wiki r90ijchb3tjhtd0uiqw4tk7i766lbgz 2433906 2433905 2026-07-07T17:02:08Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1111.eqiad.wmnet with OS trixie 2433906 wikitext text/x-wiki de3ld77q9r1hq7jjd2ar3rd1gfluko0 2433907 2433906 2026-07-07T17:04:35Z Stashbot 7414 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1001.eqiad.wmnet 2433907 wikitext text/x-wiki mr1mgesxuupu05xf3j9pive4e5eg1fz 2433909 2433907 2026-07-07T17:06:15Z Stashbot 7414 rzl: rzl@cumin2003:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' # lvs2013, T416623 2433909 wikitext text/x-wiki aqiueuznr8p60g6juyja3xxoekdpk3b 2433910 2433909 2026-07-07T17:06:43Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1002.eqiad.wmnet 2433910 wikitext text/x-wiki 4wksagwje7al0ogzavcb7eluydrlo6r 2433912 2433910 2026-07-07T17:14:48Z Stashbot 7414 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1002.eqiad.wmnet 2433912 wikitext text/x-wiki 1si1q7kzeu9fytvmwhx7hw1mx6t6og0 2433914 2433912 2026-07-07T17:15:25Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1003.eqiad.wmnet 2433914 wikitext text/x-wiki m7jv5cv7ov2mgk50axxkqd4mycxb20x 2433917 2433914 2026-07-07T17:21:51Z Stashbot 7414 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1003.eqiad.wmnet 2433917 wikitext text/x-wiki gra6fvjz8g8f2q3goduy65o9wfxzz0y 2433922 2433917 2026-07-07T17:40:50Z Stashbot 7414 sukhe: restart pybal on lvs2014 to switch from conf2004 to conf1008: T430909 2433922 wikitext text/x-wiki 51pfbdckw0suld3x1xgzxwep2s4uega 2433923 2433922 2026-07-07T17:40:51Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2433923 wikitext text/x-wiki fhtim5rmxf9rteuqg1z2hm96brbjh0h 2433924 2433923 2026-07-07T17:43:04Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2433924 wikitext text/x-wiki 4vao46hhvqth8wwxw64rs48uv9j1tc2 2433925 2433924 2026-07-07T17:44:07Z Stashbot 7414 swfrench-wmf: switched codfw, eqsin, ulsfo etcd client SRV records to eqiad - T430909 2433925 wikitext text/x-wiki 2jnucmo6c71ivp4ftb86yjrupfaoq1u 2433926 2433925 2026-07-07T17:46:42Z Stashbot 7414 sukhe: restart pybal on lvs2013 to switch from conf2004 to conf1008: T430909 2433926 wikitext text/x-wiki slncdnozu6yatsghwmq8c38r31ryr23 2433928 2433926 2026-07-07T17:51:02Z Stashbot 7414 sukhe: restart pybal on lvs2012 to switch from conf2004 to conf1008 [puppet re-enabled there]: T430909 2433928 wikitext text/x-wiki ksouscenfnukqyabznd67y2o4nu4wai 2433932 2433928 2026-07-07T17:52:42Z Stashbot 7414 sukhe: restart pybal on lvs2011 to switch from conf2004 to conf1008: T430909 2433932 wikitext text/x-wiki aez4elq8e73i70sgyuvg12wcaguuvjl 2433935 2433932 2026-07-07T17:59:16Z Stashbot 7414 swfrench-wmf: restarted ulsfo confds, confirmed now connected to eqiad backends - T430909 2433935 wikitext text/x-wiki g3iv0v5vgrp2hvo5slg7mu8g8r5sdxf 2433936 2433935 2026-07-07T18:01:15Z Stashbot 7414 swfrench-wmf: restarted navtiming on webperf2003 - T430909 2433936 wikitext text/x-wiki au9dcla700yd7m1m2uc4618kysluilz 2433937 2433936 2026-07-07T18:11:55Z Stashbot 7414 swfrench-wmf: restarted eqsin, codfw confds - T430909 2433937 wikitext text/x-wiki 5diiomfor03fqrf7qt924x1fp5r7jkr 2433941 2433937 2026-07-07T18:38:52Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T430909) 2433941 wikitext text/x-wiki qmtbzc84rfjmb49524gxtb04elq3q9s 2433942 2433941 2026-07-07T18:40:44Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T430909) 2433942 wikitext text/x-wiki 7pxl60bu8yns84p3n4xd2xec82wkt8u 2433943 2433942 2026-07-07T18:49:11Z Stashbot 7414 swfrench@cumin2002: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin (T430909) 2433943 wikitext text/x-wiki 1f8v06hbazp3r3vso7gyousjzretfg4 2433944 2433943 2026-07-07T18:52:10Z Stashbot 7414 swfrench@cumin2002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin (T430909) 2433944 wikitext text/x-wiki kn1i7kuqjk3bz3708i53fe57uaqibgj 2433945 2433944 2026-07-07T18:56:11Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1092.eqiad.wmnet with OS trixie 2433945 wikitext text/x-wiki tvibf4rbontg38xyzg2dltb1mkccolr 2433946 2433945 2026-07-07T18:58:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1109.eqiad.wmnet with OS trixie 2433946 wikitext text/x-wiki 0vpeidihq88oqusz72ku9eo9focy867 2433947 2433946 2026-07-07T18:58:45Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1123.eqiad.wmnet with OS trixie 2433947 wikitext text/x-wiki de4yobiimwi54d38xpa49whdndwqfdz 2433948 2433947 2026-07-07T19:10:58Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1123.eqiad.wmnet with reason: host reimage 2433948 wikitext text/x-wiki 0pu4r95dmbgn2xrie4vndaov44f9bgr 2433949 2433948 2026-07-07T19:11:20Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1092.eqiad.wmnet with reason: host reimage 2433949 wikitext text/x-wiki 780zcm95in24g4mbruv5cj7s86otoni 2433950 2433949 2026-07-07T19:12:29Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=dns7002.*,service=authdns-update 2433950 wikitext text/x-wiki i25z3o23j6d1ulvzfc0n5t5q40qt82h 2433951 2433950 2026-07-07T19:13:23Z Stashbot 7414 cdobbins@dns1004: START - running authdns-update 2433951 wikitext text/x-wiki g7eybeeq3r3bccldvkxeb4b2dhj49qe 2433952 2433951 2026-07-07T19:14:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1123.eqiad.wmnet with reason: host reimage 2433952 wikitext text/x-wiki bpxu07bdx6phqu4nktvg84nko7g89k9 2433953 2433952 2026-07-07T19:15:01Z Stashbot 7414 cdobbins@dns1004: END - running authdns-update 2433953 wikitext text/x-wiki tfkzjurkarwgyy96ns9oblt82mtfsdo 2433954 2433953 2026-07-07T19:15:26Z Stashbot 7414 jasmine@dns1004: START - running authdns-update 2433954 wikitext text/x-wiki 9fhzewr415kpc60lkk2q5kesi2p38ow 2433955 2433954 2026-07-07T19:15:42Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1109.eqiad.wmnet with reason: host reimage 2433955 wikitext text/x-wiki 0m6i1m161zv5lwpf1thifsk07kqelcj 2433956 2433955 2026-07-07T19:17:15Z Stashbot 7414 jasmine@dns1004: END - running authdns-update 2433956 wikitext text/x-wiki 2q3i0zulo6g11jvrbrlwzrxi8gkb2cr 2433957 2433956 2026-07-07T19:17:57Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1092.eqiad.wmnet with reason: host reimage 2433957 wikitext text/x-wiki 08dsa2csgthjarq1le9h49h3yx6lyvr 2433958 2433957 2026-07-07T19:19:17Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=dns7002.* 2433958 wikitext text/x-wiki ox7yiqmx9ygsthykq33845anbbdcbbw 2433959 2433958 2026-07-07T19:22:14Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1109.eqiad.wmnet with reason: host reimage 2433959 wikitext text/x-wiki sjhcwthgbp24yx6dfc7c3cj72azw2xu 2433960 2433959 2026-07-07T19:32:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1123.eqiad.wmnet with OS trixie 2433960 wikitext text/x-wiki 9g3xbxcofjop155mw3r6jcu1z90w4pt 2433961 2433960 2026-07-07T19:38:48Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1092.eqiad.wmnet with OS trixie 2433961 wikitext text/x-wiki q9gksd8424spsiwhzy7nrxon9fj00by 2433963 2433961 2026-07-07T19:44:33Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1109.eqiad.wmnet with OS trixie 2433963 wikitext text/x-wiki gd4icqpe9fii3pqnd1xnru1ikqkwlq1 2433964 2433963 2026-07-07T19:57:03Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1090.eqiad.wmnet with OS trixie 2433964 wikitext text/x-wiki jb28bg12mfzb70gx56ckmmielj6owwa 2433965 2433964 2026-07-07T19:58:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1108.eqiad.wmnet with OS trixie 2433965 wikitext text/x-wiki p7pe4xhipbj5ufe1wvz4i8d1tb972a0 2433966 2433965 2026-07-07T20:01:23Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cirrussearch1108.eqiad.wmnet with OS trixie 2433966 wikitext text/x-wiki q9j4bmzz9shaap3c5qz1tlda7elor91 2433967 2433966 2026-07-07T20:09:22Z Stashbot 7414 cwhite: remove 2026-04 swift log archives from centrallog2002 to free some space 2433967 wikitext text/x-wiki 2m6grj92zph9zqpmyuk3igewgskhb0a 2433969 2433967 2026-07-07T20:14:47Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1308196|Bump Parsoid image limit to 1250 (T430854)]] 2433969 wikitext text/x-wiki qail7a6i0u826uhq7uad36zgtr0rh5i 2433970 2433969 2026-07-07T20:15:10Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1108.eqiad.wmnet with OS trixie 2433970 wikitext text/x-wiki li1tnsumu33sdfp8f80pml3jwua30ok 2433971 2433970 2026-07-07T20:16:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1090.eqiad.wmnet with OS trixie 2433971 wikitext text/x-wiki acl1nn9dq2rstxeawk2sivwotaxkrc0 2433972 2433971 2026-07-07T20:16:51Z Stashbot 7414 arlolra@deploy1003: arlolra: Backport for [[gerrit:1308196|Bump Parsoid image limit to 1250 (T430854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2433972 wikitext text/x-wiki 79wugdsz3gi58mfmir39huzz03mr2gv 2433973 2433972 2026-07-07T20:17:54Z Stashbot 7414 arlolra@deploy1003: arlolra: Continuing with deployment 2433973 wikitext text/x-wiki 1u4cyh36tzsjxoz3dmudf5ld4k9w2nr 2433974 2433973 2026-07-07T20:18:03Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1091.eqiad.wmnet with OS trixie 2433974 wikitext text/x-wiki re1pvbnwmf5yhm1j0nzesz5di1b9srb 2433975 2433974 2026-07-07T20:20:42Z Stashbot 7414 jasmine_: "homer "cr*eqiad*" commit "Added new stacked control plane wikikube-ctrl1006"" 2433975 wikitext text/x-wiki rg1ybe9a8uwstfuegvjed2yo1vpg5bi 2433976 2433975 2026-07-07T20:22:16Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308196|Bump Parsoid image limit to 1250 (T430854)]] (duration: 07m 29s) 2433976 wikitext text/x-wiki r61gzsy3omxnrb7uqfxii37mnlz7vsp 2433977 2433976 2026-07-07T20:27:02Z Stashbot 7414 jasmine_: "homer lsw1-c2-eqiad* commit "Added new stacked control plane wikikube-ctrl1006"" 2433977 wikitext text/x-wiki oawbu78c458ngmlz6r08fxvfwv22x91 2433978 2433977 2026-07-07T20:30:19Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet 2433978 wikitext text/x-wiki degb0dadyf0q4mavpntr7c3rfjvjp3l 2433979 2433978 2026-07-07T20:30:20Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1090.eqiad.wmnet with reason: host reimage 2433979 wikitext text/x-wiki l2y2ldrc85yi1vidl1dofba7kpucd50 2433980 2433979 2026-07-07T20:30:22Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet 2433980 wikitext text/x-wiki tvm5lf62iomkpfh7ibfigwo9u2xm3dp 2433981 2433980 2026-07-07T20:30:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1108.eqiad.wmnet with reason: host reimage 2433981 wikitext text/x-wiki jyd7qye45a9o98tfby2zb70k2vxta0y 2433982 2433981 2026-07-07T20:33:17Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1091.eqiad.wmnet with reason: host reimage 2433982 wikitext text/x-wiki 11ao24n9y6v5j3nk5wazwwkqmyl9dyl 2433983 2433982 2026-07-07T20:33:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1090.eqiad.wmnet with reason: host reimage 2433983 wikitext text/x-wiki o1kwxvmmncqwyrlvxj493wq1lxh0hwt 2433984 2433983 2026-07-07T20:36:53Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1091.eqiad.wmnet with reason: host reimage 2433984 wikitext text/x-wiki rnzcsezgz516wfvnache4gm30oxvf3r 2433985 2433984 2026-07-07T20:43:57Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1108.eqiad.wmnet with reason: host reimage 2433985 wikitext text/x-wiki e6w6n4rgyo2mf6ozrtmx70pcjlxis7f 2433986 2433985 2026-07-07T20:54:48Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1090.eqiad.wmnet with OS trixie 2433986 wikitext text/x-wiki 676caulcqspiye6sb4dwhxppc78szp8 2433988 2433986 2026-07-07T20:58:13Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1091.eqiad.wmnet with OS trixie 2433988 wikitext text/x-wiki 1qwxl1v8ah94v1lji0tny5daf3gck0d 2433989 2433988 2026-07-07T21:05:14Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1108.eqiad.wmnet with OS trixie 2433989 wikitext text/x-wiki agk5j5x7n0laed1jjv6neev5fne1x0s 2433990 2433989 2026-07-07T21:11:33Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2433990 wikitext text/x-wiki kxtvnoeyp0t283bbg8xwmg7nv6hnhx8 2433991 2433990 2026-07-07T21:12:02Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2433991 wikitext text/x-wiki ote1evu8j9kwgce877d7wlb3nsm8k92 2433992 2433991 2026-07-07T21:12:46Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2433992 wikitext text/x-wiki g9s5uq7e3ndgubpt5jf2qyvvjswt25c 2433993 2433992 2026-07-07T21:13:58Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2433993 wikitext text/x-wiki k5d5ruiovs6tajorm7aissc0f08t7k5 2433996 2433993 2026-07-07T22:07:50Z Stashbot 7414 hashar: Restarting Gerrit on gerrit2003 2433996 wikitext text/x-wiki jtpxuir4ev480m3buow2fj891u0ue9n 2433997 2433996 2026-07-07T22:09:13Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1097.eqiad.wmnet with OS trixie 2433997 wikitext text/x-wiki tli86utgdsxl8ngpzwdvh6g2ux7jcds 2434000 2433997 2026-07-07T22:14:37Z Stashbot 7414 hashar: Restarting Gerrit on gerrit2002 and gerrit1003 (replicas) 2434000 wikitext text/x-wiki qzbmb0h2295okxfkll57zxrag5vuwpk 2434001 2434000 2026-07-07T22:24:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1097.eqiad.wmnet with reason: host reimage 2434001 wikitext text/x-wiki a0e7tjp0y208my8muxpr5kskqfow3kz 2434002 2434001 2026-07-07T22:29:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1097.eqiad.wmnet with reason: host reimage 2434002 wikitext text/x-wiki fazqadpjcryoefg3xvpzgp1zaja3jar 2434004 2434002 2026-07-07T22:49:47Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1097.eqiad.wmnet with OS trixie 2434004 wikitext text/x-wiki nlw9c2tdlmqdpjlsl8h0cnb4s1nqftt 2434008 2434004 2026-07-08T00:15:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1110.eqiad.wmnet with OS trixie 2434008 wikitext text/x-wiki pzwiyshsmudyfrd7rya7l3kdwc8z69d 2434009 2434008 2026-07-08T00:26:58Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1099.eqiad.wmnet with OS trixie 2434009 wikitext text/x-wiki 0yan1i9sibtcblg3n7diaiq31fjpn08 2434010 2434009 2026-07-08T00:32:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1110.eqiad.wmnet with reason: host reimage 2434010 wikitext text/x-wiki amx1oddl7pfbarpzircgjw9mh8w67zk 2434011 2434010 2026-07-08T00:38:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1110.eqiad.wmnet with reason: host reimage 2434011 wikitext text/x-wiki 0vh3ww939kgzf57057q121mupqetxum 2434012 2434011 2026-07-08T00:41:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1099.eqiad.wmnet with reason: host reimage 2434012 wikitext text/x-wiki rw5gteykdnz4l0suqb4xhha125ol8tb 2434013 2434012 2026-07-08T00:44:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1099.eqiad.wmnet with reason: host reimage 2434013 wikitext text/x-wiki fx5i8fhkuvpk9rns6jk56mgej30eblx 2434014 2434013 2026-07-08T00:57:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1110.eqiad.wmnet with OS trixie 2434014 wikitext text/x-wiki 8359p4ta1y6gjj2cscb7kjix6rh4gao 2434015 2434014 2026-07-08T01:03:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1099.eqiad.wmnet with OS trixie 2434015 wikitext text/x-wiki 5j1097gsydxa8g9ivx8pg8l05ot0f8t 2434017 2434015 2026-07-08T01:49:38Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1308250|Init minwikiquote (T429922)]] 2434017 wikitext text/x-wiki t8a04ald5iw7hth13rqfvfdsmxawycj 2434018 2434017 2026-07-08T01:51:40Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1308250|Init minwikiquote (T429922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2434018 wikitext text/x-wiki px3rwdrzqqcvag2e19uzsko3qhn8dqr 2434019 2434018 2026-07-08T01:55:02Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2434019 wikitext text/x-wiki 5emuacxxcdz3085rjp12xxs7vjyksdh 2434020 2434019 2026-07-08T01:59:25Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308250|Init minwikiquote (T429922)]] (duration: 09m 46s) 2434020 wikitext text/x-wiki 6fiqyuqivcpoa5mg0462449079qvobz 2434021 2434020 2026-07-08T02:19:03Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1308251|Activate minwikiquote (T429922)]] 2434021 wikitext text/x-wiki 5vs1oiyet0m2y976nfia6b3m9gviavi 2434022 2434021 2026-07-08T02:21:05Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1308251|Activate minwikiquote (T429922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2434022 wikitext text/x-wiki 843pgsv2495f2g5nicalu6o7xoymfn2 2434023 2434022 2026-07-08T02:22:56Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2434023 wikitext text/x-wiki na16trwz8h5od04mz13jgtw857btyg7 2434024 2434023 2026-07-08T02:27:17Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308251|Activate minwikiquote (T429922)]] (duration: 08m 14s) 2434024 wikitext text/x-wiki 1br2bcm31pdimr4yqh3d6562mj4krp9 2434026 2434024 2026-07-08T04:55:50Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1101.eqiad.wmnet with OS trixie 2434026 wikitext text/x-wiki 3z6phvfvrl4m0ty3kkrih5ykv6ui5ll 2434027 2434026 2026-07-08T04:56:28Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1107.eqiad.wmnet with OS trixie 2434027 wikitext text/x-wiki 1lxp5lqw8uonsbbi9vh11l5kbrfwb1t 2434028 2434027 2026-07-08T04:58:28Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1124.eqiad.wmnet with OS trixie 2434028 wikitext text/x-wiki 1jha6463a7wbs5vyayok9xty9r7xmgt 2434029 2434028 2026-07-08T05:10:55Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1101.eqiad.wmnet with reason: host reimage 2434029 wikitext text/x-wiki qw7d1rczf7b42mvtya0yk9nyn6qu0zv 2434030 2434029 2026-07-08T05:11:12Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1124.eqiad.wmnet with reason: host reimage 2434030 wikitext text/x-wiki npm5ympfqqt5i9m9nnop82hg0ev3k6i 2434031 2434030 2026-07-08T05:13:35Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1101.eqiad.wmnet with reason: host reimage 2434031 wikitext text/x-wiki f3f7gw661rfea0yqscz86byrofr61tj 2434032 2434031 2026-07-08T05:13:41Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1107.eqiad.wmnet with reason: host reimage 2434032 wikitext text/x-wiki o63e04cci9rfqcji3w522pybwe98ure 2434034 2434032 2026-07-08T05:17:38Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1124.eqiad.wmnet with reason: host reimage 2434034 wikitext text/x-wiki 3fw9au33e8zxh5m851b4e1l01chr0jp 2434035 2434034 2026-07-08T05:21:41Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1107.eqiad.wmnet with reason: host reimage 2434035 wikitext text/x-wiki 5nw4ocvzhc7b0a336y5hnazd97zpgji 2434037 2434035 2026-07-08T05:31:25Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1101.eqiad.wmnet with OS trixie 2434037 wikitext text/x-wiki 3hqabhvv1rhjhlag7otu0en9m6m2851 2434038 2434037 2026-07-08T05:35:51Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1124.eqiad.wmnet with OS trixie 2434038 wikitext text/x-wiki bf1diirumdvgiyov3kplc81xn2myk72 2434039 2434038 2026-07-08T05:42:14Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1107.eqiad.wmnet with OS trixie 2434039 wikitext text/x-wiki 9ho0slgriserjy19wdk1b4s2vocan0x 2434040 2434039 2026-07-08T06:38:02Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1125.eqiad.wmnet with OS trixie 2434040 wikitext text/x-wiki 40loxb4y6b6kqyc6nav4uka2m12o7vp 2434041 2434040 2026-07-08T06:40:34Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host cirrussearch1111.eqiad.wmnet 2434041 wikitext text/x-wiki ph8ksmzz8by0tensrfv5f1r3pjbd65d 2434042 2434041 2026-07-08T06:40:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host cirrussearch1111.eqiad.wmnet 2434042 wikitext text/x-wiki jf2712ki7nnnluvu9vphdwto5ym3l3y 2434044 2434042 2026-07-08T06:50:17Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1125.eqiad.wmnet with reason: host reimage 2434044 wikitext text/x-wiki f7qx6yblpnm7pyiem6hqs4im763o7y3 2434045 2434044 2026-07-08T06:52:51Z Stashbot 7414 elukey: upgrade all trixie hosts to pywmflib 3.1 - T430552 2434045 wikitext text/x-wiki bjy4ze7wwz33ux0rrdymmzlu6pxtd8n 2434046 2434045 2026-07-08T06:54:22Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1125.eqiad.wmnet with reason: host reimage 2434046 wikitext text/x-wiki fpg6cwyslbxev005hqxv4lbzqdex6fj 2434048 2434046 2026-07-08T06:56:51Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1005.eqiad.wmnet with OS bookworm 2434048 wikitext text/x-wiki plogjzrcwb1e5wksp382j42dhduzb0l 2434049 2434048 2026-07-08T07:07:53Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-etcd1005.eqiad.wmnet with reason: host reimage 2434049 wikitext text/x-wiki a9z4c7sc5xup5mtx7qygdp3jdsgslox 2434050 2434049 2026-07-08T07:13:32Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1005.eqiad.wmnet with reason: host reimage 2434050 wikitext text/x-wiki svndc1fbrabo1aa61o397gbavkl559s 2434051 2434050 2026-07-08T07:13:44Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1125.eqiad.wmnet with OS trixie 2434051 wikitext text/x-wiki 9j4i2txluyqory9rh2wq3fn1ovhkg4r 2434052 2434051 2026-07-08T07:21:33Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1005.eqiad.wmnet with OS trixie 2434052 wikitext text/x-wiki aimyrfegn2p329nt873qfc0pdcn0vce 2434053 2434052 2026-07-08T07:29:09Z Stashbot 7414 moritzm: installing gnutls28 security updates 2434053 wikitext text/x-wiki 2i9wxhqn86ziq0tjod84nbdot8vjtix 2434054 2434053 2026-07-08T07:36:51Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet with OS bookworm 2434054 wikitext text/x-wiki frsa1btx6lbc6j5dic76tnznvw6tcna 2434055 2434054 2026-07-08T07:38:20Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1005.eqiad.wmnet with reason: host reimage 2434055 wikitext text/x-wiki o9ncijc32yrko8mep6p6trelg1bzvi0 2434056 2434055 2026-07-08T07:44:35Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1005.eqiad.wmnet with reason: host reimage 2434056 wikitext text/x-wiki 6cuydmmr61a6p0ackakkhamk49e7z90 2434057 2434056 2026-07-08T07:56:33Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack B3 2434057 wikitext text/x-wiki j67wwaexhs7jikhuh96kvktfhzmtke6 2434058 2434057 2026-07-08T08:03:02Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 21 hosts with reason: codfw rack B3 depool for maintenance 2434058 wikitext text/x-wiki k8nr9j59mu5k8c0t66bp6ybkhkvrm4v 2434059 2434058 2026-07-08T08:03:44Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1005.eqiad.wmnet with OS trixie 2434059 wikitext text/x-wiki otcf2a2fs0zycacrc41sauun02iitdg 2434060 2434059 2026-07-08T08:05:08Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-fe1004.eqiad.wmnet with OS trixie 2434060 wikitext text/x-wiki j53j107ic0kgx1lgugo44jth1lv4lpd 2434061 2434060 2026-07-08T08:05:40Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool es2051: codfw rack B3 depool for maintenance 2434061 wikitext text/x-wiki gdhjegxkayyfcyrqytht664eb8dxxek 2434062 2434061 2026-07-08T08:06:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: codfw rack B3 depool for maintenance 2434062 wikitext text/x-wiki 7yq0smzbiibfvhrnff5oci6dyk6339v 2434064 2434062 2026-07-08T08:07:22Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet 2434064 wikitext text/x-wiki e03t5k896zw39iow337av8jovap115t 2434066 2434064 2026-07-08T08:13:37Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet 2434066 wikitext text/x-wiki eajevczs8u56r8pzbgreg7xyayj1etn 2434067 2434066 2026-07-08T08:15:14Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet 2434067 wikitext text/x-wiki tkq02dzzgd26yftm1l7okukt1z4qdsc 2434068 2434067 2026-07-08T08:15:16Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-b3-codfw,lsw1-b3-codfw IPv6,lsw1-b3-codfw.mgmt with reason: Switch maintenance 2434068 wikitext text/x-wiki cinlnj9hzhfyntqildb5ij537ukgdck 2434069 2434068 2026-07-08T08:15:52Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet 2434069 wikitext text/x-wiki 8bza064ii7gfzyc7q0x29nn1g267wkw 2434070 2434069 2026-07-08T08:15:53Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for codfw rack B3 2434070 wikitext text/x-wiki o6nqxj9bl2xfzz4dq27jext4fjd97b2 2434071 2434070 2026-07-08T08:16:52Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis minwikiquote in section s5 2434071 wikitext text/x-wiki as0z5lqs07221qu4jar20uvxhj4c5gs 2434072 2434071 2026-07-08T08:17:46Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.10 refs T430829 2434072 wikitext text/x-wiki 35kgzy6u8cobph21xhu4ygxlh0zhr7x 2434073 2434072 2026-07-08T08:19:17Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis minwikiquote in section s5 2434073 wikitext text/x-wiki 0mtd895q51ltd6371hs1nwf0xboyoer 2434074 2434073 2026-07-08T08:19:21Z Stashbot 7414 XioNoX: lsw1-b3-codfw> request system reboot - T430909 2434074 wikitext text/x-wiki 1ehlou7ap8nngvmjpc9g9jvntrsji8p 2434075 2434074 2026-07-08T08:22:41Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis minwikiquote in section s5 2434075 wikitext text/x-wiki 5gadlr88jz2ynr69u7kkspoevmsim51 2434076 2434075 2026-07-08T08:23:10Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2434076 wikitext text/x-wiki eq06mwtvttib14bojwuzqwk1kjy7w6m 2434077 2434076 2026-07-08T08:23:12Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1004.eqiad.wmnet with reason: host reimage 2434077 wikitext text/x-wiki i5mpsiqmq1ssyjz536yqxng0um9m5al 2434079 2434077 2026-07-08T08:27:53Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1004.eqiad.wmnet with reason: host reimage 2434079 wikitext text/x-wiki dmayq5ra3w9sed9driuf9rlbxixxp6i 2434080 2434079 2026-07-08T08:30:03Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis minwikiquote in section s5 2434080 wikitext text/x-wiki q2tvkna7cwxbhoxlq6sq9l1mhg5dt6q 2434081 2434080 2026-07-08T08:30:52Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis minwikiquote in section s5 2434081 wikitext text/x-wiki 9nmpgqag0ej0tdw5i3mvl9mw1w3u06l 2434082 2434081 2026-07-08T08:32:33Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis minwikiquote in section s5 2434082 wikitext text/x-wiki bgm3igdwkskk6rzm5cz29xpyf34wuil 2434083 2434082 2026-07-08T08:33:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis minwikiquote in section s3 2434083 wikitext text/x-wiki dy4gy2vwcl3w2lmd5yaxo0jo5njlm56 2434084 2434083 2026-07-08T08:35:18Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis minwikiquote in section s3 2434084 wikitext text/x-wiki djfkvmx5ewbawf9yrm7go8h5fw6my9x 2434086 2434084 2026-07-08T08:38:30Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2434086 wikitext text/x-wiki 695iplihvv2medr2a12zrzirwskz1s5 2434087 2434086 2026-07-08T08:38:36Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply 2434087 wikitext text/x-wiki 2spzqgu45qyzesahlgbg4ttxzp5cvpd 2434088 2434087 2026-07-08T08:40:58Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet 2434088 wikitext text/x-wiki rr3mtuyv4v1sb3v2m6f2c2ojkvsew3l 2434089 2434088 2026-07-08T08:41:53Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet 2434089 wikitext text/x-wiki 9zjkrolyxpn967vldpxcvo304xbd24t 2434090 2434089 2026-07-08T08:42:27Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet 2434090 wikitext text/x-wiki nitap1v50ti3wouvikfr7ci2mfesnal 2434092 2434090 2026-07-08T08:43:18Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet 2434092 wikitext text/x-wiki 9it81cj2q9cvbpmz3zw1kccvmkfyu2f 2434093 2434092 2026-07-08T08:43:20Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet 2434093 wikitext text/x-wiki byu4wg5ok80qejg59zyoxw3qvtmlsxr 2434094 2434093 2026-07-08T08:44:17Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet 2434094 wikitext text/x-wiki 6nce123if38218bzuei445jap8ft91v 2434095 2434094 2026-07-08T08:44:25Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet 2434095 wikitext text/x-wiki n41gwt2d76r0rnucjfja0xdznx60q6l 2434096 2434095 2026-07-08T08:45:35Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool es2051: codfw rack B3 pool after maintenance 2434096 wikitext text/x-wiki 96v70imr3tswfxw280u7tcn0qehyyoc 2434098 2434096 2026-07-08T08:50:08Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart 2434098 wikitext text/x-wiki kguztgu2iat9s47t464zkezrsc4sse1 2434099 2434098 2026-07-08T08:50:09Z Stashbot 7414 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) 2434099 wikitext text/x-wiki fzmb4phxtqc3c74m02zynsm4tz0vczs 2434100 2434099 2026-07-08T08:50:24Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart 2434100 wikitext text/x-wiki nhciro59xu88appqxv9qduup9sg2ut3 2434101 2434100 2026-07-08T08:50:28Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1004.eqiad.wmnet with OS trixie 2434101 wikitext text/x-wiki cc3whcq8hvhigxlcyd6so8b8b3vbdnn 2434102 2434101 2026-07-08T08:54:39Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.10 refs T430829 2434102 wikitext text/x-wiki 4xcwdkji5qvt1avx6s2n8co7o8f72jp 2434104 2434102 2026-07-08T08:57:26Z Stashbot 7414 topranks: merge patch to shift eqiad <-> esams traffic onto new 40G circuit 2434104 wikitext text/x-wiki dtd017ynyyxsyrwk4c594ptxrkve5g1 2434105 2434104 2026-07-08T09:02:34Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) 2434105 wikitext text/x-wiki ba7k918i4lot4p2iu9gk4e6ne0dckxg 2434106 2434105 2026-07-08T09:07:22Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet 2434106 wikitext text/x-wiki g6odyy356i3qybwkdv19vcjhf2gudgg 2434107 2434106 2026-07-08T09:07:23Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet 2434107 wikitext text/x-wiki by70i6w0lya2jcmwosuuk8buazhwpb8 2434111 2434107 2026-07-08T09:26:42Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434111 wikitext text/x-wiki qf8hcd964rit8dr9p0ookohqt70ego5 2434112 2434111 2026-07-08T09:27:09Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434112 wikitext text/x-wiki pe9tyboljljjosqpyix6tnfkl0azvzl 2434113 2434112 2026-07-08T09:31:04Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: codfw rack B3 pool after maintenance 2434113 wikitext text/x-wiki juckfwhanfnt1aoz38554s702nef5t4 2434114 2434113 2026-07-08T09:32:49Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet 2434114 wikitext text/x-wiki 80j7x6on9lk9xzrfvrxer8pzo2kpftv 2434115 2434114 2026-07-08T09:32:50Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet 2434115 wikitext text/x-wiki ml3ily13xh8kqox3n2s3k94hrwsmi4u 2434116 2434115 2026-07-08T09:33:50Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 41 days, 15:00:00 on db2252.codfw.wmnet with reason: Test 2434116 wikitext text/x-wiki 1xop5xnw10pocnc448sgp2cm0naw98q 2434119 2434116 2026-07-08T09:34:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet 2434119 wikitext text/x-wiki swci6bkoz9a6hy4cbydkabnnw2jtmcn 2434120 2434119 2026-07-08T09:34:37Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet 2434120 wikitext text/x-wiki 33iuonls7utopic5iipyvktpqv1ffhy 2434121 2434120 2026-07-08T09:35:15Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply 2434121 wikitext text/x-wiki r0jceed09m9b8ord2iyw3fjw46bhjmn 2434123 2434121 2026-07-08T09:43:18Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply 2434123 wikitext text/x-wiki f3vc9fievptij7oe7eic6uj74c3u18m 2434124 2434123 2026-07-08T09:43:22Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply 2434124 wikitext text/x-wiki g4c19cxjsnynmd54wt3f6bif9riyqge 2434125 2434124 2026-07-08T09:43:35Z Stashbot 7414 cgoubert@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply 2434125 wikitext text/x-wiki ldu8xtl6p4f8l1y9k6qxpzln8033rhg 2434128 2434125 2026-07-08T10:01:18Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet 2434128 wikitext text/x-wiki czroyraiw9xet6hzo5whli201f0hjhn 2434129 2434128 2026-07-08T10:01:37Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1074.eqiad.wmnet with OS trixie 2434129 wikitext text/x-wiki ee5mkdmshbfl8ph6ilhf7qnhnsnmn56 2434130 2434129 2026-07-08T10:01:41Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet 2434130 wikitext text/x-wiki bxak43m4owtn3vual8jm1wiz5qxzn3w 2434131 2434130 2026-07-08T10:04:45Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet 2434131 wikitext text/x-wiki 09d7o56urezq3o5ga4su6j9lo8gfic8 2434132 2434131 2026-07-08T10:12:08Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet 2434132 wikitext text/x-wiki cjvpl6myjuq1z9etxm163d7ylxqv9zb 2434133 2434132 2026-07-08T10:12:13Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1005.eqiad.wmnet with OS trixie 2434133 wikitext text/x-wiki s61nyhlvubqmlb70l8sbttkaadwwn21 2434134 2434133 2026-07-08T10:17:20Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1074.eqiad.wmnet with reason: host reimage 2434134 wikitext text/x-wiki a77w0urltqthioq90noa9yuxk5e4u5l 2434135 2434134 2026-07-08T10:24:22Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1074.eqiad.wmnet with reason: host reimage 2434135 wikitext text/x-wiki cndeolsa8zdkqa3iybdmhn2ltel8f81 2434136 2434135 2026-07-08T10:25:55Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1005.eqiad.wmnet with reason: host reimage 2434136 wikitext text/x-wiki saezoq7uf6221qion7g8hrp1ggikzo4 2434137 2434136 2026-07-08T10:29:22Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1005.eqiad.wmnet with reason: host reimage 2434137 wikitext text/x-wiki 4bktn9tfljgmit10jgkiiybdnwfkudr 2434138 2434137 2026-07-08T10:32:30Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet 2434138 wikitext text/x-wiki 57qjt8xh5co8sslyoliwmhgpxjcar94 2434139 2434138 2026-07-08T10:40:54Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1075.eqiad.wmnet with OS trixie 2434139 wikitext text/x-wiki 0nx5h1hek7rn2xmbklsduz1gvgnqxcu 2434140 2434139 2026-07-08T10:45:01Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1005.eqiad.wmnet with OS trixie 2434140 wikitext text/x-wiki gi5aoqhejkp9uzgdii1a46mywq8tjb0 2434141 2434140 2026-07-08T10:46:24Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1074.eqiad.wmnet with OS trixie 2434141 wikitext text/x-wiki 1hzn6ogejq4qhabp5p9eanq06ldn65d 2434142 2434141 2026-07-08T10:47:43Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1076.eqiad.wmnet with OS trixie 2434142 wikitext text/x-wiki ekx942g9t0zmhqo1dtu8y28m2qke8k9 2434143 2434142 2026-07-08T10:56:19Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1075.eqiad.wmnet with reason: host reimage 2434143 wikitext text/x-wiki 9mddin6k25komxbzf3a6xkdjdgdzpdw 2434144 2434143 2026-07-08T11:00:19Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1075.eqiad.wmnet with reason: host reimage 2434144 wikitext text/x-wiki 0bi0quxj1xtbun5sogfmc4tv30nellg 2434145 2434144 2026-07-08T11:03:25Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1076.eqiad.wmnet with reason: host reimage 2434145 wikitext text/x-wiki kjrpxd7m4dgl8ldfy46whrlgr29kjw1 2434146 2434145 2026-07-08T11:08:23Z Stashbot 7414 moritzm: installing Linux 6.1.176 on Bookworm servers 2434146 wikitext text/x-wiki tey0sk0k17nzcguulpied974js7hocb 2434147 2434146 2026-07-08T11:08:41Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1076.eqiad.wmnet with reason: host reimage 2434147 wikitext text/x-wiki mw7wippwvy4s307uqdw3xkz3w1wwz6t 2434148 2434147 2026-07-08T11:19:05Z Stashbot 7414 moritzm: temporarily remove ganeti2031 from codfw cluster T430910 2434148 wikitext text/x-wiki pkbkajhbrobumpldxijqzgsq13qqomw 2434149 2434148 2026-07-08T11:19:53Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2047.codfw.wmnet 2434149 wikitext text/x-wiki h5hjr56jis4cwj3drayn3czvirag34f 2434150 2434149 2026-07-08T11:21:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2047.codfw.wmnet 2434150 wikitext text/x-wiki gglgrcagh6duvu73al80arfmsvv7mwv 2434151 2434150 2026-07-08T11:26:10Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1075.eqiad.wmnet with OS trixie 2434151 wikitext text/x-wiki 8epl74y9k34gvy5h6rgtfalh3h4m3gf 2434152 2434151 2026-07-08T11:32:41Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1076.eqiad.wmnet with OS trixie 2434152 wikitext text/x-wiki t2cwl43au4ubvdpsryf77u9qrzickm7 2434153 2434152 2026-07-08T11:38:50Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2434153 wikitext text/x-wiki sk1nso3jr7tvhdpwrx4y55m395ziud7 2434154 2434153 2026-07-08T11:43:09Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie 2434154 wikitext text/x-wiki qa54krv7luz6moq44et4zchs1owk41x 0 17290 2433916 2433419 2026-07-07T17:18:00Z Stashbot 7414 hashar: gerrit: deleted /srv/gerrit/java_pid3571660.hprof 2433916 wikitext text/x-wiki hifkngyxq9sh0otuv9ir5lyyqv5l4hv 2433962 2433916 2026-07-07T19:41:59Z Stashbot 7414 mutante: contint1003/2003 - add jenkins-agent user to docker group; restart jenkins 2433962 wikitext text/x-wiki t9sb26eankqbfu6rxln6xw9p6inegcb 0 20225 2433842 2420528 2026-07-07T15:13:04Z JMonton-WMF 47887 add info about modifying tables. 2433842 wikitext text/x-wiki n1ddmv5960bjb636m31bvnxf0j0ue06 0 22503 2434007 2433375 2026-07-08T00:15:25Z Jon (WMF) 42946 /* Currently reserved values */ 2434007 wikitext text/x-wiki n3a1bsnafuso0x9x7yzdmw7nqfbcbu7 0 22777 2434043 2399617 2026-07-08T06:43:08Z MMuhlenhoff (WMF) 5268 /* Reboot/Shutdown for maintenance a node */ Update docs how to drain a node to mention the cookbook 2434043 wikitext text/x-wiki l6glw74wygdf6lj550pyie3h4obepbu 2434047 2434043 2026-07-08T06:54:45Z MMuhlenhoff (WMF) 5268 /* Node operations */ Revamp the docs how to drain secondaries and how to handle draining two nodes 2434047 wikitext text/x-wiki o8tzwfol9p66kyzo267jyhxbzkx99wu 2434109 2434047 2026-07-08T09:21:38Z MMuhlenhoff (WMF) 5268 /* Reboot/Shutdown for maintenance of a node */ Add a note on cluster rebalances 2434109 wikitext text/x-wiki i9s7f1xln5cqfg1oeq7c7youyg2wpky 0 324433 2433998 2433168 2026-07-07T22:13:03Z BCornwall-WMF 31521 /* Session state at disconnection */ Link to haproxy docs 2433998 wikitext text/x-wiki oltosblu7njg02trnkzh6foxdz98tm7 0 350482 2434005 2398981 2026-07-07T23:00:26Z BCornwall-WMF 31521 Remove pointless storage section, package doesn't even exist any more 2434005 wikitext text/x-wiki quw7gqjinkz8ke8vdctwmyfm1h71n5b 498 442831 2433919 2431099 2026-07-07T17:35:52Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2433919 wikitext text/x-wiki sj1w8sd3d3vtyf14mp2w5lyulsqwme0 2433920 2433919 2026-07-07T17:38:11Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) 2433920 wikitext text/x-wiki rodgp170l2nrud57t6ckf0ar03ytmov 0 444148 2433674 2429570 2026-07-07T12:32:41Z ISarantopoulos-WMF 34784 2433674 wikitext text/x-wiki 5nuv93c5hrukg8jrlwgvqh9mmtsp4cu 0 446558 2433862 2433394 2026-07-07T15:23:07Z AndrewBogott (WMF) 51047 /* Usage and Storage limits */ 2433862 wikitext text/x-wiki 955mbb14pgb4ivz7ejq27g74sal2el2 2433987 2433862 2026-07-07T20:55:43Z BryanDavis 1604 /* Usage and Storage limits */ wikitext to replace html tag formatting 2433987 wikitext text/x-wiki 1yyil0audvoixgl5m7pyncqa5e92lh9 0 448359 2433994 2420686 2026-07-07T21:33:40Z ADancy (WMF) 17694 /* GitLab Runner types */ Add cpu/memory info, and move Cloud Runners to the top since they are the default 2433994 wikitext text/x-wiki fyuxjaebga6fvsz0txdhmgjak509czd 0 449160 2434006 2433455 2026-07-08T00:01:05Z Dexbot 30554 Bot: Updating the report 2434006 wikitext text/x-wiki 7645k21d4gef2e2rl16rcjpfnlg06kk 0 455253 2434091 2429955 2026-07-08T08:42:54Z Ladsgroup 1264 /* Sanitize the Wiki Data */ codfw sanitarium doesn't exist anymore 2434091 wikitext text/x-wiki jz9ejxsfhsgcqhpp26pbuq38je72glj 0 456086 2433648 2427330 2026-07-07T12:16:16Z AKlapper (WMF) 421 +Phabricator project 2433648 wikitext text/x-wiki nmzxoxrpg0nl6zrb6dr4x2q9ui0qnaj 0 459290 2433712 2433043 2026-07-07T13:11:18Z JAllemandou (WMF) 37709 Add new item to deploy 2433712 wikitext text/x-wiki 57hzu6r38eobopjfvm5to56sy5dtujz 2433844 2433712 2026-07-07T15:15:28Z JMonton-WMF 47887 new deployment 2433844 wikitext text/x-wiki 593vh1b8l1w7ayuphvlo5dnvduqeg05 2433845 2433844 2026-07-07T15:16:20Z JMonton-WMF 47887 fix dates 2433845 wikitext text/x-wiki hbpnz8gr03gidhh9ky7zlj3rrjtkit0 0 459699 2434003 2433478 2026-07-07T22:29:28Z Stashbot 7414 jgleeson: payments-wiki upgraded from ece97dca to f34901d7 2434003 wikitext text/x-wiki qpszohryrfnklhlvlj09hg0w3994go5 2434016 2434003 2026-07-08T01:12:00Z Stashbot 7414 larssandergreen: civicrm upgraded from 941505c3 to 0057f9a1 2434016 wikitext text/x-wiki 70cy97qpgrkvi076z6gz0slvzn050xb 2434025 2434016 2026-07-08T03:48:53Z Stashbot 7414 larssandergreen: civicrm upgraded from 0057f9a1 to ead61459 2434025 wikitext text/x-wiki 2mvt1cobdn95250nvco7hslfdk73s47 1 459889 2433796 2433387 2026-07-07T14:29:19Z Benjavalero 4470 /* Access from Toolforge */ Reply 2433796 wikitext text/x-wiki 4ri4ol8p09qitit7jo23d8ph2354yhb 1 460407 2433999 2432974 2026-07-07T22:14:09Z BCornwall-WMF 31521 /* Proposal for deletion */ Reply 2433999 wikitext text/x-wiki lkke7rrmz34fg0r0hq7pnt437kpjyt0 1 460409 2433918 2433467 2026-07-07T17:24:29Z Jheald 3726 /* non-replacement of label service ? */ ctrl+space, not tab, for query completion 2433918 wikitext text/x-wiki lemml2tm1n9cyzx0rgitu4sp3fmccdm 1 460426 2433995 2026-07-07T21:52:21Z BCornwall-WMF 31521 /* Proposal to make this a disambiguation page */ new section 2433995 wikitext text/x-wiki 69ctqru1vp47pgqylmt4b8fpr3iu4rq 2 460427 2434036 2026-07-08T05:29:33Z ArielGlenn 11 mw interfaces chore doc first draft attempt to wikify partial text, still some broken formatting 2434036 wikitext text/x-wiki nmzj9se4kd4bbit931jxvo822twpcp9 3 460428 2434097 2026-07-08T08:50:05Z StrikerBot 8475 Welcome to Toolforge! 2434097 wikitext text/x-wiki fvgsiad3xlaylduvi4zpu3bdyrx9irs 116 460429 2434110 2026-07-08T09:22:55Z Viktoria Hillerud (WMSE) 40519 Created page with "{{Tool |name = Wikispeech Speech Data Collector |title = |description = |website = https://wikispeech-sdc.toolforge.org/ |keywords = phabricator, python, admin |author = [[User:Sebastian_Berlin_(WMSE)|Sebastian Berlin]], [[User:User:Viktoria_Hillerud_(WMSE)|Viktoria Hillerud]] |username = |maintainer = [[User:Sebastian_Berlin_(WMSE)]], [[User:User:Viktoria_Hillerud_(WMSE)|Viktoria Hillerud]] |repository = |license = |image..." 2434110 wikitext text/x-wiki 5rg2ud3mez1sjkswvgtbaz8r9yfhwf9 2434117 2434110 2026-07-08T09:34:00Z Viktoria Hillerud (WMSE) 40519 2434117 wikitext text/x-wiki gphiwd4ngdzlqijbvsyawwwrdxinkox 2434118 2434117 2026-07-08T09:34:28Z Viktoria Hillerud (WMSE) 40519 2434118 wikitext text/x-wiki 401ssu8ufgfafg0hbjw9igkfqrfo80i 2434122 2434118 2026-07-08T09:42:55Z Viktoria Hillerud (WMSE) 40519 Added config list with config value 2434122 wikitext text/x-wiki 8ibqpxo0s1eptoca41xiknn6vgh3pxw 2434126 2434122 2026-07-08T09:52:16Z Viktoria Hillerud (WMSE) 40519 Added description of toolforge tool 2434126 wikitext text/x-wiki 1erx978g2rk91cuanx6q0g2luslz4f8 2434127 2434126 2026-07-08T09:53:55Z Viktoria Hillerud (WMSE) 40519 added phab project 2434127 wikitext text/x-wiki rhexl49f3jy5wwh90kobd779esm6dxs