Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.10 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2434774 2434108 2026-07-09T09:00:52Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2434774 wikitext text/x-wiki 6tegynvtt025a6k34l9jaglerl37mhd 2434784 2434774 2026-07-09T09:14:19Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2434784 wikitext text/x-wiki fj2tf1dcosi4eeby291ezzcbuhzllj5 2434812 2434784 2026-07-09T09:46:12Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2434812 wikitext text/x-wiki 5b0szx2vqr953fxnwkq4ehebn0z8lpu 2434845 2434812 2026-07-09T10:00:49Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2434845 wikitext text/x-wiki 7kw1zalv098r1pks0h7pfr605tjcdd6 2434914 2434845 2026-07-09T11:24:58Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes 2434914 wikitext text/x-wiki kkc0u0h9odbwt63d7wtlo117h4bihmi 2434916 2434914 2026-07-09T11:27:29Z Stashbot 7414 taavi: rebooting bastions for kernel upgrades 2434916 wikitext text/x-wiki i7psm6fevzpa5zswtf5w46bus1urizx 0 3387 2434403 2429281 2026-07-08T16:15:59Z LSandergreen-WMF 46345 /* How we work (Team Processes) */ 2434403 wikitext text/x-wiki 47a34dfz56c4kgvijzk054hvh451xlz 0 4108 2434158 2434065 2026-07-08T12:20:42Z ScheduleDeploymentBot 37566 Add [[gerrit:1308622]] to Wednesday, July 08 UTC afternoon backport window 2434158 wikitext text/x-wiki sg4ywla3t9c5ax1nefcm1ucrdxj8ev2 2434159 2434158 2026-07-08T12:22:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1307836]] to Wednesday, July 08 UTC afternoon backport window 2434159 wikitext text/x-wiki a16jqwdj001kbguxlrh9nsvmyc2eral 2434329 2434159 2026-07-08T15:00:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1308183]] to Wednesday, July 08 UTC late backport window 2434329 wikitext text/x-wiki f4npcx0uv0jco25u8xdouj88ajj6mcj 2434440 2434329 2026-07-08T17:05:38Z ScheduleDeploymentBot 37566 Add [[gerrit:1308687]] to Thursday, July 09 UTC morning backport window 2434440 wikitext text/x-wiki m7od6p31eqd0jlf03hqn37r5k9ajkpg 2434582 2434440 2026-07-08T19:48:54Z RLazarus (WMF) 15215 /* Thursday, July 9 */ 2434582 wikitext text/x-wiki 6r2r39u11mkzd1lsmf1nloj3jxf6a22 2434851 2434582 2026-07-09T10:11:45Z ScheduleDeploymentBot 37566 Add [[gerrit:1309112]] to Thursday, July 09 UTC afternoon backport window 2434851 wikitext text/x-wiki plb5rm3oi6otdugb6x9gum7c6myhpjr 2434870 2434851 2026-07-09T10:21:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1309101]] to Thursday, July 09 UTC afternoon backport window 2434870 wikitext text/x-wiki gu6fpejzu585rvv59ah485j4p0yc15l 2434875 2434870 2026-07-09T10:25:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1309122]] to Thursday, July 09 UTC afternoon backport window 2434875 wikitext text/x-wiki snmohjy1vctks6xqz6k7hmm6xm6wifd 498 6726 2434755 2434085 2026-07-09T08:17:53Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2434755 wikitext text/x-wiki szbikyb4xgthk5ttabgfq7kle4y586t 2434757 2434755 2026-07-09T08:29:12Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2434757 wikitext text/x-wiki 4mucvvwdr184tjh04f0pponlwtrjz7r 2434795 2434757 2026-07-09T09:26:01Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2434795 wikitext text/x-wiki 0mdtumk1qh2b0ppmmy42fmx1azque0n 2434805 2434795 2026-07-09T09:37:19Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2434805 wikitext text/x-wiki 5ec54cxov4f7j467smjk0wjf0mrqcuw 2434913 2434805 2026-07-09T11:24:56Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes 2434913 wikitext text/x-wiki gk0lupd7gykgh0hpqhokpvxbnm6faxh 2434919 2434913 2026-07-09T11:52:19Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes 2434919 wikitext text/x-wiki 3lw255wb94kuyu2xe66adid49w6ftx7 0 7478 2434891 2418591 2026-07-09T10:43:29Z FGiunchedi (WMF) 39205 /* Implementation */ Expand repo name in hyperlink 2434891 wikitext text/x-wiki 3gra4u3daxz5yb3x4d6n3gwckxilbuj 0 7919 2434155 2434154 2026-07-08T12:01:55Z Stashbot 7414 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1006.eqiad.wmnet with OS trixie 2434155 wikitext text/x-wiki e6p3qwk9m1gg5wf8zhlixqpy9cs37z4 2434156 2434155 2026-07-08T12:02:34Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie 2434156 wikitext text/x-wiki jlo7n4eqnzaflke2hjfcq8x8omhylbw 2434157 2434156 2026-07-08T12:07:34Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434157 wikitext text/x-wiki i0h3v2fi1pl18wnhwofqj92nm7loy7a 2434161 2434157 2026-07-08T12:27:17Z Stashbot 7414 mvernon@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host thanos-be1006.eqiad.wmnet with OS trixie 2434161 wikitext text/x-wiki k13b4e3d1e8gwli42xqzycu079nt0d3 2434163 2434161 2026-07-08T12:27:43Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie 2434163 wikitext text/x-wiki 7fc1vfnrkv16um4miabn4ed338y75sp 2434164 2434163 2026-07-08T12:34:13Z Stashbot 7414 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1006.eqiad.wmnet with OS trixie 2434164 wikitext text/x-wiki gl31x3yfysnofbivvzs2ydo9vzu9o7z 2434165 2434164 2026-07-08T12:34:45Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2434165 wikitext text/x-wiki t6mvvtrthndyci6eqnh8g0w1y9ew74g 2434166 2434165 2026-07-08T12:36:46Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1051.eqiad.wmnet 2434166 wikitext text/x-wiki fbu7a5bfdiv73ljmsw7tlocip7oge23 2434167 2434166 2026-07-08T12:36:50Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet 2434167 wikitext text/x-wiki 1d0vtjwooiyfclbs0ocqlnllug5qhsz 2434168 2434167 2026-07-08T12:37:26Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet 2434168 wikitext text/x-wiki lx76zlg8j6wkkt73wjv61rxff61adup 2434169 2434168 2026-07-08T12:38:16Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1051.eqiad.wmnet with OS trixie 2434169 wikitext text/x-wiki 3pn5gbutos36icatij31b72sluput6e 2434170 2434169 2026-07-08T12:38:44Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1051 2434170 wikitext text/x-wiki 6hi5ocl6d08aufmd86nld280t97dxc9 2434171 2434170 2026-07-08T12:38:54Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.netbox 2434171 wikitext text/x-wiki mhr04sw0m20p935k1o32k24ay5706ih 2434172 2434171 2026-07-08T12:43:06Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1051 - blake@cumin1003" 2434172 wikitext text/x-wiki ntsyg8pw8xk9s5b1m5rd2kwq1jmbx3g 2434173 2434172 2026-07-08T12:43:11Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1051 - blake@cumin1003" 2434173 wikitext text/x-wiki gsf9o7g7mt443xmg3d8s8mzgx8ymrht 2434174 2434173 2026-07-08T12:43:12Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434174 wikitext text/x-wiki 5d84o4gjj1cgb4mwc8lbafvul6b0of6 2434175 2434174 2026-07-08T12:43:13Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1051.eqiad.wmnet 46.32.64.10.in-addr.arpa 6.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434175 wikitext text/x-wiki rp1op5yk97mjz9rnrexgl37zqd1gdpv 2434176 2434175 2026-07-08T12:43:15Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1051.eqiad.wmnet 46.32.64.10.in-addr.arpa 6.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434176 wikitext text/x-wiki 831j1flhk1z9eyoyze8j9ncqm03s04e 2434177 2434176 2026-07-08T12:43:16Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1051 2434177 wikitext text/x-wiki f1wst7lh745qe87mm5c25hdqeq55bp1 2434178 2434177 2026-07-08T12:43:26Z Stashbot 7414 moritzm: installing Python 3.11 security updates 2434178 wikitext text/x-wiki cw6g7p0q2sjr3zkoibjjajow3n47rux 2434179 2434178 2026-07-08T12:43:40Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1051 2434179 wikitext text/x-wiki 977ikebu8rapzprgtdnjdlhywy1joy8 2434180 2434179 2026-07-08T12:43:41Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1051 2434180 wikitext text/x-wiki t48ewq6l6mbnkn43b2p84njli4b8x03 2434181 2434180 2026-07-08T12:50:44Z Stashbot 7414 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2434181 wikitext text/x-wiki b9m3rv004mv7bx9gk0ft3tvkusex0vi 2434182 2434181 2026-07-08T12:52:28Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS trixie 2434182 wikitext text/x-wiki qzz3njk18w61xxu5xbtm83cfh275uo5 2434183 2434182 2026-07-08T12:58:19Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-worker@codfw 2434183 wikitext text/x-wiki 2lrhg47fk1g66g268ajycyhkqwh36tf 2434184 2434183 2026-07-08T13:01:20Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage 2434184 wikitext text/x-wiki 4ujspt2v0cmsd4jpn017nsgmu3gue2y 2434186 2434184 2026-07-08T13:04:46Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2434186 wikitext text/x-wiki tr4wusqyz6tskisc6d397n3anl6ecmt 2434187 2434186 2026-07-08T13:04:52Z Stashbot 7414 moritzm: installing jq security updates 2434187 wikitext text/x-wiki sec5u5953n7ke6lka0djzrfbahuu9rg 2434188 2434187 2026-07-08T13:05:15Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage 2434188 wikitext text/x-wiki qlc9xxw7jl5clx0r5d0wbwvhs85wuyz 2434189 2434188 2026-07-08T13:05:22Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1081.eqiad.wmnet with OS trixie 2434189 wikitext text/x-wiki 80opekbqu0pe4msaq8vvysdm3g3498w 2434190 2434189 2026-07-08T13:05:47Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2434190 wikitext text/x-wiki 8fjndcja0lc6bekh5yyd5ez4mhlb7k3 2434191 2434190 2026-07-08T13:05:48Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-worker@codfw 2434191 wikitext text/x-wiki q1gm37i8edpihla2kxp3syeukqd5mn9 2434193 2434191 2026-07-08T13:10:02Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1308553|Deploy IRS to enwiki (T431316)]] 2434193 wikitext text/x-wiki 11wibpezq9x2g6zjb4ull38p2vz68mc 2434194 2434193 2026-07-08T13:12:12Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1118.eqiad.wmnet with OS trixie 2434194 wikitext text/x-wiki mjz8186om74dnafhh099ykd1nv3d43j 2434195 2434194 2026-07-08T13:12:18Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1308553|Deploy IRS to enwiki (T431316)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2434195 wikitext text/x-wiki l48x3q2p7vxpwzvyevfbskh4kck5ms7 2434197 2434195 2026-07-08T13:12:33Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1006.eqiad.wmnet with reason: host reimage 2434197 wikitext text/x-wiki 6vrf3nm81djus4xwnjpbbjkc7b22507 2434198 2434197 2026-07-08T13:12:40Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1118 2434198 wikitext text/x-wiki 7p921eltg6nodjdxxss73rqlh8fncvc 2434199 2434198 2026-07-08T13:13:56Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.dns.netbox 2434199 wikitext text/x-wiki 4z3etq5jkr1glg2bo52qo4ssh0k6dyb 2434201 2434199 2026-07-08T13:15:59Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) 2434201 wikitext text/x-wiki p9hzjyzcrbkts9q1xwpy9fcqd84q5rg 2434202 2434201 2026-07-08T13:16:12Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough 2434202 wikitext text/x-wiki fe6gxu2v008gqj4mlrajlgwpqba498r 2434205 2434202 2026-07-08T13:17:49Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2434205 wikitext text/x-wiki bs28weew32lxt1pey2ztikzqqprge38 2434206 2434205 2026-07-08T13:18:49Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1118 - atsuko@cumin1003" 2434206 wikitext text/x-wiki nw9rzqkt31w6r5upmeau20ho4ui53eq 2434207 2434206 2026-07-08T13:18:53Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1118 - atsuko@cumin1003" 2434207 wikitext text/x-wiki l76qpjngoyh2l8d8q8jj1kielmhqjdz 2434208 2434207 2026-07-08T13:18:54Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434208 wikitext text/x-wiki tc25h5wk3v560983qq2rxi8ek28aws8 2434209 2434208 2026-07-08T13:18:55Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1118.eqiad.wmnet 90.32.64.10.in-addr.arpa 0.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434209 wikitext text/x-wiki bjrmw46vp0sz67tcp7n0sy98bt4f4bs 2434210 2434209 2026-07-08T13:18:57Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1118.eqiad.wmnet 90.32.64.10.in-addr.arpa 0.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434210 wikitext text/x-wiki bvrs23oixp27k78l6b77e2dn9ealasn 2434211 2434210 2026-07-08T13:18:58Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1118 2434211 wikitext text/x-wiki 22g0diacsgkz8d4gh3jlinzbfpwryjj 2434212 2434211 2026-07-08T13:19:00Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1006.eqiad.wmnet with reason: host reimage 2434212 wikitext text/x-wiki tm8rm0e0ampouxws2wxqog4gxfz6otq 2434213 2434212 2026-07-08T13:21:01Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage 2434213 wikitext text/x-wiki r60jeituwze2gjm11dm6s2jdr3rvupu 2434215 2434213 2026-07-08T13:22:14Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308553|Deploy IRS to enwiki (T431316)]] (duration: 12m 12s) 2434215 wikitext text/x-wiki 0k2cl8nkba1d3tt3aju55t4j8h0y84o 2434216 2434215 2026-07-08T13:24:07Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1118 2434216 wikitext text/x-wiki kwlpao7d3mw70g3s0ith07al19q4tv4 2434217 2434216 2026-07-08T13:24:08Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1118 2434217 wikitext text/x-wiki 9ojgvt74n3g2xfmirrwd2dqprmpnnvl 2434219 2434217 2026-07-08T13:24:57Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage 2434219 wikitext text/x-wiki f8lm5y6n3juh8n0adevfv8i0jzd93kn 2434222 2434219 2026-07-08T13:26:34Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1051.eqiad.wmnet with OS trixie 2434222 wikitext text/x-wiki ccwfm8rbzqe77zttblt1s12rnirzrnx 2434223 2434222 2026-07-08T13:27:42Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1308622|Enable Article Guidance extension on itwiki (T431540)]] 2434223 wikitext text/x-wiki iublg8o39bi6w6ojhxt5beej58jdtpq 2434224 2434223 2026-07-08T13:28:19Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434224 wikitext text/x-wiki es7c291pv19n38hts9rmrdz4o2hjr78 2434225 2434224 2026-07-08T13:28:48Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434225 wikitext text/x-wiki 1oib02z4nkvoz24d8r5eh082hbmdtn1 2434226 2434225 2026-07-08T13:29:56Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1308622|Enable Article Guidance extension on itwiki (T431540)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2434226 wikitext text/x-wiki exk45g9yea923xc1kh1uro9mndi09br 2434227 2434226 2026-07-08T13:30:32Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough 2434227 wikitext text/x-wiki mwez33w6tcqd0mj7mio6hy8zl5tvvn9 2434228 2434227 2026-07-08T13:30:43Z Stashbot 7414 moritzm: installing openssh security updates 2434228 wikitext text/x-wiki dvzjdn9i8ld3r7o42widle6uq085bsn 2434229 2434228 2026-07-08T13:30:58Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2434229 wikitext text/x-wiki pb2elc8i8gouio3zb560cpumy4k3u2y 2434230 2434229 2026-07-08T13:34:14Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.reimage for host cirrussearch1119.eqiad.wmnet with OS trixie 2434230 wikitext text/x-wiki 5ug5x8vbd84bthaqp49wx99sl8tbkod 2434231 2434230 2026-07-08T13:34:42Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1119 2434231 wikitext text/x-wiki gje9gjosubcazazqk4q3dn7teq70xye 2434232 2434231 2026-07-08T13:35:29Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1308622|Enable Article Guidance extension on itwiki (T431540)]] (duration: 07m 46s) 2434232 wikitext text/x-wiki fissv6t66pqrpy6oseervsemoi58cle 2434234 2434232 2026-07-08T13:36:09Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.dns.netbox 2434234 wikitext text/x-wiki 5yomfpw3j57z93qqkckrhtr5tnbes8p 2434235 2434234 2026-07-08T13:36:11Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1118.eqiad.wmnet with reason: host reimage 2434235 wikitext text/x-wiki 1niw62yio8ssiee45jd30n8zwdivrhy 2434236 2434235 2026-07-08T13:37:01Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1006.eqiad.wmnet with OS trixie 2434236 wikitext text/x-wiki tvshlb5jyz24jb9cyscony17oru6omf 2434237 2434236 2026-07-08T13:37:29Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2434237 wikitext text/x-wiki t9hnzcpzckxztm92hb4bxlndwk1rlk9 2434239 2434237 2026-07-08T13:39:25Z Stashbot 7414 moritzm: installing krb5 security updates 2434239 wikitext text/x-wiki biqz86up9iz0nflvu6pns5z0khij33x 2434240 2434239 2026-07-08T13:40:25Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1118.eqiad.wmnet with reason: host reimage 2434240 wikitext text/x-wiki 70kcmlii57x32u004piczlvb0ww7xta 2434241 2434240 2026-07-08T13:40:37Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1119 - atsuko@cumin1003" 2434241 wikitext text/x-wiki ddkgye22wqpxxtild6vrg3itvkxgj07 2434242 2434241 2026-07-08T13:40:42Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1119 - atsuko@cumin1003" 2434242 wikitext text/x-wiki 9izbffun8uhclizpi1rfnms7cph9iar 2434243 2434242 2026-07-08T13:40:43Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434243 wikitext text/x-wiki f8vda9c664ztycx6ure19o7omi0fhst 2434244 2434243 2026-07-08T13:40:44Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.dns.wipe-cache cirrussearch1119.eqiad.wmnet 97.32.64.10.in-addr.arpa 7.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434244 wikitext text/x-wiki 4dlq6s6lktv28g8zvke1ja6bmql5yyh 2434245 2434244 2026-07-08T13:40:46Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1119.eqiad.wmnet 97.32.64.10.in-addr.arpa 7.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434245 wikitext text/x-wiki nnr9mizfr2dh02qtjrxuqi4branyeht 2434246 2434245 2026-07-08T13:40:47Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1119 2434246 wikitext text/x-wiki 8madanu4eijto85867ubqq4cd71zo8o 2434247 2434246 2026-07-08T13:41:54Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1119 2434247 wikitext text/x-wiki jiqe66dr6ciznxso2muomkn4kcabwyy 2434248 2434247 2026-07-08T13:41:56Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1119 2434248 wikitext text/x-wiki rtyi18hbpflgdvqg4f71hteean2c7mv 2434250 2434248 2026-07-08T13:45:48Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1081.eqiad.wmnet with OS trixie 2434250 wikitext text/x-wiki qf6ea6izk9j77hy4ta9i763i0ukouzl 2434252 2434250 2026-07-08T13:49:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet 2434252 wikitext text/x-wiki p6p81rfqx0et4bofjrf1r4ftcaza7kh 2434253 2434252 2026-07-08T13:50:22Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet 2434253 wikitext text/x-wiki 11xtlhommx9rxcktzpfplhjc2g0zl2t 2434254 2434253 2026-07-08T13:50:23Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet 2434254 wikitext text/x-wiki 7gpfpb2kduktafwnvc4w858fxiawmnx 2434255 2434254 2026-07-08T13:50:25Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1051.eqiad.wmnet 2434255 wikitext text/x-wiki 2y0kaux85jolgbckbemx0hqfpqyvpmm 2434256 2434255 2026-07-08T13:50:52Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: ml-serve-worker@eqiad 2434256 wikitext text/x-wiki 3idhej75sfmj4f2myx2v6bss75s33sc 2434257 2434256 2026-07-08T13:52:18Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool 2434257 wikitext text/x-wiki fwizitzc9als5713j9sjgsldmdo4o4r 2434260 2434257 2026-07-08T13:52:38Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) 2434260 wikitext text/x-wiki 78cuwlr3b59l6umztps0kiu6mvanfku 2434261 2434260 2026-07-08T13:52:49Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet 2434261 wikitext text/x-wiki rbetjquu8mpncf9jhq6fvftue5rycs0 2434262 2434261 2026-07-08T13:53:27Z Stashbot 7414 atsuko@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1119.eqiad.wmnet with reason: host reimage 2434262 wikitext text/x-wiki 9s57941ewrcvh4f2rufeqwyxiyvjj24 2434263 2434262 2026-07-08T13:54:12Z Stashbot 7414 moritzm: installing libcap2 security updates 2434263 wikitext text/x-wiki b193edm2rodk87uauwehr2bmzg7xi7u 2434264 2434263 2026-07-08T13:57:39Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1119.eqiad.wmnet with reason: host reimage 2434264 wikitext text/x-wiki gn51zr2ussdstvbu78z1n9vfrjxjzj6 2434265 2434264 2026-07-08T13:58:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet 2434265 wikitext text/x-wiki 7ttby03vouwcdzmurav6ffyn1ssgsp7 2434266 2434265 2026-07-08T13:59:43Z Stashbot 7414 klausman@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2434266 wikitext text/x-wiki jmibvbuxi4hbpnmnixyluy09did28tm 2434267 2434266 2026-07-08T14:00:45Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2434267 wikitext text/x-wiki mpm5e023em0hgpr69uaj31wr5oz4xd3 2434268 2434267 2026-07-08T14:00:46Z Stashbot 7414 klausman@cumin2002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: ml-serve-worker@eqiad 2434268 wikitext text/x-wiki bo9r1l8sdkryjrrw37sw5m9mvcsp5qz 2434269 2434268 2026-07-08T14:02:15Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test1001.eqiad.wmnet 2434269 wikitext text/x-wiki 8qrx845ck6yc0qrz9sh1p51k0pu3v0i 2434270 2434269 2026-07-08T14:05:33Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1118.eqiad.wmnet with OS trixie 2434270 wikitext text/x-wiki s2eom16i30wkjxu8yj42ujebr4gf3cs 2434271 2434270 2026-07-08T14:07:10Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet 2434271 wikitext text/x-wiki km40jis3x7t2nviuwk4vphgkdjew5jx 2434272 2434271 2026-07-08T14:08:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet 2434272 wikitext text/x-wiki 7jx9xqx0im6bm63w3hh5g068cvpbb2n 2434273 2434272 2026-07-08T14:10:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet 2434273 wikitext text/x-wiki 8t7y6ubddvr21w8toquwy4p53h2d96t 2434274 2434273 2026-07-08T14:13:08Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet 2434274 wikitext text/x-wiki cfb0bs6xdgspduyzbyk17zyao42tt6f 2434275 2434274 2026-07-08T14:14:21Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2434275 wikitext text/x-wiki hnkp0ugflsdy9ik1be86ut0mgpu4e6e 2434276 2434275 2026-07-08T14:14:40Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) 2434276 wikitext text/x-wiki 8dbaxlzwsssxvquxhpz9to3x5i022am 2434277 2434276 2026-07-08T14:14:48Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2228: Pool test 2434277 wikitext text/x-wiki keoni92zc4urwbtbrhwvfpg2zgfbzk9 2434278 2434277 2026-07-08T14:15:09Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2434278 wikitext text/x-wiki gjgthhrtlb58mddy670lygsr4rwt3ot 2434279 2434278 2026-07-08T14:15:38Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2434279 wikitext text/x-wiki na2bxgkda540hzp1qskqqbtjct06yz9 2434280 2434279 2026-07-08T14:16:08Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2434280 wikitext text/x-wiki 8c1rbs32pyw95g6jhq43gix52hxmot8 2434281 2434280 2026-07-08T14:16:19Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2434281 wikitext text/x-wiki jvemi8bkughjfyj2r4recwahr7dl94p 2434282 2434281 2026-07-08T14:16:47Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2434282 wikitext text/x-wiki pb99epykp180jfeb8innyfqe71eyk9w 2434283 2434282 2026-07-08T14:18:46Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet 2434283 wikitext text/x-wiki ihudlu20dekzvq0oj34kbqvmabdccx6 2434284 2434283 2026-07-08T14:19:13Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1002.eqiad.wmnet 2434284 wikitext text/x-wiki s0fuddql8hoou61p7ulf28ad89bxkip 2434285 2434284 2026-07-08T14:19:33Z Stashbot 7414 moritzm: installing librabbitmq security updates 2434285 wikitext text/x-wiki ekswontefnx7v8jj3ltfazp4b6oonjw 2434286 2434285 2026-07-08T14:19:55Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2434286 wikitext text/x-wiki jf8vkhjykhhc3lii79ulcrhj97xleu0 2434287 2434286 2026-07-08T14:20:23Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2434287 wikitext text/x-wiki f8ofk7u00cody2ydbpmc0h6mvfx62nr 2434288 2434287 2026-07-08T14:20:37Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2434288 wikitext text/x-wiki ly2lz396lv7uthfs9z6wyt6s45g8r8f 2434289 2434288 2026-07-08T14:20:44Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2434289 wikitext text/x-wiki map1hico7tcdh9xce78g10ycuhzycy2 2434290 2434289 2026-07-08T14:21:19Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2434290 wikitext text/x-wiki 6oii1bffl4ikyejqz5tpvnm79sr8f2j 2434291 2434290 2026-07-08T14:22:01Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2434291 wikitext text/x-wiki 93riiffsdwupe3oeadljmktc0uns13g 2434292 2434291 2026-07-08T14:22:03Z Stashbot 7414 atsuko@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1119.eqiad.wmnet with OS trixie 2434292 wikitext text/x-wiki dhbl2rt22o2alk6e13006juaeaeq871 2434293 2434292 2026-07-08T14:22:08Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2434293 wikitext text/x-wiki igvrfoyw7x94yeyhjzghfweogyumv5s 2434294 2434293 2026-07-08T14:22:44Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2434294 wikitext text/x-wiki 7ezre8t5o0iyrhjncikgrwooownvqfa 2434295 2434294 2026-07-08T14:27:28Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2434295 wikitext text/x-wiki di9jsg5prm92tsc0bmgud0y5nso4wv4 2434296 2434295 2026-07-08T14:29:00Z Stashbot 7414 moritzm: installing jackson-core security updates 2434296 wikitext text/x-wiki a1xh057ppxn2kedqeyp785c0ike59y0 2434297 2434296 2026-07-08T14:29:28Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2434297 wikitext text/x-wiki 3s4pit95chjeroofihunsx4174uh2bm 2434298 2434297 2026-07-08T14:30:01Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2434298 wikitext text/x-wiki k4hdd0z5hrpkwf9y46hvvzga66e4n0e 2434299 2434298 2026-07-08T14:30:07Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: Pool test 2434299 wikitext text/x-wiki 4samygien39ntroajj1ywt9y0lf15um 2434300 2434299 2026-07-08T14:30:30Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2434300 wikitext text/x-wiki p2nag1ay3lctp9hhka0m4yqav3v3ill 2434301 2434300 2026-07-08T14:30:38Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2434301 wikitext text/x-wiki fr2vnepqpdksy1bej5lfcp6dsh18i6g 2434302 2434301 2026-07-08T14:30:56Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434302 wikitext text/x-wiki 7pf4r4eamfodyfrcznvxwoymyc9dilv 2434303 2434302 2026-07-08T14:31:01Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2434303 wikitext text/x-wiki evbel884eicz48dl1qgv42m6ctyd8te 2434304 2434303 2026-07-08T14:31:06Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2434304 wikitext text/x-wiki abde28pdl97hbggmjppahhlldfnlj1y 2434305 2434304 2026-07-08T14:31:14Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434305 wikitext text/x-wiki 5hbq7pidica1c1bigatv0t4dmpjg7u1 2434306 2434305 2026-07-08T14:31:23Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2434306 wikitext text/x-wiki srwdi4hmd2exzuj8xxn2qk6c7ej7nro 2434307 2434306 2026-07-08T14:31:39Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1007.eqiad.wmnet with OS trixie 2434307 wikitext text/x-wiki gr845gfuiwhccwa6kd6obav7d7xegxz 2434308 2434307 2026-07-08T14:32:39Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet 2434308 wikitext text/x-wiki nlwnwltuwfvsn2sr1djpsq7dd55lsfw 2434309 2434308 2026-07-08T14:32:53Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1002.eqiad.wmnet 2434309 wikitext text/x-wiki 28mg873anb52zarkbykmp67h1vfgy8j 2434310 2434309 2026-07-08T14:34:16Z Stashbot 7414 swfrench-wmf: switched codfw, eqsin, ulsfo etcd client SRV records back to codfw - T430909 2434310 wikitext text/x-wiki nu46y0qeqs1h16yjqjxd0wx5clkpxst 2434311 2434310 2026-07-08T14:35:59Z Stashbot 7414 sukhe: restart pybal on lvs2014 to revert back to conf2004 2434311 wikitext text/x-wiki rot3oz69r833r0dehlufmuenzy3t5zr 2434313 2434311 2026-07-08T14:37:49Z Stashbot 7414 sukhe: restart pybal on lvs2013 to revert back to conf2004 2434313 wikitext text/x-wiki tifpu26l774odn2dfr82g596avqxu4j 2434314 2434313 2026-07-08T14:39:07Z Stashbot 7414 mvernon@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host thanos-be1007.eqiad.wmnet with OS trixie 2434314 wikitext text/x-wiki njrohc9azrctb0pwsf19kryn9t1vn0v 2434315 2434314 2026-07-08T14:39:45Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2434315 wikitext text/x-wiki ejiup7ahd85uqq02xehipszxw9nt8p6 2434316 2434315 2026-07-08T14:40:20Z Stashbot 7414 sukhe: sudo cumin -b1 -s120 "P{lvs2011*} or P{lvs2012*}" "systemctl restart pybal.service" 2434316 wikitext text/x-wiki fcnfdbtse70b58ycprm2lapiwpqc1eg 2434317 2434316 2026-07-08T14:41:02Z Stashbot 7414 moritzm: uninstalling dhcpcd-base from trixie hosts which still have it installed T414341 2434317 wikitext text/x-wiki 93dkgvppavwydjxi0kvvrd6bxyn5i0c 2434318 2434317 2026-07-08T14:43:42Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434318 wikitext text/x-wiki lpwinzo6m40ab3fh0rd4fx8l91j62w0 2434319 2434318 2026-07-08T14:44:04Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434319 wikitext text/x-wiki 820ekzkqsnm8c93rxzvnbbbcpkatta9 2434320 2434319 2026-07-08T14:46:48Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434320 wikitext text/x-wiki c5xoxw4g45hew11u0r67iyig6yx4sdx 2434321 2434320 2026-07-08T14:47:17Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434321 wikitext text/x-wiki kjv24ohsrlk3lihvqp62jv03055z28p 2434322 2434321 2026-07-08T14:49:39Z Stashbot 7414 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2434322 wikitext text/x-wiki 3djrm0jaortmpmzq5w9k4cq3qn0fhfg 2434323 2434322 2026-07-08T14:50:47Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434323 wikitext text/x-wiki mbkemtgr02nxzqv6gphhmtlrjliwomk 2434324 2434323 2026-07-08T14:51:16Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434324 wikitext text/x-wiki ov6lu65jc8safhfr0hc7zjdf3s3ilhu 2434325 2434324 2026-07-08T14:52:30Z Stashbot 7414 swfrench-wmf: restarted ulsfo confds, confirmed now connected to codfw backends except those using wikimedia.org SRV record - T430909 2434325 wikitext text/x-wiki by4koadgnqnv29fu6mdwu43tmih2a1g 2434326 2434325 2026-07-08T14:55:19Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1007.eqiad.wmnet with OS trixie 2434326 wikitext text/x-wiki o142qfbo93nbqu51aisv4l338l9pg27 2434327 2434326 2026-07-08T14:59:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2434327 wikitext text/x-wiki 2ayosl7lfi7a04tsqzslmmy8ge2g1i5 2434328 2434327 2026-07-08T14:59:44Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2434328 wikitext text/x-wiki dsjq0g0ryf53ukax8f1f07icjltex8o 2434330 2434328 2026-07-08T15:02:48Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test1002.eqiad.wmnet 2434330 wikitext text/x-wiki ga41o8bien6oa06x0xh328gihn1ukts 2434331 2434330 2026-07-08T15:03:02Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test1002.eqiad.wmnet 2434331 wikitext text/x-wiki ljzpl1n21g7isbqb21lnlcflcex328u 2434332 2434331 2026-07-08T15:03:07Z Stashbot 7414 swfrench-wmf: restarted eqsin, codfw confds - T430909 2434332 wikitext text/x-wiki svx6pr31zixf6cmlxpulgpsyjcoeu4h 2434333 2434332 2026-07-08T15:03:17Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T430909) 2434333 wikitext text/x-wiki 6rwi23whagxablq9caio0hkgpw3f2jt 2434334 2434333 2026-07-08T15:03:31Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2434334 wikitext text/x-wiki 17d1y98a3futfe9xkf3wdftpjqbgics 2434335 2434334 2026-07-08T15:03:33Z Stashbot 7414 swfrench-wmf: restarted navtiming on webperf2003 - T430909 2434335 wikitext text/x-wiki f5dklygtrapf4s8tvswln65pnj18lx8 2434336 2434335 2026-07-08T15:03:34Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2434336 wikitext text/x-wiki b3x6a8r6ral5j83hilrfqmwai7i1qu1 2434337 2434336 2026-07-08T15:03:43Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2434337 wikitext text/x-wiki pba5220e7u44idrp9iv1qjrxkpvmn05 2434338 2434337 2026-07-08T15:03:44Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2434338 wikitext text/x-wiki 1k4vgk5dl4gzebceuugefsakp1g3alg 2434339 2434338 2026-07-08T15:04:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2434339 wikitext text/x-wiki s34lvx3grmgc0k4daz8vxzhxaw23f1h 2434340 2434339 2026-07-08T15:04:15Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2434340 wikitext text/x-wiki kl49opijsjfa0el8id73hcmxrtqgy5j 2434341 2434340 2026-07-08T15:04:20Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2434341 wikitext text/x-wiki 5mfy8gqtdi3jmdvoqis9iijjdyjavpt 2434342 2434341 2026-07-08T15:05:06Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T430909) 2434342 wikitext text/x-wiki njgdkv4075kiqrklnvcby5kwcmw9o7m 2434343 2434342 2026-07-08T15:05:26Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet 2434343 wikitext text/x-wiki 0nw66cljwjmfh8g941rvb3w3y6mgczx 2434344 2434343 2026-07-08T15:06:27Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2434344 wikitext text/x-wiki b7a0tqpmo4g44gwjzd8c7p2yd3dd3wa 2434345 2434344 2026-07-08T15:06:29Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin (T430909) 2434345 wikitext text/x-wiki his2usa1mtxiboapp6wigfrnzkbdqcf 2434346 2434345 2026-07-08T15:06:37Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet 2434346 wikitext text/x-wiki qbpash4r82aici4egpvypeo4k7da153 2434347 2434346 2026-07-08T15:06:59Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2434347 wikitext text/x-wiki 5lqihhrgn3hpcqetieg7fz0hi6hwko0 2434348 2434347 2026-07-08T15:08:11Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2434348 wikitext text/x-wiki 6gzfnqszwy84pd1qymzgdmeiscbk65f 2434349 2434348 2026-07-08T15:08:12Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434349 wikitext text/x-wiki 089y433lhrwpf6ljxnkm4cd4bbr0b4g 2434350 2434349 2026-07-08T15:08:35Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434350 wikitext text/x-wiki sf0ri4dlrq0ss319an3ajlx18s4mf6r 2434351 2434350 2026-07-08T15:10:46Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin (T430909) 2434351 wikitext text/x-wiki azx1ozo0l819pj2u483ytstuy5zrgj5 2434352 2434351 2026-07-08T15:14:13Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2228: Depool test 2434352 wikitext text/x-wiki r5t31io59j049u2tiqg2i4016v4ni0b 2434353 2434352 2026-07-08T15:14:33Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: Depool test 2434353 wikitext text/x-wiki kfr6wk4bgd6r49tfx9wr3v6xhvx3uhd 2434354 2434353 2026-07-08T15:14:40Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[1002-1003].eqiad.wmnet 2434354 wikitext text/x-wiki keq98hpkvao70rwwzz37cvq10rc971u 2434355 2434354 2026-07-08T15:14:42Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2228: Pool test 2434355 wikitext text/x-wiki 3g6wt5hbatbvtyldjj2qcxraqr5yyxe 2434356 2434355 2026-07-08T15:15:02Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2434356 wikitext text/x-wiki n2r1uobs48gnnv8afc174usxkmr3ldq 2434357 2434356 2026-07-08T15:15:12Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1007.eqiad.wmnet with reason: host reimage 2434357 wikitext text/x-wiki puf4yv2nz6ze6jds8gz93g6ldzlj5yx 2434358 2434357 2026-07-08T15:15:22Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2434358 wikitext text/x-wiki 0rs1dq1jgqw60ig0hxwfosihhcn028h 2434359 2434358 2026-07-08T15:15:31Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2434359 wikitext text/x-wiki tqdyyfzq12ahj556vy22sjx9boueetz 2434360 2434359 2026-07-08T15:15:34Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1002-1003].eqiad.wmnet 2434360 wikitext text/x-wiki 6tho8uh0j6fxzzrk6uw7gjlg28iuzvu 2434361 2434360 2026-07-08T15:15:43Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2434361 wikitext text/x-wiki 2zz9e8xrhdkro6686tya5fa4p5sf4jg 2434362 2434361 2026-07-08T15:19:59Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1007.eqiad.wmnet with reason: host reimage 2434362 wikitext text/x-wiki 3rfm6v3pmrxmdefzpqbytrvq2oxmyu7 2434363 2434362 2026-07-08T15:23:02Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2434363 wikitext text/x-wiki ifgonjujw0pc5ntys42str9c1az43ic 2434364 2434363 2026-07-08T15:23:31Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2434364 wikitext text/x-wiki lwejjss7ez1sg27aa6kwauo9aryvuvs 2434365 2434364 2026-07-08T15:30:00Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: Pool test 2434365 wikitext text/x-wiki 5i6yk18p557j7zblbu0w19pnyrvtxcj 2434366 2434365 2026-07-08T15:36:28Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1007.eqiad.wmnet with OS trixie 2434366 wikitext text/x-wiki nrrg4vjfrqx5qcwdza0bd1dqg6gi6yu 2434367 2434366 2026-07-08T15:39:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2434367 wikitext text/x-wiki gsgolso7kbdjx4rxlttekfvhm80zcdf 2434368 2434367 2026-07-08T15:39:54Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2434368 wikitext text/x-wiki rqtfwtt1kyt25qdlkfi1epy1pcd23xo 2434369 2434368 2026-07-08T15:42:08Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1164.eqiad.wmnet 2434369 wikitext text/x-wiki g3tdto9qqpzqw5nce71rmufbk23ehrb 2434370 2434369 2026-07-08T15:42:09Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1164.eqiad.wmnet 2434370 wikitext text/x-wiki e6zzzawkmyu8atxerp501fzgkw557un 2434371 2434370 2026-07-08T15:42:12Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2434371 wikitext text/x-wiki qjw758s105zp2jos1rtgdsplbqsylo4 2434372 2434371 2026-07-08T15:42:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2434372 wikitext text/x-wiki r28m2t3wrklzfkl7ba9sepvelpv0yfy 2434373 2434372 2026-07-08T15:42:43Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1164.eqiad.wmnet 2434373 wikitext text/x-wiki 4xnroer8gl1xfjlq1wk8upv4cisjb65 2434374 2434373 2026-07-08T15:42:45Z Stashbot 7414 jasmine@cumin2002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1164.eqiad.wmnet 2434374 wikitext text/x-wiki 5hmicu637nwuz9hjghxpf607ujkg0w5 2434375 2434374 2026-07-08T15:48:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1095.eqiad.wmnet with OS trixie 2434375 wikitext text/x-wiki 09uu2bnost7iyv9x77xv8rcos98f1s9 2434376 2434375 2026-07-08T15:50:09Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1164.eqiad.wmnet 2434376 wikitext text/x-wiki 5g9eijiqsbrtplrpzyumr4ya2ksg2lr 2434377 2434376 2026-07-08T15:50:13Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1164.eqiad.wmnet 2434377 wikitext text/x-wiki czfahhydzhvclanx4bmkmla0tda6tz7 2434378 2434377 2026-07-08T15:50:16Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1164.eqiad.wmnet 2434378 wikitext text/x-wiki t9jrt761ub7421lp4e023vja63zuy5e 2434379 2434378 2026-07-08T15:50:42Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker1164.eqiad.wmnet with OS trixie 2434379 wikitext text/x-wiki 97xon8j3use9tb1jepkz42ja8tdotty 2434380 2434379 2026-07-08T15:51:17Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1164 2434380 wikitext text/x-wiki ohztyea2htlh10ax71z2ys9s95b9322 2434381 2434380 2026-07-08T15:51:49Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.netbox 2434381 wikitext text/x-wiki h6tmgythdtjovldazo76w52z4q0fwkd 2434382 2434381 2026-07-08T15:52:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1093.eqiad.wmnet with OS trixie 2434382 wikitext text/x-wiki 4r7qrcbjd8d1gdp4z6mgjznktmey69o 2434383 2434382 2026-07-08T15:56:48Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1164 - jasmine@cumin2002" 2434383 wikitext text/x-wiki 6wo9q7g2wq5bqhtful7m5qeg8tjy7ti 2434384 2434383 2026-07-08T15:56:55Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1164 - jasmine@cumin2002" 2434384 wikitext text/x-wiki eb5ihrdq7bssnlryjm18k155zsq4xol 2434385 2434384 2026-07-08T15:56:56Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434385 wikitext text/x-wiki ef51icjhwg66tmqn5y4e9erfaxtc40a 2434386 2434385 2026-07-08T15:56:58Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker1164.eqiad.wmnet 114.48.64.10.in-addr.arpa 4.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434386 wikitext text/x-wiki kgd3xshf27icbq5alsr1ulo5mikkdk1 2434387 2434386 2026-07-08T15:57:00Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1164.eqiad.wmnet 114.48.64.10.in-addr.arpa 4.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434387 wikitext text/x-wiki 4flr9z8863padicql75npe1bb6zck0y 2434388 2434387 2026-07-08T15:57:01Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1164 2434388 wikitext text/x-wiki gzgmrtpn0n820l8lvk88xd0zqh2f4p8 2434389 2434388 2026-07-08T15:57:29Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1164 2434389 wikitext text/x-wiki hh1x4ppxh7mosbmwx47et5ijw4lronc 2434390 2434389 2026-07-08T15:57:30Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1164 2434390 wikitext text/x-wiki mk0kuxqzd6gqtil9icd35hz7ae97tlr 2434391 2434390 2026-07-08T15:58:40Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Depool test 2434391 wikitext text/x-wiki 3jrtewtennaulj23zgi83rbgd9igks2 2434392 2434391 2026-07-08T15:58:44Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2434392 wikitext text/x-wiki jfw2nj541sv0s3lr97i9krq5e5c8vyn 2434393 2434392 2026-07-08T15:58:49Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2434393 wikitext text/x-wiki aclizmt6l30ewbjouobr662hvfryx2p 2434394 2434393 2026-07-08T15:58:50Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Depool test 2434394 wikitext text/x-wiki 8tjo0n2mp33wi9a64eum9s3vn2bincr 2434395 2434394 2026-07-08T15:59:00Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Pool test 2434395 wikitext text/x-wiki 4flz24225fpbqdnqsgj1vyerpoyjv2n 2434396 2434395 2026-07-08T15:59:01Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2434396 wikitext text/x-wiki ju9zf8jiwb78x3hpvzfxr3bnvhcqevd 2434397 2434396 2026-07-08T15:59:15Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2434397 wikitext text/x-wiki qqcxk8wkaa78pxatx4uawnkpbltgdln 2434398 2434397 2026-07-08T15:59:16Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Pool test 2434398 wikitext text/x-wiki lqlx85o9vtggr7p99uh0kif3ejch5o5 2434399 2434398 2026-07-08T16:08:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1093.eqiad.wmnet with reason: host reimage 2434399 wikitext text/x-wiki m7ap9lfeuooscwao9i5z4048ocon5dw 2434400 2434399 2026-07-08T16:09:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2434400 wikitext text/x-wiki 46eawaycz0wwi5f1q7db08k4nrqzg2u 2434401 2434400 2026-07-08T16:09:12Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet 2434401 wikitext text/x-wiki dzw3mpe4p18rh2ocqt0a9m4e7p9jyej 2434402 2434401 2026-07-08T16:15:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1095.eqiad.wmnet with reason: host reimage 2434402 wikitext text/x-wiki qrufnn4v62zrg1cxsen2h3rfojrt974 2434404 2434402 2026-07-08T16:16:30Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1164.eqiad.wmnet with reason: host reimage 2434404 wikitext text/x-wiki dfjmij92ecjv6oben39kt97ip6r4q69 2434405 2434404 2026-07-08T16:18:52Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cirrussearch1093.eqiad.wmnet with reason: host reimage 2434405 wikitext text/x-wiki nsj96orqofvyc2s5fs1i6auysenlzki 2434406 2434405 2026-07-08T16:23:37Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1164.eqiad.wmnet with reason: host reimage 2434406 wikitext text/x-wiki e75nf27emisfclcc2ptickx6wt0zp22 2434407 2434406 2026-07-08T16:26:26Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply 2434407 wikitext text/x-wiki e2nyj0qkjbqcg7544y9jw4yf55ntbsc 2434408 2434407 2026-07-08T16:26:51Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply} 2434408 wikitext text/x-wiki 6tv71yr0i1eh5v2pk1wjrf3lqrkw263 2434409 2434408 2026-07-08T16:27:57Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1095.eqiad.wmnet with reason: host reimage 2434409 wikitext text/x-wiki mb241102789mwabbmuv0oapm4p3fxbe 2434414 2434409 2026-07-08T16:43:10Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1164.eqiad.wmnet with OS trixie 2434414 wikitext text/x-wiki cqnvrqdt1dqgu0abxablbbyk0rjxd5n 2434415 2434414 2026-07-08T16:45:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1093.eqiad.wmnet with OS trixie 2434415 wikitext text/x-wiki fmhe5rf0tuz3agnl8svjr8rstwtg59t 2434416 2434415 2026-07-08T16:46:37Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1095.eqiad.wmnet with OS trixie 2434416 wikitext text/x-wiki agcxwohnnym6cm2x5s4870hve2qvltf 2434418 2434416 2026-07-08T16:49:42Z Stashbot 7414 kamila@deploy1003: Locking from deployment [MediaWiki]: switching deployment server 2434418 wikitext text/x-wiki pdmblz5yisz1tf1qin4qsdzi38zdwpo 2434420 2434418 2026-07-08T16:52:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie 2434420 wikitext text/x-wiki 6pyizhixkzrls5uv9kepgumaj1vqj22 2434423 2434420 2026-07-08T16:53:45Z Stashbot 7414 kamila@deploy1003: Unlocked for deployment [MediaWiki]: switching deployment server (duration: 04m 02s) 2434423 wikitext text/x-wiki l9lf2jo8qdvtu9imhxn7fihio87e9n3 2434424 2434423 2026-07-08T16:54:08Z Stashbot 7414 kamila@deploy1003: Locking from deployment [MediaWiki]: switching deployment server 2434424 wikitext text/x-wiki smzswt3rk27n67a7uxlypstzduc1ty8 2434427 2434424 2026-07-08T16:56:04Z Stashbot 7414 kamila@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet,releases1003.eqiad.wmnet with reason: Deployment server switchover 2434427 wikitext text/x-wiki 1okov8uezkdvon4b7oj10il8wbrnqqg 2434435 2434427 2026-07-08T17:04:19Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1164.eqiad.wmnet 2434435 wikitext text/x-wiki 2satzs56ni0ys7frq44lpyg6kgk7pdq 2434436 2434435 2026-07-08T17:04:22Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1164.eqiad.wmnet 2434436 wikitext text/x-wiki ax84vcepzx17yeksergjytjrwjbd1gx 2434437 2434436 2026-07-08T17:04:25Z Stashbot 7414 jasmine@cumin2002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1164.eqiad.wmnet 2434437 wikitext text/x-wiki hcinzrx3r34nbyem4rxhzzikvcrvdg2 2434442 2434437 2026-07-08T17:07:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage 2434442 wikitext text/x-wiki gpec997z4v2l7krd0qeyflq83yo9fdr 2434443 2434442 2026-07-08T17:09:29Z Stashbot 7414 kamila@dns1005: START - running authdns-update 2434443 wikitext text/x-wiki pkxcbm36m6xouvkaed7b94ywmoqcvw2 2434445 2434443 2026-07-08T17:11:19Z Stashbot 7414 kamila@dns1005: END - running authdns-update 2434445 wikitext text/x-wiki nzx42doaneghgdhu27wy5654lvhxd7t 2434446 2434445 2026-07-08T17:15:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage 2434446 wikitext text/x-wiki 23mdjamvnh7sz9pb0yft5p4xv4bb7ug 2434447 2434446 2026-07-08T17:16:15Z Stashbot 7414 kamila@deploy1003: Unlocked for deployment [MediaWiki]: switching deployment server (duration: 22m 07s) 2434447 wikitext text/x-wiki 3nd08dsl9ccvhaw7gztq4pkg47bjfzu 2434449 2434447 2026-07-08T17:18:12Z Stashbot 7414 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet 2434449 wikitext text/x-wiki t0bkr8i46bzc18ajychn6zduk8xegvk 2434456 2434449 2026-07-08T17:31:54Z Stashbot 7414 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet 2434456 wikitext text/x-wiki mk4dj3p0svtyd1tpkuk9an1tu52w3qw 2434457 2434456 2026-07-08T17:35:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1102.eqiad.wmnet with OS trixie 2434457 wikitext text/x-wiki 4vqg9z4jjfh9803xm7wro32ouov396t 2434460 2434457 2026-07-08T17:42:15Z Stashbot 7414 kamila@deploy2003: Rolling back deployment 2434460 wikitext text/x-wiki fqeqxvg1jz0fa3stcd8urroln0nr73m 2434462 2434460 2026-07-08T17:42:51Z Stashbot 7414 kamila@deploy2003: Finished scap sync-world: Test deployment to validate deployment server switchover - T423714 (duration: 19m 50s) 2434462 wikitext text/x-wiki sosdstw6yhcdsa8p3io1tnouvg4dt0d 2434463 2434462 2026-07-08T17:42:59Z Stashbot 7414 kamila@deploy2003: Started scap sync-world: Test deployment to validate deployment server switchover - T423714 2434463 wikitext text/x-wiki 189qf2vy4j2ixich8cbwj5vcerm6hn1 2434466 2434463 2026-07-08T17:46:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie 2434466 wikitext text/x-wiki gl69u187m1jjipmal5va7fw9bq16aww 2434468 2434466 2026-07-08T17:55:43Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cirrussearch1102.eqiad.wmnet with OS trixie 2434468 wikitext text/x-wiki 4mlmp7ydj9axf0x6djn91sxh6j4vfvj 2434470 2434468 2026-07-08T18:00:43Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1102.eqiad.wmnet with OS trixie 2434470 wikitext text/x-wiki 230wwquawj10n6yjv3ulwzs5lgf2yna 2434471 2434470 2026-07-08T18:01:28Z Stashbot 7414 kamila@deploy2003: Finished scap sync-world: Test deployment to validate deployment server switchover - T423714 (duration: 18m 29s) 2434471 wikitext text/x-wiki ei86j968ust3m0xlazv9fd06upput3y 2434472 2434471 2026-07-08T18:05:52Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1068.eqiad.wmnet with OS trixie 2434472 wikitext text/x-wiki 0ow94otd9d2c3wwhnuor9fw0cpzv7bp 2434479 2434472 2026-07-08T18:10:58Z Stashbot 7414 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2434479 wikitext text/x-wiki iln0ydk03vmgtdmmpi6mujmyz756sam 2434480 2434479 2026-07-08T18:13:48Z Stashbot 7414 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2434480 wikitext text/x-wiki d7ksb4j6eju7ikvgiimf3q4kl696ply 2434481 2434480 2026-07-08T18:13:56Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2434481 wikitext text/x-wiki p3csxylx3p11tnifzzbowb0i8tndi19 2434482 2434481 2026-07-08T18:15:03Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2434482 wikitext text/x-wiki i00exzjzotfpqxkg7zyxuoat5g33jqc 2434484 2434482 2026-07-08T18:15:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage 2434484 wikitext text/x-wiki ixqmppx5symkmjm9582vvu84asxu8al 2434485 2434484 2026-07-08T18:18:26Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1122.eqiad.wmnet with OS trixie 2434485 wikitext text/x-wiki n3vgcnlegtj7aotda52hjan5xfgn27t 2434486 2434485 2026-07-08T18:18:58Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1122 2434486 wikitext text/x-wiki 013mcuww227gs4zw58z8ywa6n56l9nh 2434487 2434486 2026-07-08T18:19:05Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2434487 wikitext text/x-wiki e9qa7oqpxq2oj306lfsgcaycx3pl75z 2434488 2434487 2026-07-08T18:19:23Z Stashbot 7414 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2434488 wikitext text/x-wiki 44hsmcgk1nqb0xb72w830i9e33779c5 2434489 2434488 2026-07-08T18:20:15Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1102.eqiad.wmnet with reason: host reimage 2434489 wikitext text/x-wiki eooakzxls8w6nk315pdlnw07gir6zlq 2434490 2434489 2026-07-08T18:21:05Z Stashbot 7414 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434490 wikitext text/x-wiki 5ocb0u1sy7bpjws0irmd5tye7xjorco 2434491 2434490 2026-07-08T18:21:11Z Stashbot 7414 rzl@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2434491 wikitext text/x-wiki cr2d68yc5erjgf38ydk292vuf9wdnb8 2434492 2434491 2026-07-08T18:21:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1068.eqiad.wmnet with reason: host reimage 2434492 wikitext text/x-wiki 89tvmz3m9eydbcpvexqvuqqw6pwwaun 2434493 2434492 2026-07-08T18:21:59Z Stashbot 7414 rzl@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2434493 wikitext text/x-wiki irn5oxs0phr4s44idao0z8h7r2rnliz 2434494 2434493 2026-07-08T18:25:33Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1122 - bking@cumin2003" 2434494 wikitext text/x-wiki ru51z59rbb3vkrtcpst8q4vb78lyzup 2434495 2434494 2026-07-08T18:25:36Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1122 - bking@cumin2003" 2434495 wikitext text/x-wiki q8qrj7l98g8o72903yyvxr3us306724 2434496 2434495 2026-07-08T18:25:37Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434496 wikitext text/x-wiki mrg2kxrwuu9z9u0cididndq3mkojc2c 2434497 2434496 2026-07-08T18:25:38Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache cirrussearch1122.eqiad.wmnet 31.48.64.10.in-addr.arpa 1.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434497 wikitext text/x-wiki h9jh88y6lv9kph9filorjnd8lz9e6xf 2434498 2434497 2026-07-08T18:25:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1122.eqiad.wmnet 31.48.64.10.in-addr.arpa 1.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434498 wikitext text/x-wiki fjc84ouood606z9z1gas2m0cylff290 2434499 2434498 2026-07-08T18:25:42Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1122 2434499 wikitext text/x-wiki 1zoxe8l97o2h2es0xbk6grxarsgwdfz 2434500 2434499 2026-07-08T18:26:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1122 2434500 wikitext text/x-wiki gssdzypf1jb0nwjoebnyo5brlxrcpuc 2434501 2434500 2026-07-08T18:26:05Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1122 2434501 wikitext text/x-wiki 7rrdayln72n3167uyb3teqzzrfi9z6s 2434502 2434501 2026-07-08T18:28:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1068.eqiad.wmnet with reason: host reimage 2434502 wikitext text/x-wiki tholorqk9ssvvmj816017899oelvio8 2434503 2434502 2026-07-08T18:38:38Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1122.eqiad.wmnet with reason: host reimage 2434503 wikitext text/x-wiki 23vmch6uo6osj9ir9gexj6v7wk0se7u 2434506 2434503 2026-07-08T18:44:30Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1122.eqiad.wmnet with reason: host reimage 2434506 wikitext text/x-wiki 7n41wq2w91cq5cep49lwxn3fa0e2y7s 2434509 2434506 2026-07-08T18:46:01Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2434509 wikitext text/x-wiki 9kpprx25wdob0bhmv0gqsar0e5e2bav 2434510 2434509 2026-07-08T18:47:19Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2434510 wikitext text/x-wiki phd0sdzv4ai2gnduzdor6l42es3qi0s 2434511 2434510 2026-07-08T18:47:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1102.eqiad.wmnet with OS trixie 2434511 wikitext text/x-wiki 5qqe44jv6a527subgksgv6cr9qs9iim 2434512 2434511 2026-07-08T18:48:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1068.eqiad.wmnet with OS trixie 2434512 wikitext text/x-wiki 1tv71e61zfklknxthq5mwt95x6xl9zq 2434514 2434512 2026-07-08T18:50:32Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1100.eqiad.wmnet with OS trixie 2434514 wikitext text/x-wiki 15zy2szlk01e6ket7oo7f45r3crx929 2434515 2434514 2026-07-08T18:52:17Z Stashbot 7414 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2434515 wikitext text/x-wiki acvyhcrurw74jfh1gr9vytknkwwjra4 2434516 2434515 2026-07-08T18:53:00Z Stashbot 7414 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2434516 wikitext text/x-wiki sl9r122o5l8cfgfwz6kdfk3lp3nkaip 2434517 2434516 2026-07-08T18:55:46Z Stashbot 7414 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'. 2434517 wikitext text/x-wiki 71nnk3hmbnlzoc0bnzosaicw9kkhm0q 2434518 2434517 2026-07-08T18:57:22Z Stashbot 7414 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2434518 wikitext text/x-wiki ligje9wha0jfhvhts4gbgil6wd5j40u 2434519 2434518 2026-07-08T18:58:44Z Stashbot 7414 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2434519 wikitext text/x-wiki i3j1winqaby7mrhbp346cnftsam7y8h 2434520 2434519 2026-07-08T18:59:11Z Stashbot 7414 topranks: rolling out update to BGP ACL on Nokia Switches eqiad, codfw & ulsfo T425703 2434520 wikitext text/x-wiki 9sveocxea5h1e8shbj6f7oo95ph56vd 2434521 2434520 2026-07-08T19:00:36Z Stashbot 7414 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2434521 wikitext text/x-wiki pmsdkmctx0zqjsgv91istxqta9hstbi 2434522 2434521 2026-07-08T19:03:41Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1122.eqiad.wmnet with OS trixie 2434522 wikitext text/x-wiki b4lrqko40p6gmy6w0u5spzyci005hv6 2434524 2434522 2026-07-08T19:05:55Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1100.eqiad.wmnet with reason: host reimage 2434524 wikitext text/x-wiki 2vv74eoj0xe2vwwzfyg127gp8mygcze 2434526 2434524 2026-07-08T19:08:14Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-b3-magru 2434526 wikitext text/x-wiki t59ccstqahiyveb6rhbslf56adun4xf 2434527 2434526 2026-07-08T19:08:36Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b3-magru 2434527 wikitext text/x-wiki dl04adfjt7f63xy03rw3h5iew8j9yew 2434528 2434527 2026-07-08T19:09:09Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device asw1-b4-magru 2434528 wikitext text/x-wiki hf1hbsp86rg17xt3i3t0hay3cbrkei9 2434529 2434528 2026-07-08T19:09:31Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b4-magru 2434529 wikitext text/x-wiki i6r0ka8j8bi8hvrhy51skdf7ev45khz 2434530 2434529 2026-07-08T19:09:34Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c3-codfw 2434530 wikitext text/x-wiki ord4kmr1xgpgca1ssrsbyp0bu1u0gm5 2434531 2434530 2026-07-08T19:09:44Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c3-codfw 2434531 wikitext text/x-wiki 5cak0qcbwoyh09w68s3sdx41hh5e9b0 2434532 2434531 2026-07-08T19:09:47Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c6-codfw 2434532 wikitext text/x-wiki comyx65hghgylaltr8z88wuyo6pv7nu 2434533 2434532 2026-07-08T19:09:57Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c6-codfw 2434533 wikitext text/x-wiki 4i4zmatyj8y3al3tapjjhv7lxpdss5m 2434534 2434533 2026-07-08T19:10:00Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d6-codfw 2434534 wikitext text/x-wiki p43d1bbwwk1ukymjjetazrg84sbfbxy 2434535 2434534 2026-07-08T19:10:03Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1100.eqiad.wmnet with reason: host reimage 2434535 wikitext text/x-wiki aflj9pqs9rwzpkyeb7o57fvpa08ucv5 2434536 2434535 2026-07-08T19:10:10Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-codfw 2434536 wikitext text/x-wiki c1kz59i2jssrwwxpwpc976zkrij9ed5 2434537 2434536 2026-07-08T19:10:13Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d8-codfw 2434537 wikitext text/x-wiki icdtpwkzj811689x7jspq4lqm5ght6t 2434538 2434537 2026-07-08T19:10:23Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d8-codfw 2434538 wikitext text/x-wiki 53csvhw30pjq72qs8lhgrcr02w6ss8q 2434539 2434538 2026-07-08T19:10:26Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device cr1-magru 2434539 wikitext text/x-wiki ivjt5hyara58n3a9qea2o9p7ygosdv0 2434540 2434539 2026-07-08T19:10:47Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-magru 2434540 wikitext text/x-wiki 29atd948vpbzprrmy1turxkehlbddou 2434541 2434540 2026-07-08T19:10:50Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device ssw1-d1-codfw 2434541 wikitext text/x-wiki bu71kfttra5hvf62o68mjk5elbbqavi 2434542 2434541 2026-07-08T19:11:00Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d1-codfw 2434542 wikitext text/x-wiki sqpnntv0uixc7f9146a141khlaei0qo 2434544 2434542 2026-07-08T19:18:44Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c2-codfw 2434544 wikitext text/x-wiki sx03wfw718r879spfvxkch3d9u6txys 2434546 2434544 2026-07-08T19:18:56Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c2-codfw 2434546 wikitext text/x-wiki tl684wbo9rlq2i829d3znfzqkr1enmo 2434547 2434546 2026-07-08T19:18:57Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d1-codfw 2434547 wikitext text/x-wiki 05xnccn6839xg94sr3bs5tjqgcyobwd 2434548 2434547 2026-07-08T19:19:07Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d1-codfw 2434548 wikitext text/x-wiki o31m7vryycc1wkhquzbjczttow7tns1 2434549 2434548 2026-07-08T19:19:09Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d3-codfw 2434549 wikitext text/x-wiki tb78o4sb4wvujzqmnjo5v7n9gkffe3f 2434550 2434549 2026-07-08T19:19:20Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d3-codfw 2434550 wikitext text/x-wiki nt8bbdjkdz9leprkaj6pv9ibp5rd4ut 2434551 2434550 2026-07-08T19:19:22Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d7-codfw 2434551 wikitext text/x-wiki du6yhj7in0uxgjt1pbtipobhzsmljni 2434552 2434551 2026-07-08T19:19:33Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d7-codfw 2434552 wikitext text/x-wiki m919bhql52rf70bgixk49pv5ly7whug 2434553 2434552 2026-07-08T19:19:35Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device cr2-magru 2434553 wikitext text/x-wiki lovtzfb77dnv44bwow86hp0x5pcj0h6 2434556 2434553 2026-07-08T19:19:56Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-magru 2434556 wikitext text/x-wiki rdouj2vj9tequr57gnnapo0odn1vp94 2434557 2434556 2026-07-08T19:19:57Z Stashbot 7414 mutante: gerrit - replacing private key for registerEmail verification 2434557 wikitext text/x-wiki rxqxf3tzxq1hcr7jgcke16p74wbi08v 2434558 2434557 2026-07-08T19:19:59Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device ssw1-d8-codfw 2434558 wikitext text/x-wiki kwjlaiijedx56nbx6gsdotun70il6se 2434559 2434558 2026-07-08T19:20:09Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d8-codfw 2434559 wikitext text/x-wiki qm42cocg0muw0w6j6zuh73u9acndx1x 2434568 2434559 2026-07-08T19:29:04Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1100.eqiad.wmnet with OS trixie 2434568 wikitext text/x-wiki ptt2vj0u7n3m1yw4b3yfe05pndutzvc 2434569 2434568 2026-07-08T19:30:24Z Stashbot 7414 jasmine_: ran homer on lsw1-d8-codfw, adding wikikube-worker2331 to cluster 2434569 wikitext text/x-wiki cf423930pq73wecqc8ztcb3v3wo2l54 2434570 2434569 2026-07-08T19:38:11Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c5-codfw 2434570 wikitext text/x-wiki 98961qe45f50ncnpo0w5co5xwodim5y 2434571 2434570 2026-07-08T19:38:26Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c5-codfw 2434571 wikitext text/x-wiki tneold69a1t2d1ywqmqud6hukae6q0u 2434572 2434571 2026-07-08T19:38:29Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c7-codfw 2434572 wikitext text/x-wiki je6wb58fht4ezuumgge3g418x0xn4em 2434573 2434572 2026-07-08T19:38:39Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c7-codfw 2434573 wikitext text/x-wiki j4elcjw37r6nf9hiqw770ltgq6z3v3p 2434574 2434573 2026-07-08T19:38:42Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d5-codfw 2434574 wikitext text/x-wiki acdadzrytrr28ln657x58ot30kc9iub 2434575 2434574 2026-07-08T19:38:52Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d5-codfw 2434575 wikitext text/x-wiki 6s030q9wza28595de2ibxy5eome41zy 2434576 2434575 2026-07-08T19:40:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1098.eqiad.wmnet with OS trixie 2434576 wikitext text/x-wiki ddgvsk50ya4mr8xnpn4r5w0r2hmg8vz 2434577 2434576 2026-07-08T19:43:24Z Stashbot 7414 jasmine@cumin2002: conftool action : set/weight=10; selector: name=wikikube-worker2331.codfw.wmnet,cluster=kubernetes,service=kubesvc 2434577 wikitext text/x-wiki s01pc8ygyguuqz2qsn4bcc32opvtpcr 2434578 2434577 2026-07-08T19:43:56Z Stashbot 7414 jasmine@cumin2002: conftool action : set/pooled=yes; selector: name=wikikube-worker2331.codfw.wmnet,cluster=kubernetes,service=kubesvc 2434578 wikitext text/x-wiki 8971dmztkb961uvjnbaezgnfsjln1hh 2434579 2434578 2026-07-08T19:46:29Z Stashbot 7414 mutante: restarting gerrit on gerrit-spare.wikimedia.org (gerrit2002) 2434579 wikitext text/x-wiki ebbcgw5qnruspy1jbda3ki8gar90fcr 2434580 2434579 2026-07-08T19:48:23Z Stashbot 7414 mutante: restarting gerrit on gerrit-replica.wikimedia.org (gerrit1003) 2434580 wikitext text/x-wiki 1j30x2tr342weh39ra8hcidq4gxoyfm 2434581 2434580 2026-07-08T19:48:52Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2331.codfw.wmnet 2434581 wikitext text/x-wiki 24z36kvfacw63ofdg0l6vnwzvw3yjj5 2434583 2434581 2026-07-08T19:48:55Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2331.codfw.wmnet 2434583 wikitext text/x-wiki q2nz4b0gya5osxx817nlolgl945og09 2434585 2434583 2026-07-08T19:52:58Z Stashbot 7414 mutante: restarting gerrit on gerrit.wikimedia.org (gerrit2003) 2434585 wikitext text/x-wiki ezv9u298dxfeatzhdrd9wjkvsbnj3jm 2434586 2434585 2026-07-08T19:54:37Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c1-codfw 2434586 wikitext text/x-wiki jlyt86izljgrgr58kv47p4o4dr64eys 2434587 2434586 2026-07-08T19:54:47Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-codfw 2434587 wikitext text/x-wiki 8ophgt9tbv4w3pxx0jchrx11tzbzdnb 2434588 2434587 2026-07-08T19:54:50Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d4-codfw 2434588 wikitext text/x-wiki c0cniqbnlkcjo5kogfh2rqyg79nykw6 2434589 2434588 2026-07-08T19:55:00Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d4-codfw 2434589 wikitext text/x-wiki 0v35ln016dktul35apqqzy0r5gm7b8s 2434590 2434589 2026-07-08T19:56:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1098.eqiad.wmnet with reason: host reimage 2434590 wikitext text/x-wiki k4ud3gxrd2elscevw2oy73kf4n0pnwo 2434591 2434590 2026-07-08T20:00:20Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1098.eqiad.wmnet with reason: host reimage 2434591 wikitext text/x-wiki 91nz62ikgp291ob93f1rmdanlqc703m 2434593 2434591 2026-07-08T20:09:55Z Stashbot 7414 cjming@deploy2003: Started scap sync-world: Backport for [[gerrit:1308183|Move Test Kitchen config from CommonSettings.php (T431257)]] 2434593 wikitext text/x-wiki q5b5f6frlotyhzbyrug7aavilvsq4d7 2434594 2434593 2026-07-08T20:13:03Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1073.eqiad.wmnet with OS trixie 2434594 wikitext text/x-wiki d6vb6qckzh5ofkecynu7sl9ghpdcasz 2434596 2434594 2026-07-08T20:20:58Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1098.eqiad.wmnet with OS trixie 2434596 wikitext text/x-wiki 6nt4pupmzzjc348ncx4yscuna6d421c 2434597 2434596 2026-07-08T20:28:44Z Stashbot 7414 cjming@deploy2003: cjming: Backport for [[gerrit:1308183|Move Test Kitchen config from CommonSettings.php (T431257)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2434597 wikitext text/x-wiki lqq58hyijxls2lcomsch9ue48igqpal 2434599 2434597 2026-07-08T20:30:09Z Stashbot 7414 cjming@deploy2003: cjming: Continuing with deployment 2434599 wikitext text/x-wiki 3oxmn07yto89ker5qx3l8ye9m50l64b 2434600 2434599 2026-07-08T20:32:26Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1073.eqiad.wmnet with reason: host reimage 2434600 wikitext text/x-wiki n67jd6q17xnmh6xgjpxwvm2aojwe3dj 2434601 2434600 2026-07-08T20:37:59Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1073.eqiad.wmnet with reason: host reimage 2434601 wikitext text/x-wiki tbsfqjvevstri3gjetkigoh2xk8tb6y 2434602 2434601 2026-07-08T20:42:04Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host restbase2039.codfw.wmnet with OS bullseye 2434602 wikitext text/x-wiki izbr8ta8p0l0a48y1h5l2dhwx62e7jh 2434603 2434602 2026-07-08T20:42:57Z Stashbot 7414 cjming@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308183|Move Test Kitchen config from CommonSettings.php (T431257)]] (duration: 33m 02s) 2434603 wikitext text/x-wiki hin64jm8633sj5nb9rgkhvuu3h3cvm6 2434604 2434603 2026-07-08T20:48:26Z Stashbot 7414 mutante: deploy2003 - kill 1102 (stunnel4) ; systemctl start stunnel4 (T418262) 2434604 wikitext text/x-wiki gdoketkzd5sh9yvo30jmzr04zpqrcr7 2434608 2434604 2026-07-08T20:58:52Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1073.eqiad.wmnet with OS trixie 2434608 wikitext text/x-wiki ohp1woh80yh07pbnhssu78rnudlz9gl 2434611 2434608 2026-07-08T21:00:01Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase2039.codfw.wmnet with reason: host reimage 2434611 wikitext text/x-wiki hrxo2jypkz62p9kdmmuq0yr1q3mx15g 2434614 2434611 2026-07-08T21:04:13Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase2039.codfw.wmnet with reason: host reimage 2434614 wikitext text/x-wiki c8n2qszk7neekfhi2kkdpvkcq5ino7j 2434615 2434614 2026-07-08T21:21:07Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2434615 wikitext text/x-wiki qemiorc1tfmc04jl327n1y21hdbrpoi 2434616 2434615 2026-07-08T21:21:28Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2434616 wikitext text/x-wiki gl69mqix295txtbgfjbgltwu9924kfp 2434617 2434616 2026-07-08T21:21:29Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2039.codfw.wmnet with OS bullseye 2434617 wikitext text/x-wiki axdtdtaoy7q3vcx4vp0e8yijolp48e0 2434618 2434617 2026-07-08T21:22:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch1094.eqiad.wmnet with OS trixie 2434618 wikitext text/x-wiki dtvg0ky6tb98l04kcwgsbvd76mg7o1n 2434619 2434618 2026-07-08T21:27:36Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2434619 wikitext text/x-wiki bytkgw27v2nkfl0mzpsc3d39v7sauen 2434620 2434619 2026-07-08T21:29:21Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2434620 wikitext text/x-wiki 7xix0s01jpey8ee93dl2eqxl9ubmwo2 2434622 2434620 2026-07-08T21:35:54Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply 2434622 wikitext text/x-wiki q95u229v4zxo6xgofxtkvjo8jwqrzpn 2434623 2434622 2026-07-08T21:36:28Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply} 2434623 wikitext text/x-wiki b44y3d7gfjehb2pea29iwynln9823cu 2434624 2434623 2026-07-08T21:37:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1094.eqiad.wmnet with reason: host reimage 2434624 wikitext text/x-wiki jsy54rjbiu1g6s6ynhkj9qa9k0e9qsw 2434625 2434624 2026-07-08T21:39:26Z Stashbot 7414 rzl@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2434625 wikitext text/x-wiki h09gu3s4ssk95comkcmafw7i15nwl3h 2434627 2434625 2026-07-08T21:42:07Z Stashbot 7414 rzl@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2434627 wikitext text/x-wiki ew82eq7ismc7t08xfmkndg13qblpxmw 2434628 2434627 2026-07-08T21:44:11Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1094.eqiad.wmnet with reason: host reimage 2434628 wikitext text/x-wiki 4vluckp28284wjbgjm0ax06os65yo2b 2434629 2434628 2026-07-08T21:49:21Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2002.codfw.wmnet with OS trixie 2434629 wikitext text/x-wiki o4qrtrjjkervzjx7ldfis7rbkk5503f 2434630 2434629 2026-07-08T21:49:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2002 2434630 wikitext text/x-wiki msdxqx00muosrn4o6roey577gkru59k 2434631 2434630 2026-07-08T21:50:00Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2434631 wikitext text/x-wiki llu2ps3xx4igkgf18o32turuf7egwkw 2434632 2434631 2026-07-08T21:54:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2002 - bking@cumin2003" 2434632 wikitext text/x-wiki tmjo98i0e3eqzh9i7tveqap7ulwd641 2434633 2434632 2026-07-08T21:54:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2002 - bking@cumin2003" 2434633 wikitext text/x-wiki iyy5nrccv09ygd12svig6zu0eujohgp 2434634 2434633 2026-07-08T21:54:08Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434634 wikitext text/x-wiki 6flg73381kx32bj05lcuios7i4f26j2 2434635 2434634 2026-07-08T21:54:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2002.codfw.wmnet 50.32.192.10.in-addr.arpa 0.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434635 wikitext text/x-wiki mupbe50zq21jqgh425rmokn6bxz6xxw 2434636 2434635 2026-07-08T21:54:11Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2002.codfw.wmnet 50.32.192.10.in-addr.arpa 0.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434636 wikitext text/x-wiki qfoglyp58akqdxof4mm4o615xo35n0q 2434637 2434636 2026-07-08T21:54:12Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2002 2434637 wikitext text/x-wiki 74o7xnwtn1wr5f9uu4p4dhcv9oh7rpz 2434638 2434637 2026-07-08T21:54:29Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2002 2434638 wikitext text/x-wiki hfdlo25ajpsqums4sf8ignghflkzsei 2434639 2434638 2026-07-08T21:54:30Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2002 2434639 wikitext text/x-wiki pja1gwow4xar833ozzs3ie32qfcdxnj 2434640 2434639 2026-07-08T21:55:20Z Stashbot 7414 urbanecm@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply 2434640 wikitext text/x-wiki p5av4kdi8xtwf8zzw24b1ws7iypuj9m 2434641 2434640 2026-07-08T21:55:23Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-c4-codfw 2434641 wikitext text/x-wiki tdegmcf2jnhxlm9gd1k03enx689w30m 2434642 2434641 2026-07-08T21:55:33Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c4-codfw 2434642 wikitext text/x-wiki cy2wd34flotocw5z72q53voaho4yls4 2434643 2434642 2026-07-08T21:55:55Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lsw1-d2-codfw 2434643 wikitext text/x-wiki iex0ar7tyz329kepk5c93x4ny4a8ehr 2434644 2434643 2026-07-08T21:55:59Z Stashbot 7414 urbanecm@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply 2434644 wikitext text/x-wiki c6u33cgu4n753g1iojxvheyif8qtcac 2434645 2434644 2026-07-08T21:56:04Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d2-codfw 2434645 wikitext text/x-wiki ex9ca93mrwc5lxkuhhg2w2225dpnkzr 2434646 2434645 2026-07-08T22:01:11Z Stashbot 7414 urbanecm: Make https://test.wikipedia.org/w/index.php?title=MediaWiki:GrowthExperimentsSuggestedEdits.json&diff=prev&oldid=750552 with GrowthExperiments disabled (via mw-experimental), then run `\MediaWiki\MediaWikiServices::getInstance()->get('CommunityConfiguration.ProviderFactory')->newProvider('GrowthSuggestedEdits')->getStore()->invalidate()` (T431625) 2434646 wikitext text/x-wiki i2yz8fvuj6v4rgyinjz3irleqt23lg8 2434647 2434646 2026-07-08T22:05:25Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1094.eqiad.wmnet with OS trixie 2434647 wikitext text/x-wiki 9ie2pdz4qs70xuhl39cs8f5csr0gbwh 2434648 2434647 2026-07-08T22:06:16Z Stashbot 7414 rzl@deploy2003: helmfile [codfw] START helmfile.d/admin 'apply'. 2434648 wikitext text/x-wiki fbrd9vzq7mive0v523yo5i2wc2xrath 2434649 2434648 2026-07-08T22:09:07Z Stashbot 7414 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2434649 wikitext text/x-wiki gd2aed8wx8fcexx2uo8zjoor566ms8x 2434650 2434649 2026-07-08T22:13:42Z Stashbot 7414 rzl@deploy2003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2434650 wikitext text/x-wiki avyoqxa8h9wx0e2np8su66vb4ey5t0x 2434652 2434650 2026-07-08T22:13:49Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply 2434652 wikitext text/x-wiki k13kjn5qhoc70w44mxpxzpvlxhzqnsj 2434653 2434652 2026-07-08T22:13:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage 2434653 wikitext text/x-wiki b372cthob8o9i74nendsi3ih6l5xd7u 2434654 2434653 2026-07-08T22:14:27Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: apply} 2434654 wikitext text/x-wiki pt4quz5x52c2261jcele8ys0b9jsdsq 2434656 2434654 2026-07-08T22:17:10Z Stashbot 7414 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2434656 wikitext text/x-wiki o66z6pa0w7b1iz7xs031p0f2n36qq3g 2434657 2434656 2026-07-08T22:19:50Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2002.codfw.wmnet with reason: host reimage 2434657 wikitext text/x-wiki cwnnz76447j9k1dnz8q639qkoeez3fc 2434658 2434657 2026-07-08T22:19:51Z Stashbot 7414 rzl@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2434658 wikitext text/x-wiki 7aivrh52wy4ajd97yhegknffdus2r3m 2434659 2434658 2026-07-08T22:21:16Z Stashbot 7414 rzl@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2434659 wikitext text/x-wiki nlbv6lhus4h2etmbblc19iyq70n054q 2434660 2434659 2026-07-08T22:22:34Z Stashbot 7414 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2434660 wikitext text/x-wiki mmjmflwwuig94xijz37el7952vt61pp 2434661 2434660 2026-07-08T22:26:38Z Stashbot 7414 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434661 wikitext text/x-wiki a35vecp42xmjb9sm5s9pir46wj5vuwe 2434662 2434661 2026-07-08T22:27:48Z Stashbot 7414 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1308782|fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781|fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]] 2434662 wikitext text/x-wiki fd65arl0xft1satdti2m76v42y9u44m 2434663 2434662 2026-07-08T22:29:09Z Stashbot 7414 rzl@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2434663 wikitext text/x-wiki niu0usmcmwb9higi2dpj1fedgem91h3 2434664 2434663 2026-07-08T22:30:10Z Stashbot 7414 rzl@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2434664 wikitext text/x-wiki s1iuqrfbszj8h38n5ck2cc9de9odmcw 2434665 2434664 2026-07-08T22:30:47Z Stashbot 7414 rzl@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2434665 wikitext text/x-wiki dt9i4wpaisjzp7zu8x9wa1cvq0l4glw 2434666 2434665 2026-07-08T22:32:33Z Stashbot 7414 rzl@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2434666 wikitext text/x-wiki arjxf57pydvaz0t3g26saob6rep9m2s 2434667 2434666 2026-07-08T22:33:28Z Stashbot 7414 rzl@deploy2003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2434667 wikitext text/x-wiki ob2tlpg0m7ga0k7wghdejef0ltiltz3 2434668 2434667 2026-07-08T22:33:34Z Stashbot 7414 urbanecm@deploy2003: urbanecm: Backport for [[gerrit:1308782|fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781|fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2434668 wikitext text/x-wiki 35go7pm7fu312p8c3peah5nn3kav1f2 2434670 2434668 2026-07-08T22:34:07Z Stashbot 7414 urbanecm@deploy2003: urbanecm: Continuing with deployment 2434670 wikitext text/x-wiki 5jh1nit1f25kuhj9avnj3ajf3ezj1z2 2434671 2434670 2026-07-08T22:35:34Z Stashbot 7414 rzl@deploy2003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2434671 wikitext text/x-wiki s1mots4jn2zubzfpmx7kyr8c5d9674g 2434672 2434671 2026-07-08T22:36:15Z Stashbot 7414 rzl@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2434672 wikitext text/x-wiki k89k4zbuiz8p76itryiw6q2uytpjhkf 2434673 2434672 2026-07-08T22:37:47Z Stashbot 7414 rzl@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2434673 wikitext text/x-wiki aipsv7az60d2vex4ph9vec2k22u9817 2434674 2434673 2026-07-08T22:40:14Z Stashbot 7414 rzl@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2434674 wikitext text/x-wiki 0rv1ne1y4pn0y8ac831ew9q6aant2h8 2434675 2434674 2026-07-08T22:40:43Z Stashbot 7414 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308782|fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]], [[gerrit:1308781|fix: Return consistent tuple from filterLimitReachedAddLink (T431625)]] (duration: 12m 55s) 2434675 wikitext text/x-wiki 5pit0eh0w4z0x273mchqsatg4s7jqiy 2434676 2434675 2026-07-08T22:42:05Z Stashbot 7414 rzl@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2434676 wikitext text/x-wiki szuwxef88rmb4ihpmnk1f0cyyclg10d 2434678 2434676 2026-07-08T23:15:52Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2434678 wikitext text/x-wiki 30yz8h8w3zpjnwdhfy4uczcq3fwjw27 2434679 2434678 2026-07-08T23:16:33Z Stashbot 7414 rzl@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2434679 wikitext text/x-wiki 0zr5rtburjsdp8q4c6bimjs0wjy5m2a 2434680 2434679 2026-07-08T23:40:14Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2434680 wikitext text/x-wiki bvi49w1dhlwxauv0vlbuo3fcthvonw0 2434681 2434680 2026-07-08T23:46:37Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add missing PTR for 2001:df2:e500:fe08::1 - cmooney@cumin1003" 2434681 wikitext text/x-wiki luwdc14oe9f70aa35346391chi2o9iv 2434682 2434681 2026-07-08T23:46:44Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add missing PTR for 2001:df2:e500:fe08::1 - cmooney@cumin1003" 2434682 wikitext text/x-wiki rotlc17h1fcxmrl9g1r2vy81fqcp4hi 2434683 2434682 2026-07-08T23:46:45Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434683 wikitext text/x-wiki 0in9fz5umj9ape80f0cfz4wlhs2i8ol 2434685 2434683 2026-07-08T23:52:49Z Stashbot 7414 Amir1: ladsgroup@deploy2003:~$ mwscript-k8s --follow -- extensions/ORES/maintenance/PurgeScoreCache.php --wiki=simplewiki --model damaging --old (T431159) 2434685 wikitext text/x-wiki dib7t2bafu65q1vx1da1ypomdysuj3m 2434690 2434685 2026-07-09T02:00:54Z Stashbot 7414 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image 2434690 wikitext text/x-wiki rypkp2u9f3qxx2psx1ehwn2hdrt67ol 2434691 2434690 2026-07-09T02:07:26Z Stashbot 7414 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 31s) 2434691 wikitext text/x-wiki p631mio6nwoi3ufqe9vawwiutcd1b4e 2434707 2434691 2026-07-09T03:27:29Z Stashbot 7414 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad 2434707 wikitext text/x-wiki a9imuzwknrlbwnixkw9vthze5td85d2 2434708 2434707 2026-07-09T03:27:30Z Stashbot 7414 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=eqiad 2434708 wikitext text/x-wiki rp13ol0q8u2mpfak78wd27tx11i6v21 2434709 2434708 2026-07-09T03:27:31Z Stashbot 7414 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=eqiad 2434709 wikitext text/x-wiki 9h977d6c0d5nygrmnpjbt9jvs07z8n3 2434710 2434709 2026-07-09T03:29:23Z Stashbot 7414 ryankemper: T431311 Repooled eqiad cirrussearch clusters (`chi/omega/psi`) following completion of OpenSearch 2.19 migration 2434710 wikitext text/x-wiki bq4mshzekb1oyn7jd7pxner49e4f69e 2434715 2434710 2026-07-09T03:55:35Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6008.* 2434715 wikitext text/x-wiki 4gq4fmyw09566ojggt0cpl849kziwc1 2434717 2434715 2026-07-09T04:10:01Z Stashbot 7414 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cp6008.drmrs.wmnet with reason: Hardware failure - T431651 2434717 wikitext text/x-wiki cab7416b3yfio623i3xsc2r5hav6rbx 2434719 2434717 2026-07-09T06:47:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1008.eqiad.wmnet with OS trixie 2434719 wikitext text/x-wiki hxkqqdj1gb7j53k4w1azyk81zfc5cow 2434720 2434719 2026-07-09T06:57:01Z Stashbot 7414 Emperor: rebalance thanos swift rings after previous re-image of thanos-fe1004 to trixie 2434720 wikitext text/x-wiki i7ntw2rdz8wjto547lidab7xi60qmvc 2434721 2434720 2026-07-09T06:59:45Z Stashbot 7414 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be1008.eqiad.wmnet with OS trixie 2434721 wikitext text/x-wiki p8i51bjrm81de3zxnsxr3z4e1asd6ep 2434722 2434721 2026-07-09T07:00:14Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2434722 wikitext text/x-wiki fgnceniqolby8uj9usqfp7pvcz6ja73 2434723 2434722 2026-07-09T07:10:01Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2434723 wikitext text/x-wiki gz366d2lxfthvmv67npets4801b0w69 2434724 2434723 2026-07-09T07:12:04Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be1008.eqiad.wmnet with OS trixie 2434724 wikitext text/x-wiki 7820pos0sdpj3wors3rgfq6dny572cz 2434725 2434724 2026-07-09T07:31:32Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1008.eqiad.wmnet with reason: host reimage 2434725 wikitext text/x-wiki r96a0t7bwpyeqw18molcqqn4p6blh7b 2434726 2434725 2026-07-09T07:35:29Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1008.eqiad.wmnet with reason: host reimage 2434726 wikitext text/x-wiki 8iv886v2ggp5eq5lzlxzsxbr2f0853p 2434727 2434726 2026-07-09T07:41:05Z Stashbot 7414 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1308687|Enable sub-references on more group2 wikis (batch2) (T430941)]] 2434727 wikitext text/x-wiki cy1rn33uudww9wqnr61qfcgfyoas050 2434728 2434727 2026-07-09T07:43:23Z Stashbot 7414 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1308687|Enable sub-references on more group2 wikis (batch2) (T430941)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2434728 wikitext text/x-wiki 74m6eqdtwxbp3exoin6ladey4m8w29c 2434729 2434728 2026-07-09T07:44:50Z Stashbot 7414 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment 2434729 wikitext text/x-wiki 8sv4v092vvy4pwlybh5shehgcs7lh46 2434730 2434729 2026-07-09T07:49:41Z Stashbot 7414 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1308687|Enable sub-references on more group2 wikis (batch2) (T430941)]] (duration: 08m 36s) 2434730 wikitext text/x-wiki qksvap6nz5kndbundapp1d8lz216368 2434731 2434730 2026-07-09T07:54:06Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1008.eqiad.wmnet with OS trixie 2434731 wikitext text/x-wiki 0b7wtv6bzfpibqu6gpvz4wgikmxr3z7 2434732 2434731 2026-07-09T07:56:17Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack B4 2434732 wikitext text/x-wiki 9gi48xyan2dm047qudhqqck1wlrpf4j 2434735 2434732 2026-07-09T08:03:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cumin2003.codfw.wmnet 2434735 wikitext text/x-wiki ew82evyuj8q8nq7nfssyqdr584lod6k 2434736 2434735 2026-07-09T08:08:01Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 27 hosts with reason: codfw rack B4 depool for maintenance 2434736 wikitext text/x-wiki krqmz6eq5c5g9gubmq1qo3bbqb9i8kw 2434737 2434736 2026-07-09T08:08:07Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2204: codfw rack B4 depool for maintenance 2434737 wikitext text/x-wiki dkxjl919e0593ir8ua9kghgc5ihkd2i 2434738 2434737 2026-07-09T08:08:48Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2204: codfw rack B4 depool for maintenance 2434738 wikitext text/x-wiki t2i97kxffeuk3r2l3agyg0hjoakbiiu 2434739 2434738 2026-07-09T08:08:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2003.codfw.wmnet 2434739 wikitext text/x-wiki nev7nguk85jbvuv9c4b248s23ia1l4f 2434740 2434739 2026-07-09T08:08:58Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2205: codfw rack B4 depool for maintenance 2434740 wikitext text/x-wiki 0am5q7uhixexkerl5bt2elm6w7pj69k 2434741 2434740 2026-07-09T08:09:49Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2205: codfw rack B4 depool for maintenance 2434741 wikitext text/x-wiki tcse9f6rc4z42yrpxbzifhursuiy3wc 2434742 2434741 2026-07-09T08:09:54Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet 2434742 wikitext text/x-wiki 4ck9d1kcsqbo9kj8p48mo05rl6nixuo 2434743 2434742 2026-07-09T08:10:28Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet 2434743 wikitext text/x-wiki p43ap62tndsbsqzd4rmx8to19oaoex3 2434744 2434743 2026-07-09T08:10:32Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2004.codfw.wmnet 2434744 wikitext text/x-wiki m4444qwe7ayvs0z92vgmw6n9n249k6v 2434745 2434744 2026-07-09T08:13:53Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet 2434745 wikitext text/x-wiki 8hbhhbgn9nj92o7gch5px1i7vambz5l 2434746 2434745 2026-07-09T08:13:54Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for codfw rack B4 2434746 wikitext text/x-wiki i7e4h4r1vezqzs9hxoy2jndo1rcpfj0 2434747 2434746 2026-07-09T08:15:29Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b4-codfw,lsw1-b4-codfw IPv6,lsw1-b4-codfw.mgmt with reason: Switch maintenance 2434747 wikitext text/x-wiki 5wq7ee1v9z30b9ho4qspv5tg9vmc8wv 2434748 2434747 2026-07-09T08:15:35Z Stashbot 7414 XioNoX: lsw1-b4-codfw> request system reboot - T430910 2434748 wikitext text/x-wiki oa7zfujhr55o4jrnkc7piwnb3fw3c96 2434749 2434748 2026-07-09T08:15:58Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet 2434749 wikitext text/x-wiki ajwcs4h25v2u87m6dfuxw0zcmtifg2t 2434750 2434749 2026-07-09T08:16:22Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{ml-serve1003.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2434750 wikitext text/x-wiki 1c2johtyuso27ej5oi426i19o5cman4 2434751 2434750 2026-07-09T08:16:26Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet 2434751 wikitext text/x-wiki h0hmi5wrhj45pyh6oto78rbl2bhtdoi 2434752 2434751 2026-07-09T08:16:27Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet 2434752 wikitext text/x-wiki clhldj8qv5hl2ges1vqcc6vtgwyjpw7 2434753 2434752 2026-07-09T08:16:32Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2004.codfw.wmnet 2434753 wikitext text/x-wiki mlrsv9hmgb75de6wbrzjemcsvx10zcz 2434754 2434753 2026-07-09T08:16:46Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1003.eqiad.wmnet with OS bookworm 2434754 wikitext text/x-wiki loxo5quumanoj31yxsm4lx2aacky7ww 2434756 2434754 2026-07-09T08:26:39Z Stashbot 7414 moritzm: failover Ganeti master in codfw to ganeti2048 T430928 2434756 wikitext text/x-wiki 1pkwguvxah9ls9wsqewag7m3tprg8wj 2434758 2434756 2026-07-09T08:31:48Z Stashbot 7414 hashar@deploy2003: Rolling back deployment 2434758 wikitext text/x-wiki dt5tzkvxr4f46bho7u5avcieus9dg5n 2434759 2434758 2026-07-09T08:32:05Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet 2434759 wikitext text/x-wiki e3ctjuix6exow9jwu6ad3o8ehqee83m 2434760 2434759 2026-07-09T08:32:12Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2129,2137-2138,2156,2270-2271].codfw.wmnet 2434760 wikitext text/x-wiki c7qdgu7zldtce6imz6ygpoix8fyydb9 2434762 2434760 2026-07-09T08:35:33Z Stashbot 7414 hashar@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.10 refs T430829 2434762 wikitext text/x-wiki 3rmlpquf3fcbr084af0rd73bvrqv960 2434763 2434762 2026-07-09T08:36:13Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage 2434763 wikitext text/x-wiki h3u52wnfklt8fd4ym62t1nsn02ga6q3 2434764 2434763 2026-07-09T08:37:41Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2204: codfw rack B4 repool after maintenance 2434764 wikitext text/x-wiki lg17xd8h6ic9hwo48xxhh63b6r48zlp 2434765 2434764 2026-07-09T08:38:18Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2205: codfw rack B4 repool after maintenance 2434765 wikitext text/x-wiki 6ce2wr6b4fhmm308pcvxrpffmidep34 2434766 2434765 2026-07-09T08:39:18Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage 2434766 wikitext text/x-wiki 4m73pmhm6j3qpfk8tdq99hibcjfi25b 2434767 2434766 2026-07-09T08:55:13Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1003.eqiad.wmnet with OS bookworm 2434767 wikitext text/x-wiki hms6bwybbo6hh672ywxyzr6jbrxkgix 2434768 2434767 2026-07-09T08:55:18Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet 2434768 wikitext text/x-wiki 1be4fh5xwv059ok2s4wtbl1sbymol6p 2434769 2434768 2026-07-09T08:55:19Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet 2434769 wikitext text/x-wiki 1bznm07nnhytkqvmdw2wmvm4iuuj1ys 2434770 2434769 2026-07-09T08:55:20Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{ml-serve1003.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2434770 wikitext text/x-wiki 8jkixxoijf6ukc41m0uq0ey4l6tcp88 2434771 2434770 2026-07-09T08:57:32Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1003.eqiad.wmnet with OS bookworm 2434771 wikitext text/x-wiki i2hkjki69ab3h1nikjwnwyldeohxtic 2434772 2434771 2026-07-09T08:57:49Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.move-vlan for host ml-serve1003 2434772 wikitext text/x-wiki naw1559t944buko6aovfommenqk4v9b 2434773 2434772 2026-07-09T08:58:32Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.dns.netbox 2434773 wikitext text/x-wiki 2mimn3vwgf32aidym7k09zj920avv6g 2434775 2434773 2026-07-09T09:06:24Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1003 - klausman@cumin1003" 2434775 wikitext text/x-wiki aeg6zqogvjsbtc3nw5kkciugeomjtvj 2434776 2434775 2026-07-09T09:07:17Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1003 - klausman@cumin1003" 2434776 wikitext text/x-wiki i79mgj0drar33rvl5bkm3l1x7semrf8 2434777 2434776 2026-07-09T09:07:18Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434777 wikitext text/x-wiki c1ltdplvwmufghw2a16lsbfgsy1a3xj 2434778 2434777 2026-07-09T09:07:19Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.dns.wipe-cache ml-serve1003.eqiad.wmnet 81.32.64.10.in-addr.arpa 1.8.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434778 wikitext text/x-wiki d1z6pn2x7vtro24rq7md03tbcq480hf 2434779 2434778 2026-07-09T09:07:21Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-serve1003.eqiad.wmnet 81.32.64.10.in-addr.arpa 1.8.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434779 wikitext text/x-wiki bthpu5j0fhin07p4hoah3gj6prsomzr 2434780 2434779 2026-07-09T09:07:22Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1003 2434780 wikitext text/x-wiki r64o8pbem9xcq2k45qpu1nob1k226j5 2434781 2434780 2026-07-09T09:08:08Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1003 2434781 wikitext text/x-wiki ohen6137b57k1717o18hqfmcci4qlzl 2434782 2434781 2026-07-09T09:08:09Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ml-serve1003 2434782 wikitext text/x-wiki s3oc4v24s9nepwh61rpc5sw1vs6flle 2434785 2434782 2026-07-09T09:17:27Z Stashbot 7414 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1309102|NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]] 2434785 wikitext text/x-wiki d911t8ufdv0543ttomnv6ekyfvlpqxr 2434786 2434785 2026-07-09T09:18:59Z Stashbot 7414 jmm@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: reboot 2434786 wikitext text/x-wiki 6hfeg7k1btj48cur3napc0v6id4ocfy 2434787 2434786 2026-07-09T09:19:08Z Stashbot 7414 urbanecm@deploy2003: urbanecm: Backport for [[gerrit:1309102|NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2434787 wikitext text/x-wiki li5d913ountzalcwg2981ur5398xsml 2434788 2434787 2026-07-09T09:19:36Z Stashbot 7414 urbanecm@deploy2003: urbanecm: Continuing with deployment 2434788 wikitext text/x-wiki f4ouiqb6wxhn84hhfway5o6vf9mop32 2434789 2434788 2026-07-09T09:23:21Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2204: codfw rack B4 repool after maintenance 2434789 wikitext text/x-wiki kfe8dyytrqmx0zvirgtdu971yulw3lg 2434790 2434789 2026-07-09T09:23:29Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet 2434790 wikitext text/x-wiki ba5npqo6iiyfl8v7ywot045toi13rra 2434791 2434790 2026-07-09T09:23:45Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2205: codfw rack B4 repool after maintenance 2434791 wikitext text/x-wiki opuet23fugpbswjfnd741qjeg39ir9k 2434792 2434791 2026-07-09T09:23:56Z Stashbot 7414 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309102|NewcomerTasks: Don't fatal on an unconfigured conversion-map fallback (T431668)]] (duration: 06m 27s) 2434792 wikitext text/x-wiki iwq6kyb30lzbyed4j66o39wxdogxkjq 2434793 2434792 2026-07-09T09:25:27Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet 2434793 wikitext text/x-wiki rzb3l5dy42klfzoaifsz378p0xtap45 2434794 2434793 2026-07-09T09:25:55Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor-dev2001.codfw.wmnet 2434794 wikitext text/x-wiki 9j927bkfny223plz6f6u25t5z33nsvc 2434796 2434794 2026-07-09T09:27:16Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage 2434796 wikitext text/x-wiki 769lfw979yuzk36d5gkyipf9ele4py1 2434797 2434796 2026-07-09T09:29:15Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet 2434797 wikitext text/x-wiki oak7e16db2w614nn1bwk23htf5cluww 2434798 2434797 2026-07-09T09:29:26Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet 2434798 wikitext text/x-wiki 1x8eqf5ggm2kwkq5arsgrivzv8q4xa2 2434799 2434798 2026-07-09T09:29:55Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor-dev2001.codfw.wmnet 2434799 wikitext text/x-wiki rov5ap2zkacx8zv3je5aqzwaynr8un1 2434800 2434799 2026-07-09T09:33:06Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1003.eqiad.wmnet with reason: host reimage 2434800 wikitext text/x-wiki p7r7rcxu99bjbc3dn2itesd5qv50gon 2434801 2434800 2026-07-09T09:33:16Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet 2434801 wikitext text/x-wiki lig939k0imaepthiie4gtrywgn3glwp 2434802 2434801 2026-07-09T09:34:55Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2007.codfw.wmnet 2434802 wikitext text/x-wiki i2zd5zcid68syuqj7kktqdaf8l6xdta 2434803 2434802 2026-07-09T09:35:57Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetboard2003.codfw.wmnet 2434803 wikitext text/x-wiki rnfdfsoskl9njzy4nkf4su397wby7ju 2434804 2434803 2026-07-09T09:37:15Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet 2434804 wikitext text/x-wiki m50u1mlapz9j4d81c0dtbozb9l51wr1 2434806 2434804 2026-07-09T09:37:21Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet 2434806 wikitext text/x-wiki 50eo9bj7ibqsn5eirbryrg6sua3u8vj 2434807 2434806 2026-07-09T09:39:37Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.netbox 2434807 wikitext text/x-wiki jsbkgur6xexo8dw1um4lisuy4zklwz6 2434808 2434807 2026-07-09T09:39:48Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet 2434808 wikitext text/x-wiki tgntvsxmkfz0fv608zwm9ty2ne4smue 2434809 2434808 2026-07-09T09:41:12Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet 2434809 wikitext text/x-wiki 36u0chtb1px8bed0mcy0yo3jd6vl4ij 2434813 2434809 2026-07-09T09:49:02Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2007.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" 2434813 wikitext text/x-wiki 0xdd767fsb2nnpefoeby0fy14zzx8gk 2434815 2434813 2026-07-09T09:49:21Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet 2434815 wikitext text/x-wiki tqujyr2cvlv166k0yrkd9j3n7m6k4ad 2434816 2434815 2026-07-09T09:49:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2007.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" 2434816 wikitext text/x-wiki 4m31wbc5prmai1h386pphe3sr46j882 2434817 2434816 2026-07-09T09:49:41Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434817 wikitext text/x-wiki 4twqkd4v8odnt0uyahf6hrrpfa8lrog 2434818 2434817 2026-07-09T09:49:42Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2007.codfw.wmnet 2434818 wikitext text/x-wiki stbzfvwl5twcqoph3vvekjxqsczj0w6 2434819 2434818 2026-07-09T09:49:43Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2008.wikimedia.org 2434819 wikitext text/x-wiki 8ty2ytv25vvhwrw46n9g06zqwfigdql 2434820 2434819 2026-07-09T09:49:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet 2434820 wikitext text/x-wiki sg5eqcquu76qjbtpwuq3gcdb1bc2d22 2434821 2434820 2026-07-09T09:50:00Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1001.eqiad.wmnet 2434821 wikitext text/x-wiki snrkxhh1s2g6qxjxo29boraq8j66u0j 2434822 2434821 2026-07-09T09:50:03Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1003.eqiad.wmnet with OS bookworm 2434822 wikitext text/x-wiki 3tc4ps6nxeivc3chfzrkwdtr5syjlq0 2434823 2434822 2026-07-09T09:50:17Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.reimage for host ml-serve1004.eqiad.wmnet with OS bookworm 2434823 wikitext text/x-wiki 2brx5o0i12vdwo8jv7etyhv6b5p3p5w 2434824 2434823 2026-07-09T09:50:45Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.move-vlan for host ml-serve1004 2434824 wikitext text/x-wiki j48ufyttn41gverd5vafva23naieqq4 2434825 2434824 2026-07-09T09:50:51Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org 2434825 wikitext text/x-wiki 4yrjdx7jsn9s53alk5hj1u52jg94zsk 2434826 2434825 2026-07-09T09:52:07Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.dns.netbox 2434826 wikitext text/x-wiki r7gta0d6y41nthnb7gj0nqkjgid2qes 2434827 2434826 2026-07-09T09:52:50Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet 2434827 wikitext text/x-wiki c8bqib9xn6fas9pxxr9exphrmm20461 2434828 2434827 2026-07-09T09:52:52Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-coord1001.eqiad.wmnet 2434828 wikitext text/x-wiki palvbs31m0439lckft7zjdkn7nig6nd 2434829 2434828 2026-07-09T09:55:26Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet 2434829 wikitext text/x-wiki gixab92wol8lpniovo33f8urh92m56i 2434830 2434829 2026-07-09T09:55:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet 2434830 wikitext text/x-wiki 9mpim9f04nd8is42n1xqu5i04zl2kw1 2434831 2434830 2026-07-09T09:55:51Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host an-test-coord1001.eqiad.wmnet 2434831 wikitext text/x-wiki t57m4cg9gaokcezsco193psiisdp6x9 2434832 2434831 2026-07-09T09:56:30Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1004 - klausman@cumin1003" 2434832 wikitext text/x-wiki oj3k59w5l237bjd3dur3izv59i1i1un 2434833 2434832 2026-07-09T09:57:06Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ml-serve1004 - klausman@cumin1003" 2434833 wikitext text/x-wiki 0qm160vwqxuc55a2oxufd45bk4xxofu 2434834 2434833 2026-07-09T09:57:07Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434834 wikitext text/x-wiki cy08z84i8lve3gexv8os7w6vayjaxew 2434835 2434834 2026-07-09T09:57:09Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.dns.wipe-cache ml-serve1004.eqiad.wmnet 50.48.64.10.in-addr.arpa 0.5.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434835 wikitext text/x-wiki dcu7q3srd4v2ses2ytdh6z4jyvtm5bj 2434836 2434835 2026-07-09T09:57:10Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-serve1004.eqiad.wmnet 50.48.64.10.in-addr.arpa 0.5.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2434836 wikitext text/x-wiki bjjzkrso20q5ea36p466hn60m91o21p 2434837 2434836 2026-07-09T09:57:12Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1004 2434837 wikitext text/x-wiki 3ce16vhu3ep6knb4h1et4vj5iuvye8c 2434838 2434837 2026-07-09T09:57:19Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org 2434838 wikitext text/x-wiki 02zqpnz1io66rqrdst8kagyfpjr31p3 2434839 2434838 2026-07-09T09:57:29Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.netbox 2434839 wikitext text/x-wiki dan6ku5cm1yk0nepfdmaub2xnai1i2o 2434840 2434839 2026-07-09T09:57:48Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1004 2434840 wikitext text/x-wiki s1inxqxjp3s1lgkhl8nz988eikt9c00 2434841 2434840 2026-07-09T09:57:49Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ml-serve1004 2434841 wikitext text/x-wiki 57u3vuu1evuo5x1gjqbaj5ywjn50lo1 2434843 2434841 2026-07-09T10:00:33Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434843 wikitext text/x-wiki hq92s80kunzd7xit2aywipxvg39j8h6 2434844 2434843 2026-07-09T10:00:34Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org 2434844 wikitext text/x-wiki q9nj8qdb6s1uab0mtxcny9gu90yh1i6 2434856 2434844 2026-07-09T10:14:10Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1001.eqiad.wmnet 2434856 wikitext text/x-wiki 4xyvw05b5of4kob7h2at2txr9g9g44e 2434857 2434856 2026-07-09T10:14:27Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1001.eqiad.wmnet 2434857 wikitext text/x-wiki 3c1zhpifkcnaffapcpmotihdn0p0wnu 2434858 2434857 2026-07-09T10:15:04Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1004.eqiad.wmnet with reason: host reimage 2434858 wikitext text/x-wiki t7trxpjbyx5stxca8drypz86s9u2zst 2434859 2434858 2026-07-09T10:15:59Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org 2434859 wikitext text/x-wiki itr0ioi58a0dz07s2vo7d2b7c1uu08l 2434861 2434859 2026-07-09T10:16:54Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org 2434861 wikitext text/x-wiki 5vgie0q8yqrc9udjk3g5azppdcn3jh6 2434862 2434861 2026-07-09T10:17:27Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org 2434862 wikitext text/x-wiki ceaoe22drqp34cijph97g8mfd01fy04 2434863 2434862 2026-07-09T10:17:59Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.addnode for new host ganeti2031.codfw.wmnet to cluster codfw and group B 2434863 wikitext text/x-wiki brwuy5l78ipem9ugykood4wqk5fgcps 2434864 2434863 2026-07-09T10:18:29Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1003 2434864 wikitext text/x-wiki 2detzbmh8pkvb5o4yeersc2bu21gyfo 2434865 2434864 2026-07-09T10:18:39Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1003 2434865 wikitext text/x-wiki b1bak8bhbujs2j0ihmknhcj81tv5qet 2434866 2434865 2026-07-09T10:18:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org 2434866 wikitext text/x-wiki jinvp3jdheg3coebqsfvobv6j64p0ys 2434867 2434866 2026-07-09T10:18:50Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1004.eqiad.wmnet with reason: host reimage 2434867 wikitext text/x-wiki 9ux7lu99syhacvulknjes2l4ayosnno 2434868 2434867 2026-07-09T10:19:25Z Stashbot 7414 moritzm: readded ganeti2031 to the codfw Ganeti cluster T430910 2434868 wikitext text/x-wiki 6ju6fxbb7lwdzu7jfnme17mw4p78xbm 2434869 2434868 2026-07-09T10:19:40Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2031.codfw.wmnet to cluster codfw and group B 2434869 wikitext text/x-wiki prvyadlxaa51u9zhvjahqz6lx6c7qn8 2434871 2434869 2026-07-09T10:21:44Z Stashbot 7414 moritzm: failover Ganeti master in codfw/routed to ganeti2034 T430928 2434871 wikitext text/x-wiki 3o7pkulgw5lv1v0rpvo5n0w1miriyh3 2434872 2434871 2026-07-09T10:24:00Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet 2434872 wikitext text/x-wiki tuqfdfb3u9t24ct2aqupzumwx5xxwta 2434873 2434872 2026-07-09T10:24:13Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org 2434873 wikitext text/x-wiki mxlow37evzsz0ixzggpy8z8qjg8lml2 2434874 2434873 2026-07-09T10:24:17Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org 2434874 wikitext text/x-wiki 9om18pv8qxije836uronofpj95zp4s6 2434877 2434874 2026-07-09T10:29:57Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T431656 2434877 wikitext text/x-wiki 8l4czixknu76y1k9df2ilkg4tmy42iy 2434878 2434877 2026-07-09T10:30:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org 2434878 wikitext text/x-wiki qmfln1ifayqhdvqezh725znl0vzksw7 2434880 2434878 2026-07-09T10:30:48Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org 2434880 wikitext text/x-wiki tr6phmwropq4n6fro5xbwkegokvwvlu 2434881 2434880 2026-07-09T10:31:03Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install2005.wikimedia.org 2434881 wikitext text/x-wiki 2jppoa8y54b56ihhwpm59jbh7hg9fe4 2434882 2434881 2026-07-09T10:31:07Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org 2434882 wikitext text/x-wiki 79u1xr5mauhmlfqikkpv7v2iv3x70e1 2434883 2434882 2026-07-09T10:35:22Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1004.eqiad.wmnet with OS bookworm 2434883 wikitext text/x-wiki aplejnf8vfhpo6mt5qoj6kqdc3zd0v9 2434884 2434883 2026-07-09T10:37:30Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org 2434884 wikitext text/x-wiki sfntxqj7vrnu0fs73nw4tzxvdrn0zer 2434885 2434884 2026-07-09T10:37:35Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org 2434885 wikitext text/x-wiki 9xnc4kunymgs1iuwt44wi1x5of00baz 2434886 2434885 2026-07-09T10:38:53Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet 2434886 wikitext text/x-wiki m085908ydhwez6wnppb73ptui9cfr5l 2434887 2434886 2026-07-09T10:39:34Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T431656 2434887 wikitext text/x-wiki shqqi2z9mzwpaqpt0sc6elc0hehgrsy 2434888 2434887 2026-07-09T10:40:08Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.host-bgp for host ml-serve1003 2434888 wikitext text/x-wiki f0o1mognmm2p1w7soh25hbsbh5domju 2434889 2434888 2026-07-09T10:40:19Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.host-bgp (exit_code=0) for host ml-serve1003 2434889 wikitext text/x-wiki ob60fyhe0ddxxu30kouxrhz4nxioeqc 2434890 2434889 2026-07-09T10:41:58Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T431656 2434890 wikitext text/x-wiki bkl8eqysna7ukg7dp5goq85kt9860f3 2434892 2434890 2026-07-09T10:50:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2005.codfw.wmnet 2434892 wikitext text/x-wiki hlhuyft9u687zpwkq75ltnxwkgkkfbg 2434893 2434892 2026-07-09T10:51:39Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T431656 2434893 wikitext text/x-wiki 5mx1xjsbqvn1wb6fj065nuy9xkhxync 2434894 2434893 2026-07-09T10:51:42Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org 2434894 wikitext text/x-wiki 51on3ut6d5re9djc6a4ejnpjvgrop98 2434896 2434894 2026-07-09T10:55:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.netbox 2434896 wikitext text/x-wiki ojhgu0f3w4h9g68lbcuiztwhgymdfs4 2434897 2434896 2026-07-09T10:57:28Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org 2434897 wikitext text/x-wiki oo61bzdn2dsm9wtrhs65p0z38qjpjjc 2434898 2434897 2026-07-09T10:57:48Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org 2434898 wikitext text/x-wiki rjvf185wr9h17r09l7ffxx13fdmppsp 2434899 2434898 2026-07-09T10:59:44Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" 2434899 wikitext text/x-wiki irgs7vj7b5uv8hp6lokkzekp77mupyu 2434900 2434899 2026-07-09T11:00:49Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" 2434900 wikitext text/x-wiki r4mu5148hz643gva3rbbb0ui63iseob 2434901 2434900 2026-07-09T11:00:50Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2434901 wikitext text/x-wiki 5zs94lklu334gss4dfz3kohoh7973hd 2434902 2434901 2026-07-09T11:00:51Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2005.codfw.wmnet 2434902 wikitext text/x-wiki f9lvcflbj8ml2b1mxo20y8zhq15qsbt 2434903 2434902 2026-07-09T11:03:29Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org 2434903 wikitext text/x-wiki 94dfq5s3oeyrcu1p49gzkmv1hv3twe0 2434904 2434903 2026-07-09T11:04:15Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-rw1001.wikimedia.org 2434904 wikitext text/x-wiki 8zlyi2g8w2vsbdrpgg54j3tc1zel4sx 2434905 2434904 2026-07-09T11:04:24Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-rw2001.wikimedia.org 2434905 wikitext text/x-wiki 0m5h9h1kkhtolbrygy6vabhqbvjjptw 2434906 2434905 2026-07-09T11:08:08Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw1001.wikimedia.org 2434906 wikitext text/x-wiki rza6a85qbs9mqdrobd4ewrxg6io8nsx 2434907 2434906 2026-07-09T11:08:17Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw2001.wikimedia.org 2434907 wikitext text/x-wiki crqxeuaf29g0mvso2ludtfi4t32xlzp 2434908 2434907 2026-07-09T11:12:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint1001.eqiad.wmnet 2434908 wikitext text/x-wiki 17pw8794mme7smmb29azyk8554xbil1 2434909 2434908 2026-07-09T11:12:32Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint2001.codfw.wmnet 2434909 wikitext text/x-wiki 9pd3kkf68c9mxjjkld0zdzjym7gru3g 2434910 2434909 2026-07-09T11:16:04Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint1001.eqiad.wmnet 2434910 wikitext text/x-wiki 8moc62fpidyu2zaux3sp7fb02vm06sp 2434911 2434910 2026-07-09T11:16:13Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint2001.codfw.wmnet 2434911 wikitext text/x-wiki 1ydbjpmi7guvsft3gh8wr81hqc078cn 2434912 2434911 2026-07-09T11:23:27Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org 2434912 wikitext text/x-wiki fnbgd3q9hw7pdyt4e2lf55yb5y6ce6x 2434915 2434912 2026-07-09T11:27:03Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org 2434915 wikitext text/x-wiki 98heojhv67icwxtu1ejk8qfa18wxth0 2434917 2434915 2026-07-09T11:48:29Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2434917 wikitext text/x-wiki tov17lvitc208zodna2lq6y030xw06s 2434918 2434917 2026-07-09T11:50:15Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2434918 wikitext text/x-wiki s5g3mu0fqfhwrw65o00oqsmgrh93na0 2434920 2434918 2026-07-09T11:53:37Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2434920 wikitext text/x-wiki 98l8gjslmi1n6pg7behahf9nn1hho73 2434922 2434920 2026-07-09T11:55:19Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2434922 wikitext text/x-wiki 4o2p53ze6zonwnizyoju64qpv81mdnm 2434923 2434922 2026-07-09T11:58:23Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org 2434923 wikitext text/x-wiki 3c9tu7k4nvevf7u0rcttsyshmrsdryv 2434924 2434923 2026-07-09T11:58:56Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid2003.codfw.wmnet 2434924 wikitext text/x-wiki ivc7zy72quncafxor7c0wbx5yr5942n 0 17290 2434162 2433962 2026-07-08T12:27:20Z Stashbot 7414 hashar: deployment-prep: on deployment server: clearing old branches for mediawiki/extensions and mediawiki/skins # T428864 2434162 wikitext text/x-wiki n6sc9zdi4gr4igjaq8j0tp10e6zgcjd 2434185 2434162 2026-07-08T13:04:44Z Stashbot 7414 hashar: deployment-prep: git repack on /srv/mediawiki-staging/php-master 2434185 wikitext text/x-wiki 6ghb0ho21a22mtvkl41ae1ox0c0c9fj 2434312 2434185 2026-07-08T14:36:25Z Stashbot 7414 hashar: contint1003, contint2003: manually installed `docker-buildx` Debian package to validate https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308659 # T431582 2434312 wikitext text/x-wiki 5hcu32qotyynwpdch4337iyoa6k9one 2434562 2434312 2026-07-08T19:21:33Z Stashbot 7414 mutante: gerrit - replacing registerEmailPrivateKey in Gerrit config - this invalidates pending/outstanding email validation links for gerrit users - but does not affect active accounts or already verified email addresses 2434562 wikitext text/x-wiki 1cccz5axbyghus6i2e3yg97helyudhy 2434609 2434562 2026-07-08T20:58:55Z Stashbot 7414 brennen: patchdemo: deployed https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/367 (T427964) 2434609 wikitext text/x-wiki ijutzehldtv339r9nd43ykqnb4rs714 2434921 2434609 2026-07-09T11:55:06Z Stashbot 7414 hashar: retriggering postmerge change for T431582: zuul enqueue --trigger gerrit --pipeline postmerge --project machinelearning/liftwing/inference-services --change 1308631,3 2434921 wikitext text/x-wiki c5z0zeld7gg5sknwgd6g0p3mcw88p14 498 30942 2434259 2433545 2026-07-08T13:52:28Z Stashbot 7414 andrewbogott: removing some cloudvirts from the 'maintenance' aggregate. I imagine they are there in error after some automated reboots. cloudvirt1049, cloudvirt1053, cloudvirt1064, cloudvirt1078, cloudvirt1080, cloudvirtlocal1001 2434259 wikitext text/x-wiki bz0rybvjnw5r3chnq0kquk3r921ucok 2434417 2434259 2026-07-08T16:49:36Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 2434417 wikitext text/x-wiki g3ffdxewjm39wz8cw1vtfhewxodc83v 2434419 2434417 2026-07-08T16:49:53Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 2434419 wikitext text/x-wiki 7k4war279q4na5e81i7ihtpimau9kgu 2434421 2434419 2026-07-08T16:53:07Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 2434421 wikitext text/x-wiki 1clqhvg1au0o7maw5lp0syofihz1346 2434422 2434421 2026-07-08T16:53:28Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 2434422 wikitext text/x-wiki qrdugjgdywxf697cy9w0mh4nvqt4yl7 2434425 2434422 2026-07-08T16:55:27Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 2434425 wikitext text/x-wiki ewrkzf5k136we7ysv4wix4xlzr0ge0v 2434426 2434425 2026-07-08T16:55:47Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 2434426 wikitext text/x-wiki ef24mbxjn9oomhlydk6nn6y7q2n7v9q 2434429 2434426 2026-07-08T16:56:31Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2434429 wikitext text/x-wiki 2ujf4rfvevw0u744sexj5mjtwc0jjgc 2434430 2434429 2026-07-08T16:57:12Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2434430 wikitext text/x-wiki 5rsb811k10sil2qze7bql5wkkr4yq75 2434431 2434430 2026-07-08T16:58:08Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 2434431 wikitext text/x-wiki h3w2145c4hw3a7rf38wrp6ifes3qw5k 2434432 2434431 2026-07-08T16:58:29Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 2434432 wikitext text/x-wiki pfpmvem1vq9gm1a7vv5bnsx052tx6se 2434433 2434432 2026-07-08T17:03:14Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 2434433 wikitext text/x-wiki ju1ndhjyvexypwhcoi5x23kaumcjb2a 2434434 2434433 2026-07-08T17:03:34Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 2434434 wikitext text/x-wiki p7g19d4ac3wjrurcnvma2eaw8ufkucq 2434438 2434434 2026-07-08T17:04:45Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2434438 wikitext text/x-wiki q12sondahojf16uy4s871tjwudpax08 2434439 2434438 2026-07-08T17:05:19Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2434439 wikitext text/x-wiki g6tp26gntko4np47zrq3lhkotkez6jy 2434451 2434439 2026-07-08T17:20:22Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/326 2434451 wikitext text/x-wiki okyjrnytsbmnc7ch2rd6rw6b7qnk0od 2434452 2434451 2026-07-08T17:20:44Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/326 2434452 wikitext text/x-wiki a5ndgf7eas8wari49cpw3qwoo7wsg7k 2434453 2434452 2026-07-08T17:21:04Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2434453 wikitext text/x-wiki f2cz2wqor3dj6ijmntiz2o1ah2rx79v 2434454 2434453 2026-07-08T17:21:29Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2434454 wikitext text/x-wiki 9gky7tnmwxzps32eqn0xmpm7tbw2qym 2434564 2434454 2026-07-08T19:23:20Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch 2434564 wikitext text/x-wiki a43sadgw54ndtv64xhkp5xflq4qourf 2434565 2434564 2026-07-08T19:23:53Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch 2434565 wikitext text/x-wiki bf6ntp3vqouk7g4totxh86t129pywja 2434566 2434565 2026-07-08T19:25:34Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2434566 wikitext text/x-wiki nas19wx37ciu37ekbcyy9rdj67v1vg7 2434567 2434566 2026-07-08T19:25:52Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2434567 wikitext text/x-wiki jgbh77bvfb6m0h8wara6rhmxrsqjy03 2434847 2434567 2026-07-09T10:10:35Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance 2434847 wikitext text/x-wiki teru3h9pb2tpi1seo4cnold6oiblbhh 2434848 2434847 2026-07-09T10:11:01Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) 2434848 wikitext text/x-wiki 85eo7uc5x6c434bkci0ibb2wzx0jp3u 2434849 2434848 2026-07-09T10:11:32Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance 2434849 wikitext text/x-wiki pk04qdl35khkibxoxcyfk6dj8nkcias 2434850 2434849 2026-07-09T10:11:35Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99) 2434850 wikitext text/x-wiki jk5n5qsuqc7qez996jb2q1vy0ne3hs3 0 63011 2434476 2396488 2026-07-08T18:08:29Z KSoučková-WMF 35993 /* Procedure */ add updating tcpircbot 2434476 wikitext text/x-wiki hqxthgrf0u5g9p86uvsc30q0frykfhn 0 441014 2434450 2367549 2026-07-08T17:18:46Z SFrench-WMF 37362 /* Operations */ Add section on depooling a cluster 2434450 wikitext text/x-wiki dlwzs8b3874c3mq0zxo3viysvtv4ebf 0 442952 2434783 2428493 2026-07-09T09:11:48Z TLepage-WMF 57137 add myself 2434783 wikitext text/x-wiki 94bi9sok820kuhhtzeaxsory4wjxyz8 498 443261 2434192 2421868 2026-07-08T13:08:40Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-idp' 2434192 wikitext text/x-wiki i4iipauciup4mcze4a8lz8m1b99b2v7 2434196 2434192 2026-07-08T13:12:20Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-idp' 2434196 wikitext text/x-wiki tgpp35d5ebotxkz15ml2ljd26tnd62n 2434200 2434196 2026-07-08T13:14:01Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud 2434200 wikitext text/x-wiki bdmw7j4i5ldx6f2agegwgo2qmc9xdn0 2434203 2434200 2026-07-08T13:16:27Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud 2434203 wikitext text/x-wiki r2klkld2carbo91p3mu5usksxyod4bx 2434204 2434203 2026-07-08T13:17:40Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud 2434204 wikitext text/x-wiki mwe2asebs8mzxomqvb3vqdhyw1vsgu0 2434214 2434204 2026-07-08T13:22:09Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud 2434214 wikitext text/x-wiki qu7ok23r4492xcrmnn97z6iaqppvirt 2434218 2434214 2026-07-08T13:24:28Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-idp-2 2434218 wikitext text/x-wiki dhn1eegbz060yez7c7kpcov2muznjcb 2434220 2434218 2026-07-08T13:25:18Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-idp-2 2434220 wikitext text/x-wiki 7aaicxagngwzhdj24g94q8ff7eym42v 2434221 2434220 2026-07-08T13:25:48Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-idp' 2434221 wikitext text/x-wiki 8s6rgk9q0c9f7axohum4zjhuft024ar 2434233 2434221 2026-07-08T13:35:49Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-idp' 2434233 wikitext text/x-wiki c253aowvjzenr2m82bdea1yxhoexjz1 2434410 2434233 2026-07-08T16:28:54Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'syslog-server-audit' 2434410 wikitext text/x-wiki tlptpd3gnthvytnvn879n5iycx8rwc7 2434411 2434410 2026-07-08T16:29:01Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'syslog-server-audit' 2434411 wikitext text/x-wiki 4ui64iwxc9mkxip1lchwaontsi2mcle 2434412 2434411 2026-07-08T16:34:41Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'syslog-server-audit' 2434412 wikitext text/x-wiki aq5elx73vfx76urxampag3wonqzrhbz 2434413 2434412 2026-07-08T16:40:38Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'syslog-server-audit' 2434413 wikitext text/x-wiki qvv80aac2eaestdqs20slnox5oxmjqz 2434428 2434413 2026-07-08T16:56:21Z Stashbot 7414 taavi: converting syslogaudit svc names to CNAMEs for easier manageability 2434428 wikitext text/x-wiki jq0b1277e157z0f6y6spiix3o5yqe1y 2434441 2434428 2026-07-08T17:06:41Z Stashbot 7414 taavi: move syslogaudit1.svc to syslog-server-audit-3 T402007 2434441 wikitext text/x-wiki qn9q7hiqv7rfyn3sb5jj6nq5czhl1jp 2434444 2434441 2026-07-08T17:10:46Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'syslog-server-audit' 2434444 wikitext text/x-wiki 9x9n5f09jrwvch12wooztta1ogjl8b5 2434448 2434444 2026-07-08T17:16:50Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'syslog-server-audit' 2434448 wikitext text/x-wiki prf95d5pfjz8rkxwosm1t2mq1ti9i5v 2434455 2434448 2026-07-08T17:21:56Z Stashbot 7414 taavi: move syslogaudit2.svc to syslog-server-audit-4 T402007 2434455 wikitext text/x-wiki kfmzcyne2cgu9xidwknqubrp3itwcwv 2434458 2434455 2026-07-08T17:36:18Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-db' 2434458 wikitext text/x-wiki pjghac8t47n51uytisqiie1fggdx7fr 2434459 2434458 2026-07-08T17:41:19Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-db' 2434459 wikitext text/x-wiki 08hjfzp30u1tsudh2m3uagub065n3sm 2434461 2434459 2026-07-08T17:42:18Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-db-3 2434461 wikitext text/x-wiki 9orjb5d8vvoh02lx4kc787sz54lvh2u 2434464 2434461 2026-07-08T17:43:08Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-db-3 2434464 wikitext text/x-wiki 57gsw0goy4wczup0m74zgcrhacl90bz 2434465 2434464 2026-07-08T17:43:47Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-db' 2434465 wikitext text/x-wiki jteyznpap9ir6lsxnv7gw17ej829unn 2434467 2434465 2026-07-08T17:48:38Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-db' 2434467 wikitext text/x-wiki lqnb1j8o1yg9jrd2thylwv2s1pu0mq0 2434477 2434467 2026-07-08T18:08:33Z Stashbot 7414 taavi: Replace cloudinfra-db04 with cloudinfra-db-5 replicating off of db03 T402005 2434477 wikitext text/x-wiki i37psk18qfc8am79i262ef8adyc55a7 2434810 2434477 2026-07-09T09:45:32Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-db' 2434810 wikitext text/x-wiki gsz7xkasul23fgu0r9m72cwrugh5ldx 2434811 2434810 2026-07-09T09:45:50Z Stashbot 7414 taavi: make cloudinfra-db-5 the new primary T402005 2434811 wikitext text/x-wiki pol4ta991m7j10u0wvt3jeehbvm77t9 2434814 2434811 2026-07-09T09:49:15Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-db' 2434814 wikitext text/x-wiki suxip4ytl6jbw0wb8lf1zh1aadywbam 2434842 2434814 2026-07-09T09:58:49Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on cloudinfra-db-6.cloudinfra.eqiad1.wikimedia.cloud 2434842 wikitext text/x-wiki hc0rkuvi62orpduv54cqotkptx157y7 2434846 2434842 2026-07-09T10:01:34Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on cloudinfra-db-6.cloudinfra.eqiad1.wikimedia.cloud 2434846 wikitext text/x-wiki 2ovz4sks1opj3ififkblgx1ogs1e7a2 2434852 2434846 2026-07-09T10:12:45Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance syslog-server-audit01 2434852 wikitext text/x-wiki 3sbnv0g9esd0345yeh924ixluv8tl9a 2434853 2434852 2026-07-09T10:13:02Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance syslog-server-audit01 2434853 wikitext text/x-wiki e9bgshkz97phgqwfxsmkf31g2os6f08 2434854 2434853 2026-07-09T10:13:07Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance syslog-server-audit02 2434854 wikitext text/x-wiki e0c7s6nuylg1bfdf9po6xj4j9ah70mi 2434855 2434854 2026-07-09T10:13:54Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance syslog-server-audit02 2434855 wikitext text/x-wiki fbv6914a3befpa6us83wbefawzf3h1n 2434860 2434855 2026-07-09T10:16:48Z Stashbot 7414 taavi: Replace cloudinfra-db03 with cloudinfra-db-6 replicating off of db-5 T402005 2434860 wikitext text/x-wiki snte85yhvur369l3johw9amt60fhubq 0 444677 2434700 2426568 2026-07-09T02:49:31Z RLazarus (WMF) 15215 remove port 4250 (was ipoid, decommissioned, T416623) 2434700 wikitext text/x-wiki dpa1xah066mjmrr2ylswmid6dvbf4jy 0 447310 2434605 2430095 2026-07-08T20:51:42Z MPopov (WMF) 5891 /* Templates */ New templates for experimentation 2434605 wikitext text/x-wiki 29a5ngogf86ttecs21bwqxm7544ff85 2434606 2434605 2026-07-08T20:53:57Z MPopov (WMF) 5891 /* Templates */ Minor section renames 2434606 wikitext text/x-wiki rnzq1enhr218f3bz5rntk2z3uslzpvi 2434621 2434606 2026-07-08T21:32:09Z MPopov (WMF) 5891 /* Meta */ Link to why we experiment brief 2434621 wikitext text/x-wiki 8yp4d85xsj495u5zju8xmf1pe6e6bd8 2434677 2434621 2026-07-08T23:10:11Z MPopov (WMF) 5891 /* Experimentation templates */ Link to GDoc template for experiment planning and update epic task template to v1.2 2434677 wikitext text/x-wiki 63ksfwo5cg2ob4r0a5pdrfcjk9ak5ee 0 447761 2434669 2420447 2026-07-08T22:34:01Z Brennen 54136 /* Restarting */ Add hashar's notes re: restart procedure 2434669 wikitext text/x-wiki o5dffs3dna0qw042ks6ko9xvhv1aedj 0 448732 2434703 2428633 2026-07-09T03:05:28Z LSandergreen-WMF 46345 /* Aborting! Previous process (XXX) is still alive. Remove lockfile manually if in error: /tmp/<jobname>.lock */ how to kill jobs 2434703 wikitext text/x-wiki dnua4r1x6wu95uvuvq81to898vwvqj0 0 448997 2434592 2421776 2026-07-08T20:06:56Z WFan (WMF) 38087 2434592 wikitext text/x-wiki qxj9llluc5zf47ps63kn7s3525j2sfs 2434595 2434592 2026-07-08T20:16:01Z WFan (WMF) 38087 2434595 wikitext text/x-wiki fovo5wdl1bz6pjufzfzb7kkql02vzgo 0 449160 2434686 2434006 2026-07-09T00:00:47Z Dexbot 30554 Bot: Updating the report 2434686 wikitext text/x-wiki evmkyj96u2v8ck610qxfbanj6nbxuhd 0 450971 2434895 2236965 2026-07-09T10:52:57Z FGiunchedi (WMF) 39205 /* Cloudvirt down */ remove mentions of maintenance aggregate 2434895 wikitext text/x-wiki rj7hsm4hcp7bivrjdpp8f2238jrxkpo 116 453426 2434688 2432034 2026-07-09T00:26:02Z Phabbanbot 37210 VVMFOfffce was disabled by SomeRandomDeveloper 2434688 wikitext text/x-wiki b1y6cos033k95exgyvfx5orv2rape29 0 454559 2434733 2430224 2026-07-09T08:01:51Z MMuhlenhoff (WMF) 5268 /* How to roll out kernel updates */ Add a note on how to install the Linux updates 2434733 wikitext text/x-wiki 7airt6d6djeuifvtnj5j9h74ui16n1f 2434734 2434733 2026-07-09T08:02:08Z MMuhlenhoff (WMF) 5268 /* How to roll out kernel updates */ typo 2434734 wikitext text/x-wiki ltdq4i338kraqo6jiibd3piqglwcwzp 498 459134 2434543 2333904 2026-07-08T19:17:55Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.vps.create_project for project magnum in codfw1dev 2434543 wikitext text/x-wiki q9kcn0kbpdbml72uiey18b6sjrjqn90 2434545 2434543 2026-07-08T19:18:50Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project magnum in codfw1dev 2434545 wikitext text/x-wiki pdzfr51h1fhnij1ym0hfetw9nwqacy4 2434554 2434545 2026-07-08T19:19:44Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.vps.create_project for project magnum in codfw1dev 2434554 wikitext text/x-wiki 0mpkkol7dtki9x6q267kwdunnc63ily 2434555 2434554 2026-07-08T19:19:50Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project magnum in codfw1dev 2434555 wikitext text/x-wiki ttublsqwzzdxb4y3zay71psa1gcozg7 2434560 2434555 2026-07-08T19:20:36Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.vps.create_project for project magnum in codfw1dev 2434560 wikitext text/x-wiki rtqh4j7dijnrzvhj5rgvlo6vngihm4t 2434563 2434560 2026-07-08T19:23:01Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project magnum in codfw1dev 2434563 wikitext text/x-wiki rcmcrd8hfx2b88jjlqrbrzwrlzusv4l 0 459290 2434876 2433845 2026-07-09T10:27:23Z LToscano (WMF) 7191 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2434876 wikitext text/x-wiki a34ccb7smgpxpwbyv1hxdhnikwea5ra 0 459699 2434251 2434025 2026-07-08T13:46:28Z Stashbot 7414 jgleeson: payments-wiki upgraded from f34901d7 to 196c7bc1 2434251 wikitext text/x-wiki 73sjrbfxtam2jrbzo5i7oc0a4osri3w 2434584 2434251 2026-07-08T19:50:11Z Stashbot 7414 jgleeson: payments-wiki upgraded from 196c7bc1 to 5d2bb8e8 2434584 wikitext text/x-wiki h0qd5aozokm356x4xv9zi5fgemmmavn 2434651 2434584 2026-07-08T22:13:43Z Stashbot 7414 larssandergreen: donorwiki upgraded from c2cdbf24 to d59684ed 2434651 wikitext text/x-wiki bulskrvmgmyudv40jkv2db0nu6x2hfi 2434655 2434651 2026-07-08T22:15:54Z Stashbot 7414 larssandergreen: civicrm upgraded from ead61459 to f4ff79c8 2434655 wikitext text/x-wiki 8nqkmqtyijyl81kzdq56v9ftogceh5e 2434684 2434655 2026-07-08T23:52:17Z Stashbot 7414 eileen: SmashPig upgraded from d8393666 to 2042f7f7 2434684 wikitext text/x-wiki hp8bn68rvlrsjhdfnkwvpvmcau4shwa 2434687 2434684 2026-07-09T00:01:56Z Stashbot 7414 eileen: onfig 1c59ae42 -> fa79267c 2434687 wikitext text/x-wiki e55cw6ubre37iie0qqzkpfdp7frzibd 2434692 2434687 2026-07-09T02:12:53Z Stashbot 7414 eileen: SmashPig upgraded from 2042f7f7 to 87dfbd6b 2434692 wikitext text/x-wiki qdpdevn67w1itj5bapsh8yhz6yim293 2434693 2434692 2026-07-09T02:28:05Z Stashbot 7414 eileen: civicrm upgraded from f4ff79c8 to 3a2952ff 2434693 wikitext text/x-wiki ig86cr01jqt2pg6v9xh4wsgetzapic7 2434705 2434693 2026-07-09T03:12:55Z Stashbot 7414 eileen: civicrm upgraded from 3a2952ff to 3a1a0891 2434705 wikitext text/x-wiki ck8o852ztd519z6e1nsgdkeokk2er4t 2434718 2434705 2026-07-09T05:02:59Z Stashbot 7414 eileen: civicrm upgraded from 3a1a0891 to 20b3dabe 2434718 wikitext text/x-wiki mrkxmn9wat9foxfy1vqlfwamwbupqp7 2 460039 2434761 2428640 2026-07-09T08:34:41Z Atsuko-WMF 54082 /* Tools/tricks */ 2434761 wikitext text/x-wiki bq3qb0sv8jgidonfmg0tej0ucameyc5 0 460375 2434478 2433209 2026-07-08T18:09:55Z AWesterinen-WMF 54189 /* POINT Decomposition */ 2434478 wikitext text/x-wiki 1lcu6prheeoa5x4qnout1m61344lp9j 2434713 2434478 2026-07-09T03:43:54Z AWesterinen-WMF 54189 /* Example */ 2434713 wikitext text/x-wiki chu8tyddretciy7zu8q33npr25pnv6v 0 460376 2434694 2433465 2026-07-09T02:28:08Z AWesterinen-WMF 54189 /* wikibase:label */ 2434694 wikitext text/x-wiki 6rx1a1nx6c52220i7f4i0pxhvlngzav 2434695 2434694 2026-07-09T02:33:14Z AWesterinen-WMF 54189 /* Query Rewrite Pattern for wikibase:label */ 2434695 wikitext text/x-wiki n3xgc9wczte4vdsrimkf6zw18rxgenc 2434696 2434695 2026-07-09T02:34:32Z AWesterinen-WMF 54189 /* Query Rewrite Pattern for wikibase:label */ 2434696 wikitext text/x-wiki k3m4pow7yfbhkyxmhc5jkgnhha4jo2c 2434697 2434696 2026-07-09T02:45:13Z AWesterinen-WMF 54189 /* Label Service Results for an Unbound Variable */ 2434697 wikitext text/x-wiki 2hobxux07nreo8so4gxlogppkkl4e4j 2434698 2434697 2026-07-09T02:45:35Z AWesterinen-WMF 54189 /* Label Service Results for an Unbound Variable */ 2434698 wikitext text/x-wiki 8ddlzg4o18v2b364lq4ucdzcquczzpk 2434699 2434698 2026-07-09T02:46:28Z AWesterinen-WMF 54189 /* Label Service Results for an Unbound Variable */ 2434699 wikitext text/x-wiki clm2n9jn2lqv3rq65nk315yvolwqi56 0 460388 2434626 2433336 2026-07-08T21:39:44Z AKhatun (WMF) 50461 /* Examples */ typo and wording fix 2434626 wikitext text/x-wiki al48ue3003rn0zzv4twbytybcy0afx0 1 460409 2434238 2433918 2026-07-08T13:38:12Z Udehb-WMF 40856 /* non-replacement of label service ? */ Reply 2434238 wikitext text/x-wiki dpmd8y58v9i4o5pxntbjt0qhjz4h3tf 2434249 2434238 2026-07-08T13:42:11Z Udehb-WMF 40856 /* named subqueries */ Reply 2434249 wikitext text/x-wiki q1pczph9edewbe5aqaok4iracvd84xo 2434258 2434249 2026-07-08T13:52:31Z Jheald 3726 /* named subqueries */ fix link 2434258 wikitext text/x-wiki g96l5eiy9l0wac14zh6o5f39qjxhviw 2434598 2434258 2026-07-08T20:29:12Z Jheald 3726 /* non-replacement of label service ? */ Reply 2434598 wikitext text/x-wiki a7jv2gv85z75i4dk5zh1fhx6f02kfox 2434607 2434598 2026-07-08T20:56:47Z Jheald 3726 + link to etherpad 2434607 wikitext text/x-wiki sys4z9bi58kureyaygyqj289j27glcy 2434610 2434607 2026-07-08T20:59:08Z Jheald 3726 /* non-replacement of label service ? */ 2434610 wikitext text/x-wiki dk02nl9k444nggur6ny54dhpoh11my6 2434612 2434610 2026-07-08T21:00:33Z Jheald 3726 2434612 wikitext text/x-wiki jx63t41y9455mygis0znxb5fiqd3g22 2434613 2434612 2026-07-08T21:01:23Z Jheald 3726 2434613 wikitext text/x-wiki ns1t0300gup012b3sawxen9obbj1g4g 2434689 2434613 2026-07-09T01:32:46Z AWesterinen-WMF 54189 /* burying the lede */ 2434689 wikitext text/x-wiki qu92iny8x8wmnta03ii5572k5kejqkp 2434701 2434689 2026-07-09T02:54:42Z AWesterinen-WMF 54189 /* incorrect and inefficient replacement for label service */ Reply 2434701 wikitext text/x-wiki bify7xup1pokffxz0d9s72fz753aa9j 2434702 2434701 2026-07-09T03:00:43Z AWesterinen-WMF 54189 /* non-replacement of label service ? */ Reply 2434702 wikitext text/x-wiki 262646y89hi8cyip70zc321g8qkprp8 2434704 2434702 2026-07-09T03:07:13Z AWesterinen-WMF 54189 /* named subqueries */ Reply 2434704 wikitext text/x-wiki aay10op2dwgvprc73o76sbg014bc0h0 2434706 2434704 2026-07-09T03:24:55Z AWesterinen-WMF 54189 /* bd:slice and bd:sample */ Reply 2434706 wikitext text/x-wiki 9d9mieih2mgdl6w56stiqexypb8efew 2434711 2434706 2026-07-09T03:29:51Z AWesterinen-WMF 54189 /* bd:slice and bd:sample */ Reply 2434711 wikitext text/x-wiki sptwpvjqwm8wncvwykc2tz8ai1ifzzn 1 460410 2434712 2433159 2026-07-09T03:37:46Z AWesterinen-WMF 54189 /* Performance of wikibase:around replacement ? */ Reply 2434712 wikitext text/x-wiki nbx9fy303li76iboiqsdonrl5a0e97u 2434714 2434712 2026-07-09T03:49:08Z AWesterinen-WMF 54189 /* Getting latitude and longitude */ Reply 2434714 wikitext text/x-wiki l4eec15qbwdate0xpzuim7fjgfm37yu 2434716 2434714 2026-07-09T04:00:02Z AWesterinen-WMF 54189 /* Getting latitude and longitude */ 2434716 wikitext text/x-wiki r2ec3lxpvzmnhreco3c47d52hxotbp4 2 460427 2434469 2434036 2026-07-08T17:59:53Z ArielGlenn 11 /* Logstash */ add an intro howto paragraph, convert all links to short urls 2434469 wikitext text/x-wiki ef3hc5dmm0clwa07qo1zwv8wj2lbtht 2434473 2434469 2026-07-08T18:06:51Z ArielGlenn 11 /* Grafana */ change all links to be for the last 2 days of coverage 2434473 wikitext text/x-wiki 9y8ov2sy82zcnul14ipauh09r3ku3c6 2434474 2434473 2026-07-08T18:07:36Z ArielGlenn 11 /* MediaWiki API Platform Team Chore Rotation= */ 2434474 wikitext text/x-wiki tvoudb5h4qagsj9ip67wh3970oyu8od 2434475 2434474 2026-07-08T18:08:10Z ArielGlenn 11 /* team workboard in phabricator:= */ 2434475 wikitext text/x-wiki 71rbfw6dwguk4t0eb5prff3rewng2et 2434483 2434475 2026-07-08T18:15:14Z ArielGlenn 11 /* MediaWiki API Platform Team Chore Rotation */ menu entries to get you to the place to add calendars via webcal links 2434483 wikitext text/x-wiki q4o62tt16uzikmezafot6e0osmmkj2a 2434504 2434483 2026-07-08T18:42:23Z ArielGlenn 11 /* Grafana */ made a vast overgeneralization in the intro paragraph, added a note on higher traffic modules 2434504 wikitext text/x-wiki dxhdfhmwx9p6p7d8awiy2vaxm2h5wzt 2434505 2434504 2026-07-08T18:43:20Z ArielGlenn 11 /* IRC */ remove redundant comments in favour of the intro sentence 2434505 wikitext text/x-wiki plo15h9bwh59innia0dfwqbhnewdsu9 2434507 2434505 2026-07-08T18:44:39Z ArielGlenn 11 /* MediaWiki API Platform Team Chore Rotation */ avoid overloading the term "tasks" 2434507 wikitext text/x-wiki fehilkrnm5whvjh28dz34ppbt8ksi4l 2434508 2434507 2026-07-08T18:45:43Z ArielGlenn 11 /* GitLab repos */ flag where we're missing info 2434508 wikitext text/x-wiki 4z6jv46py8tfjhj3duk607uyaigpmy9 2434513 2434508 2026-07-08T18:50:21Z ArielGlenn 11 /* Mailing lists */ add en wp village pump to the list of things to check 2434513 wikitext text/x-wiki e5qrinvfw90n3mj7bwr1usc51o090wd 2434523 2434513 2026-07-08T19:04:24Z ArielGlenn 11 /* IRC */ expand a little on participation in the channels 2434523 wikitext text/x-wiki fw57m1rt0wulo91sp7bkcr5gg7yurtf 2434525 2434523 2026-07-08T19:06:50Z ArielGlenn 11 /* Mailing lists/other */ make it clear that people can jump in to answer questions if they've got the info. 2434525 wikitext text/x-wiki mu8odshq4a6dausjlaspz0faa5dkqpa 2434561 2434525 2026-07-08T19:21:09Z ArielGlenn 11 /* Logstash */ what to do with new errors, heh 2434561 wikitext text/x-wiki mrrubatcvez2yp4ba143gd1gcau4waa 498 460430 2434160 2026-07-08T12:24:20Z Labslogbot 55 Auto update of instance info. 2434160 wikitext text/x-wiki khb5l438jlzoozliubtbdf1ow09jyq7 0 460431 2434879 2026-07-09T10:30:48Z Taavi 13997 Redirected page to [[Portal:Cloud VPS/Admin/Cloudinfra#cloudinfra-db]] 2434879 wikitext text/x-wiki 2lzu6id9cj1axkkxigf68y95vb7v3vj