Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.10 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2435756 2435682 2026-07-13T12:21:38Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api 2435756 wikitext text/x-wiki 78qfjafbqlttwa1pijorwkdxrofchwg 2435759 2435756 2026-07-13T12:27:03Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api 2435759 wikitext text/x-wiki b24mtcvm9fwgb8hqcfee8ww39m3alkx 2435790 2435759 2026-07-13T12:53:59Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2435790 wikitext text/x-wiki 27wsm3liaexslab3dydo9mljo90pzae 2435791 2435790 2026-07-13T12:59:23Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2435791 wikitext text/x-wiki eglgr00sm5fkxcnxoduqafi7g6wbura 2435815 2435791 2026-07-13T13:49:19Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-8.tools.eqiad1.wikimedia.cloud 2435815 wikitext text/x-wiki h7zz7tp0pa72skjo7t3s9f81g3pwt78 2435816 2435815 2026-07-13T13:50:47Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-8.tools.eqiad1.wikimedia.cloud 2435816 wikitext text/x-wiki lndpxuc0v0am480u66x27us1n1nfrbq 2435817 2435816 2026-07-13T13:51:59Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435817 wikitext text/x-wiki fk1suw65tot4ko4fl0270zziem3wytw 2435831 2435817 2026-07-13T14:05:24Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435831 wikitext text/x-wiki 8hnjd28jznc47omi1os5l71x65955p1 2435839 2435831 2026-07-13T14:26:42Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435839 wikitext text/x-wiki 47nuz4qtodf13lxaqp79p7hckilsmmx 2435850 2435839 2026-07-13T14:40:34Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435850 wikitext text/x-wiki 3ofncfx7z3lond9avb29winoplndfi3 2435851 2435850 2026-07-13T14:42:10Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud 2435851 wikitext text/x-wiki 0hex59xh7yeth47jv1c55rgp3kywu0g 2435852 2435851 2026-07-13T14:43:51Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud 2435852 wikitext text/x-wiki smyu1itrwqqwg8z72w5qdjr4mruooa4 2435903 2435852 2026-07-13T16:36:19Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435903 wikitext text/x-wiki cujwcuayniunjhtn17r19yu580gf8gy 2435913 2435903 2026-07-13T16:49:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435913 wikitext text/x-wiki 1ez1b3ohy6q924qkcdlw42mxny9208m 2435934 2435913 2026-07-13T18:46:35Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-cumin' 2435934 wikitext text/x-wiki bdj81n0642e749ofai784a66uo5188v 2435935 2435934 2026-07-13T18:53:55Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-cumin' 2435935 wikitext text/x-wiki gll362d6v5a6wb7fy4nh3uzhaixorjz 2435936 2435935 2026-07-13T19:01:09Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-cumin-2.tools.eqiad1.wikimedia.cloud 2435936 wikitext text/x-wiki gcfteszxyggw0mi09epcwe3o1mgbj2h 2435937 2435936 2026-07-13T19:05:13Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-cumin-2.tools.eqiad1.wikimedia.cloud 2435937 wikitext text/x-wiki rbgj0okv1gtjfqh17wps3qkg3yd1lvf 2435983 2435937 2026-07-14T00:29:18Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435983 wikitext text/x-wiki 6xrieaamosonecmtn5ylwkv754o6jk0 2435984 2435983 2026-07-14T00:42:20Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435984 wikitext text/x-wiki fcg4mtdp3wvwflzrwp64zs0d0wjcevq 2435987 2435984 2026-07-14T01:06:13Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435987 wikitext text/x-wiki heoi5ibymnmcy4dlo7ir16c8qjrhskz 2435988 2435987 2026-07-14T01:18:13Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435988 wikitext text/x-wiki qmf6hyp0io9g8bem04hb66idabkkhp3 2436219 2435988 2026-07-14T11:13:53Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-cumin-1 2436219 wikitext text/x-wiki 88wbfm5h1dljaygxlr7wlv7xc0m0tox 2436220 2436219 2026-07-14T11:14:17Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-cumin-1 2436220 wikitext text/x-wiki e0si071j7p2df34o335ubr3qxg01d9u 2436235 2436220 2026-07-14T11:31:27Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-7 2436235 wikitext text/x-wiki k5ze0ie75v7y8sxporwhil12u2m38e7 2436237 2436235 2026-07-14T11:32:24Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-7 2436237 wikitext text/x-wiki s3gpgw7qch3yjq9p1wxtg5jnu9b17u4 2436239 2436237 2026-07-14T11:34:16Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-6 2436239 wikitext text/x-wiki 9vf134cb6np9y7iex9kba76grljpao2 2436240 2436239 2026-07-14T11:35:12Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-6 2436240 wikitext text/x-wiki m99pmtnum0r3h5ot4bm4hiaxgf23joh 2436247 2436240 2026-07-14T11:38:58Z Stashbot 7414 taavi: failing over redis to new trixie cluster T360704 2436247 wikitext text/x-wiki r9w30rj3u4m5w8tceozf02w3iwklr6i 2436254 2436247 2026-07-14T11:43:55Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud 2436254 wikitext text/x-wiki p39xxd0nalg6f5a5t06vn6u7d9z1g59 2436255 2436254 2026-07-14T11:44:00Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-10.tools.eqiad1.wikimedia.cloud 2436255 wikitext text/x-wiki a5y9bltplqs8jahlpzdj8n2tuvnr9un 2436258 2436255 2026-07-14T11:45:43Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-10.tools.eqiad1.wikimedia.cloud 2436258 wikitext text/x-wiki k1w31ofjmpf2x9eveyauzyj7d9m80gl 2436259 2436258 2026-07-14T11:46:01Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud 2436259 wikitext text/x-wiki lv7px7np9vx5w0zmi2mvwlrwk4kl2pb 0 4108 2435747 2435732 2026-07-13T12:11:43Z Msz2001 16141 /* {{Deployment_day|date=2026-07-13}} */ deployed now, removing patch from window 2435747 wikitext text/x-wiki 0bnnerhifp6w0nfz8i2blgrf9yxgvwr 2435775 2435747 2026-07-13T12:43:35Z WBrown (WMF) 37696 /* {{Deployment_day|date=2026-07-13}} */ done 2435775 wikitext text/x-wiki 6j0398fdggpfnsedm7gtdtc6pmyynth 2435925 2435775 2026-07-13T17:22:05Z ScheduleDeploymentBot 37566 Add [[gerrit:1212157]] to Monday, July 13 UTC late backport window 2435925 wikitext text/x-wiki q0itfcfvr7gej03vd626u2cqji2gr9f 2435938 2435925 2026-07-13T19:16:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1310161]] to Monday, July 13 UTC late backport window 2435938 wikitext text/x-wiki qhunrmsr42v8vkx7coviuu8kopnck30 2435940 2435938 2026-07-13T19:55:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1304630]] to Monday, July 13 UTC late backport window 2435940 wikitext text/x-wiki bs50tc7a4bn782ffr6qwma2907o96l9 498 6726 2435748 2435735 2026-07-13T12:15:03Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api 2435748 wikitext text/x-wiki aw9yb9vi3io20vija8dk3lxdjd9316i 2435754 2435748 2026-07-13T12:19:40Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api 2435754 wikitext text/x-wiki hb4r340fb3p2txjhzopw4n87pbg0960 2435761 2435754 2026-07-13T12:31:53Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud 2435761 wikitext text/x-wiki 4jwdzrux0jxjihp9gqth8833i9m8s43 2435762 2435761 2026-07-13T12:34:07Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud 2435762 wikitext text/x-wiki 0vkzlho8j6vjy1srg4w48h2j9u0ysuw 2435763 2435762 2026-07-13T12:34:27Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud 2435763 wikitext text/x-wiki argx7qy7ubdawjmn3gmmz8c9kbo590m 2435764 2435763 2026-07-13T12:35:20Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud 2435764 wikitext text/x-wiki 4z4ngq9y76akiouryywi51zbub5pvo6 2435765 2435764 2026-07-13T12:35:22Z Stashbot 7414 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud 2435765 wikitext text/x-wiki qlj9mk5ub9c2rrixatczbecb1658h8r 2435766 2435765 2026-07-13T12:35:41Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud 2435766 wikitext text/x-wiki 90o18g56mvh8mnv2bcxpryhwp87yog9 2435769 2435766 2026-07-13T12:37:51Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2435769 wikitext text/x-wiki etsm3t0uhh67z63obr22quu4mwjz8eg 2435770 2435769 2026-07-13T12:37:59Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud 2435770 wikitext text/x-wiki 2t9km3fgv1q335znrs522t570ckhzl1 2435771 2435770 2026-07-13T12:38:17Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud 2435771 wikitext text/x-wiki hm2sp829bio966dkh8fw9ltplywjxvh 2435773 2435771 2026-07-13T12:42:36Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2435773 wikitext text/x-wiki r2s9n8w6jngycgnsi241pliphop0ugm 2435807 2435773 2026-07-13T13:31:49Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435807 wikitext text/x-wiki dazi8u8piwzxwpgxxxdrj28xau1sfn2 2435810 2435807 2026-07-13T13:44:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435810 wikitext text/x-wiki qc3avf6h92fqfsjrajvnd4u5x9jxtet 2435833 2435810 2026-07-13T14:13:25Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435833 wikitext text/x-wiki 64cf65ocy4e8muyfuys8w3t725urefd 2435838 2435833 2026-07-13T14:25:17Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435838 wikitext text/x-wiki 67wb1m7kl7j9y3w6a6jq0z0l8g64llq 2435885 2435838 2026-07-13T16:23:51Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435885 wikitext text/x-wiki 3vwe0hlajwnh51szt4omn0oy4c6m1or 2435902 2435885 2026-07-13T16:36:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435902 wikitext text/x-wiki hi6zibi9w4ejd1omo7y85rc0hvn1pf8 2435954 2435902 2026-07-13T21:43:16Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435954 wikitext text/x-wiki p87kp7wr0zbmte74g2gxnzeycb7q4m1 2435955 2435954 2026-07-13T21:47:05Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2435955 wikitext text/x-wiki 9ijnqaxod23e4vj5ayq9hi6ri4f14x8 2435956 2435955 2026-07-13T21:47:25Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435956 wikitext text/x-wiki 1yedy84un8w5vu9613eccs4mgnztfmw 2435957 2435956 2026-07-13T21:50:45Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2435957 wikitext text/x-wiki 135w37vhveowniatg752anoekyuan92 2435958 2435957 2026-07-13T21:56:00Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435958 wikitext text/x-wiki phdqbhiz2kojb4azr3qdkgm8f42gwl2 2435959 2435958 2026-07-13T21:59:19Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2435959 wikitext text/x-wiki mtgfddvzgzn3mvb9mcgene9ogkjeayg 2435980 2435959 2026-07-14T00:10:39Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435980 wikitext text/x-wiki a2l8qamfay40a3kj9q6gbhnzvic0uiz 2435982 2435980 2026-07-14T00:21:13Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435982 wikitext text/x-wiki mwoip3go0du4n92q1jrejkomyzjtvz4 2435985 2435982 2026-07-14T00:53:30Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2435985 wikitext text/x-wiki 9jwheqsly25qruubc33ryiyigj2ra11 2435986 2435985 2026-07-14T01:04:19Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2435986 wikitext text/x-wiki e0upll70vtjazcn377phk97ddvbw1gd 2436046 2435986 2026-07-14T07:44:23Z Stashbot 7414 godog: set 'dumps_use_nfs_lb: true' for toolsbeta-test-k8s-worker-nfs prefix 2436046 wikitext text/x-wiki edbt0g02nieog8663ysmwokf8y6fpfb 2436066 2436046 2026-07-14T07:58:26Z Stashbot 7414 godog: test dumps-nfs switchover on toolsbeta-test-k8s-worker-nfs-8 2436066 wikitext text/x-wiki h1wb9vshoaxykkhu1clkgiclbac1aoq 2436095 2436066 2026-07-14T08:22:25Z Stashbot 7414 godog: test reboot all workers 2436095 wikitext text/x-wiki al4mez3jbc8mq9xhwbo4a5djlmdoidx 2436120 2436095 2026-07-14T08:49:46Z Stashbot 7414 godog: set 'dumps_use_nfs_lb: true' for mail, bastion, static prefixes 2436120 wikitext text/x-wiki dxztmpfanburvrppffuaekbdiknp070 2436178 2436120 2026-07-14T10:24:37Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-cumin' 2436178 wikitext text/x-wiki a4pn1o56o4vtkrwsdpt8f1prjn703p8 2436192 2436178 2026-07-14T10:30:48Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-cumin' 2436192 wikitext text/x-wiki 4hbsvsx0flkm6w3uj0lfs7gcetv6c1t 2436214 2436192 2026-07-14T11:11:24Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-cumin-1 2436214 wikitext text/x-wiki eufv2d0qyizgqxbf5k50b872z62k729 2436215 2436214 2026-07-14T11:11:46Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-cumin-1 2436215 wikitext text/x-wiki tmantz97xdf0euz2955i67vfx4pee7f 2436222 2436215 2026-07-14T11:16:58Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-4 2436222 wikitext text/x-wiki 051m8utstxhbs1pl20jta0qrge2uq9k 2436223 2436222 2026-07-14T11:17:50Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-4 2436223 wikitext text/x-wiki ev7f9fgnelwe2n4zgmnhk234g5v6z99 2436224 2436223 2026-07-14T11:18:10Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-5 2436224 wikitext text/x-wiki dpwc3nvkwxn6mi8fxkk5pud8sxfmurp 2436225 2436224 2026-07-14T11:19:01Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-5 2436225 wikitext text/x-wiki t6irkjnlbyev6715douswaof3eyvjcg 2436226 2436225 2026-07-14T11:19:21Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-6 2436226 wikitext text/x-wiki 50zcn634xxn82o60y32dzuu6cm75zpp 2436230 2436226 2026-07-14T11:20:14Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-6 2436230 wikitext text/x-wiki rstiwqnzmrj94a6fu50kvyschepn1oc 0 7919 2435740 2435739 2026-07-13T12:00:20Z Stashbot 7414 zabe@deploy2003: Finished scap sync-world: Backport for [[gerrit:1310064|Do not apply elwiki abusefilter settings to dewiki (T431934)]], [[gerrit:1310061|etcd: Add support for x4 (T431989)]] (duration: 07m 37s) 2435740 wikitext text/x-wiki m61nujctw4b14qek1qmbl3ik4v8xd30 2435741 2435740 2026-07-13T12:01:33Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be2008.codfw.wmnet with OS trixie 2435741 wikitext text/x-wiki 430u5b1rfg4zizq8t9bl3e7800ye3jk 2435742 2435741 2026-07-13T12:03:55Z Stashbot 7414 mszwarc@deploy2003: Started scap sync-world: Backport for [[gerrit:1310027|SI: Fix client side instrumentation (T431977)]] 2435742 wikitext text/x-wiki 6hj2yvr8cmdu1952ix2s2x4volz05m6 2435743 2435742 2026-07-13T12:04:24Z Stashbot 7414 mszwarc@deploy2003: sync-world aborted: Backport for [[gerrit:1310027|SI: Fix client side instrumentation (T431977)]] (duration: 00m 29s) 2435743 wikitext text/x-wiki 449jg5l7chtedq5ksgndh9se7i3n56a 2435744 2435743 2026-07-13T12:07:52Z Stashbot 7414 mszwarc@deploy2003: Started scap sync-world: Backport for [[gerrit:1310058|Revert^2 "SI: Fix client side instrumentation" (T431977)]] 2435744 wikitext text/x-wiki 9m0eol6cr396ps19gf08saycfwxpxyc 2435745 2435744 2026-07-13T12:09:28Z Stashbot 7414 mszwarc@deploy2003: mszwarc: Backport for [[gerrit:1310058|Revert^2 "SI: Fix client side instrumentation" (T431977)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2435745 wikitext text/x-wiki tbbn3dt2a6bxeibkxgntqqq3amhbo06 2435746 2435745 2026-07-13T12:10:42Z Stashbot 7414 mszwarc@deploy2003: mszwarc: Continuing with deployment 2435746 wikitext text/x-wiki 6gerc14588nf3n719kwvhngx638gulm 2435749 2435746 2026-07-13T12:15:06Z Stashbot 7414 mszwarc@deploy2003: Finished scap sync-world: Backport for [[gerrit:1310058|Revert^2 "SI: Fix client side instrumentation" (T431977)]] (duration: 07m 14s) 2435749 wikitext text/x-wiki g96qlc2toz8iwzvei58y0xhtl5ukvpx 2435750 2435749 2026-07-13T12:16:30Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2435750 wikitext text/x-wiki dvuk171dt2g8yg74x0528bs8fvlkmt9 2435751 2435750 2026-07-13T12:17:01Z Stashbot 7414 atsuko@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2435751 wikitext text/x-wiki 36jrfms6psrc67koe65uzvfm40xa2rp 2435752 2435751 2026-07-13T12:17:06Z Stashbot 7414 atsuko@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2435752 wikitext text/x-wiki jdus89ordsnvrk8tswq1d0j430i2vkt 2435753 2435752 2026-07-13T12:19:35Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2435753 wikitext text/x-wiki 5ez9xqyw6yp4dhd94ptcffny3002qa1 2435755 2435753 2026-07-13T12:20:48Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2435755 wikitext text/x-wiki 8gjlqkcnjvu1h522mv9qriztyrbybf1 2435757 2435755 2026-07-13T12:22:51Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2008.codfw.wmnet with reason: host reimage 2435757 wikitext text/x-wiki 0hp5puxa6ytb7tq0u8ahoeql22m6hp8 2435758 2435757 2026-07-13T12:23:40Z Stashbot 7414 Msz2001: Deployed changes to private code for Suggested Investigations 2435758 wikitext text/x-wiki krjet87c4ofy6q98y9uzgdxu0tf8lku 2435760 2435758 2026-07-13T12:29:22Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2008.codfw.wmnet with reason: host reimage 2435760 wikitext text/x-wiki 4dj1aruzgwa331yg93el3oydn84k2wi 2435767 2435760 2026-07-13T12:36:11Z Stashbot 7414 dreamyjazz@deploy2003: Started scap sync-world: Backport for [[gerrit:1309684|ProductionServices: Drop NodeJS iPoid URL (T416623)]] 2435767 wikitext text/x-wiki q6d2i6heijvpypyg9omk8rkxp31mh8n 2435768 2435767 2026-07-13T12:37:49Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Backport for [[gerrit:1309684|ProductionServices: Drop NodeJS iPoid URL (T416623)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2435768 wikitext text/x-wiki fbbk7dz0a0vsc234unsx86atxpsvwv1 2435772 2435768 2026-07-13T12:38:54Z Stashbot 7414 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment 2435772 wikitext text/x-wiki 5k1xocg9wibl7vl6ur3gmpor2o1yl54 2435774 2435772 2026-07-13T12:43:12Z Stashbot 7414 dreamyjazz@deploy2003: Finished scap sync-world: Backport for [[gerrit:1309684|ProductionServices: Drop NodeJS iPoid URL (T416623)]] (duration: 07m 02s) 2435774 wikitext text/x-wiki ndltuo2q4uaaxsnxfjmqf9ab4v8nx8i 2435776 2435774 2026-07-13T12:44:15Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2435776 wikitext text/x-wiki m5wdtva4cfumr48ch6z0u48vh0u90nv 2435777 2435776 2026-07-13T12:45:18Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2435777 wikitext text/x-wiki 0gob2an7sdqjoyfplm6is2ij87xle1b 2435779 2435777 2026-07-13T12:45:38Z Stashbot 7414 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply 2435779 wikitext text/x-wiki leedqyepsy8e81nf22397nff1xpmbta 2435780 2435779 2026-07-13T12:45:52Z Stashbot 7414 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2435780 wikitext text/x-wiki 17l564qa5qjtijqotoozadi496jdjnn 2435781 2435780 2026-07-13T12:46:01Z Stashbot 7414 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2435781 wikitext text/x-wiki 9c2g8s3y9kpyigqogv8egvbbr2clckn 2435782 2435781 2026-07-13T12:47:19Z Stashbot 7414 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2435782 wikitext text/x-wiki d62ilidn4b6ac8salwghqfe7ysbaw20 2435783 2435782 2026-07-13T12:47:43Z Stashbot 7414 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply 2435783 wikitext text/x-wiki o27a5lflcpvuu20iqgvixk19ims4ath 2435784 2435783 2026-07-13T12:47:49Z Stashbot 7414 atsuko@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2435784 wikitext text/x-wiki kvfldvhhspzw19jumcl70gekh8tlpei 2435785 2435784 2026-07-13T12:47:59Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be2008.codfw.wmnet with OS trixie 2435785 wikitext text/x-wiki 58ppqh5c6mgeudl9q9cfatincplqgua 2435786 2435785 2026-07-13T12:48:09Z Stashbot 7414 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2435786 wikitext text/x-wiki 42goci3ourbo03y6o4lkcetrfg5k4dz 2435787 2435786 2026-07-13T12:48:12Z Stashbot 7414 atsuko@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2435787 wikitext text/x-wiki 75xo5t1r00f3eyex7szrv79r649jyf1 2435788 2435787 2026-07-13T12:51:21Z Stashbot 7414 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2435788 wikitext text/x-wiki kvkkcmiauqqgvohhygplgp1ley6ix1y 2435789 2435788 2026-07-13T12:52:44Z Stashbot 7414 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2435789 wikitext text/x-wiki do51xkurk6le9b7zhiugqoxmjxh5il1 2435792 2435789 2026-07-13T12:59:38Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2435792 wikitext text/x-wiki tekkpcf7jkkp3daa7gqh933ty70bzbu 2435793 2435792 2026-07-13T13:00:14Z Stashbot 7414 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2435793 wikitext text/x-wiki 1s89pmi0ytqqbfqnn01plx72ea1a17d 2435794 2435793 2026-07-13T13:01:09Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2435794 wikitext text/x-wiki owv4ikwcn5unrga6c4j4b5s1o7x8y5w 2435796 2435794 2026-07-13T13:01:57Z Stashbot 7414 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2435796 wikitext text/x-wiki is99rw30a07bycsxextz9qlf03egzkb 2435797 2435796 2026-07-13T13:17:41Z Stashbot 7414 aude@deploy2003: Started scap sync-world: Backport for [[gerrit:1310059|Enable ChartWizard on the beta cluster (T431990)]], [[gerrit:1308656|streams: webrequest - pageview - trending (T430675)]], [[gerrit:1307438|EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1 (T420883)]], [[gerrit:1309894|Remove nonexistent autopatrolled group from Outreach Wiki (T431959)]] 2435797 wikitext text/x-wiki 5e24oc7j9kibe4fx3xrbk5xapbfmok9 2435798 2435797 2026-07-13T13:19:21Z Stashbot 7414 aude@deploy2003: aikochou, javiermonton, aude, gkm563: Backport for [[gerrit:1310059|Enable ChartWizard on the beta cluster (T431990)]], [[gerrit:1308656|streams: webrequest - pageview - trending (T430675)]], [[gerrit:1307438|EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1 (T420883)]], [[gerrit:1309894|Remove nonexistent autopatrolled group from Outreach Wiki (T431959)]] synced to the testservers 2435798 wikitext text/x-wiki 8qt5wyqma3mgzztvugi30gcom8dmaau 2435799 2435798 2026-07-13T13:19:39Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2435799 wikitext text/x-wiki s80yudcwrtte5jpemg47u8n190yls7b 2435800 2435799 2026-07-13T13:22:13Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2435800 wikitext text/x-wiki lzi5l2ybuwj70yjqx6lsgsa46tfs2k0 2435801 2435800 2026-07-13T13:22:39Z Stashbot 7414 aude@deploy2003: aikochou, javiermonton, aude, gkm563: Continuing with deployment 2435801 wikitext text/x-wiki am8ofh6hw8zb74xlw9c9mybii2wcmd3 2435802 2435801 2026-07-13T13:23:54Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2435802 wikitext text/x-wiki tpan3ewfz4crb9iq6melx0xavw2g2d4 2435804 2435802 2026-07-13T13:28:54Z Stashbot 7414 aude@deploy2003: Finished scap sync-world: Backport for [[gerrit:1310059|Enable ChartWizard on the beta cluster (T431990)]], [[gerrit:1308656|streams: webrequest - pageview - trending (T430675)]], [[gerrit:1307438|EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1 (T420883)]], [[gerrit:1309894|Remove nonexistent autopatrolled group from Outreach Wiki (T431959)]] (duration: 11m 12s) 2435804 wikitext text/x-wiki cfca3rfnzcnjndq5bv2l4gih4fh1pqf 2435805 2435804 2026-07-13T13:30:25Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1070.eqiad.wmnet with reason: vacuum overlarge container dbs 2435805 wikitext text/x-wiki 4h9hfzjk61rmvyf79n0h8m120fudv9g 2435806 2435805 2026-07-13T13:31:17Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host thanos-be2009.codfw.wmnet with OS trixie 2435806 wikitext text/x-wiki fd4fc4wolk7b8w4oparwjdosrjv7thb 2435808 2435806 2026-07-13T13:33:50Z Stashbot 7414 gmodena@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2435808 wikitext text/x-wiki cs688nz54p10jjiio1049j382dqv279 2435809 2435808 2026-07-13T13:40:41Z Stashbot 7414 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2435809 wikitext text/x-wiki fo0z1w99d14b891iudse1up3wz68bvc 2435811 2435809 2026-07-13T13:47:46Z Stashbot 7414 rscout@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2435811 wikitext text/x-wiki 5wuu8nzse88s4nrwit2mibwteuidicy 2435812 2435811 2026-07-13T13:48:07Z Stashbot 7414 rscout@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2435812 wikitext text/x-wiki 0kssqvv49n7ingo4rtlfex4pfzljmoe 2435813 2435812 2026-07-13T13:48:22Z Stashbot 7414 rscout@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2435813 wikitext text/x-wiki fz4c4lhr6fnoddf8ca5h0hv7pe307e1 2435814 2435813 2026-07-13T13:48:44Z Stashbot 7414 rscout@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2435814 wikitext text/x-wiki bgfqqtvenvshtdtl566xszgy7ggu8iz 2435818 2435814 2026-07-13T13:52:38Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2009.codfw.wmnet with reason: host reimage 2435818 wikitext text/x-wiki k5loifj8i6vtnledklyf86rcwcjhpgo 2435821 2435818 2026-07-13T13:57:48Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=no; selector: name=dns7002.* 2435821 wikitext text/x-wiki hofgpw7vwqp14e911vhvz03205lj97q 2435823 2435821 2026-07-13T13:58:24Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2009.codfw.wmnet with reason: host reimage 2435823 wikitext text/x-wiki g4z8oasworkfmipao422r3pcekdzxt6 2435825 2435823 2026-07-13T14:00:06Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.hosts.remove-downtime for ms-be1070.eqiad.wmnet 2435825 wikitext text/x-wiki gp3gnwxvzwcs75h16k1djdp3lc4qtow 2435826 2435825 2026-07-13T14:00:07Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1070.eqiad.wmnet 2435826 wikitext text/x-wiki 8i69mizu8xao13lxzzwjljaetgqvtjp 2435827 2435826 2026-07-13T14:00:28Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2435827 wikitext text/x-wiki 890isff2mku9vti112nsd5x1ifh6q89 2435828 2435827 2026-07-13T14:02:18Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2435828 wikitext text/x-wiki k1e2ckilghlk51zc9kr4lujfd9vnotj 2435829 2435828 2026-07-13T14:05:07Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS trixie 2435829 wikitext text/x-wiki c3zq6ysbytpglaj4nl3nuo3jkx2itdt 2435830 2435829 2026-07-13T14:05:21Z Stashbot 7414 swfrench-wmf: disable-puppet on A:cp for ATS config change - T428909 T431838 2435830 wikitext text/x-wiki ggcq0d7cqk1n53ctqf6jbm6hi31h5ot 2435834 2435830 2026-07-13T14:14:40Z Stashbot 7414 swfrench-wmf: start rolling run-puppet-agent on A:cp for ATS config change - T428909 T431838 2435834 wikitext text/x-wiki 1rti8nvodhfb90cjle9jrtex21ru78h 2435835 2435834 2026-07-13T14:16:34Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be2009.codfw.wmnet with OS trixie 2435835 wikitext text/x-wiki toisqu1sctr3jt52weehxivf35mrcia 2435836 2435835 2026-07-13T14:18:02Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2435836 wikitext text/x-wiki lru3x4vocthgq4zk8qylckl98vbv8n6 2435837 2435836 2026-07-13T14:23:53Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1023.eqiad.wmnet with reason: reboot 2435837 wikitext text/x-wiki 6ptwvssqr4mu5emhsp4x92gzqvudc5r 2435840 2435837 2026-07-13T14:27:05Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2435840 wikitext text/x-wiki jwj2lpvadvvykzmjwhbg8shfefn12pq 2435841 2435840 2026-07-13T14:28:24Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2435841 wikitext text/x-wiki 5349db2e67gy7nfw0q89rxa3cj6zz7n 2435842 2435841 2026-07-13T14:28:55Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2435842 wikitext text/x-wiki 95gqm0zyxmroh3ukhe1ux1uu84ak74z 2435843 2435842 2026-07-13T14:29:54Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage 2435843 wikitext text/x-wiki rwbp1wsoqxjspzjqpm0wownx6fk83u7 2435845 2435843 2026-07-13T14:32:46Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2435845 wikitext text/x-wiki creoznohqbs0chm107lsum7nlgzgv1g 2435846 2435845 2026-07-13T14:33:27Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2435846 wikitext text/x-wiki baxpofuydpppkm6n2324nnauftcb9lc 2435847 2435846 2026-07-13T14:33:28Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2435847 wikitext text/x-wiki qz3m55mxaq70k5tmmz3xlf59sn7pz55 2435848 2435847 2026-07-13T14:34:04Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2435848 wikitext text/x-wiki 1gf66g3b50lhub2j6yb9upa88zm263s 2435849 2435848 2026-07-13T14:35:27Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage 2435849 wikitext text/x-wiki tuwpoy7hbohv73qfwd5t1fnrtstn4jp 2435854 2435849 2026-07-13T15:01:08Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2435854 wikitext text/x-wiki a5wlkl8tn358fq1u76i7yaozcmzilz8 2435855 2435854 2026-07-13T15:01:34Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2435855 wikitext text/x-wiki asm4uq2ylkj1czc06ppftj3y63mmd8i 2435857 2435855 2026-07-13T15:06:44Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2435857 wikitext text/x-wiki qefcbo2kuck3j3d9nrmrlz9uhe07ewm 2435858 2435857 2026-07-13T15:08:38Z Stashbot 7414 btullis@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2435858 wikitext text/x-wiki rdbfiaclo0ruq4uvylnfs8dn5to8cqf 2435860 2435858 2026-07-13T15:34:01Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1002.eqiad.wmnet 2435860 wikitext text/x-wiki q1epws54ct536g4f6a2best0zsa6xoi 2435861 2435860 2026-07-13T15:34:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1002.eqiad.wmnet 2435861 wikitext text/x-wiki qtow6o3z0li7dri4ey1uyoa198nflo3 2435862 2435861 2026-07-13T15:36:24Z Stashbot 7414 sukhe: restart pybal on lvs1020 2435862 wikitext text/x-wiki m0gd4m9d795rb7a5jmauz192zsylyrp 2435863 2435862 2026-07-13T15:40:09Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host netbox2003.codfw.wmnet 2435863 wikitext text/x-wiki fc6kyvgf6e16gsz5iapp5m7ji71gnfe 2435864 2435863 2026-07-13T15:40:10Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host netbox2003.codfw.wmnet 2435864 wikitext text/x-wiki ocs9gl787oc0f2b4wr2k8aaj5gtgfwt 2435865 2435864 2026-07-13T15:40:28Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host netbox1003.eqiad.wmnet 2435865 wikitext text/x-wiki 4zuxdziuna0zlawpg93xppcn97lfb2a 2435866 2435865 2026-07-13T15:40:29Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host netbox1003.eqiad.wmnet 2435866 wikitext text/x-wiki l0vj64ulacxk3bp3trodgp8uukp3s7n 2435867 2435866 2026-07-13T15:40:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host ganeti-test[2001-2003].codfw.wmnet 2435867 wikitext text/x-wiki 68mfq2750qtdrnasbo5kyddubems5pq 2435868 2435867 2026-07-13T15:41:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host ganeti-test[2001-2003].codfw.wmnet 2435868 wikitext text/x-wiki 2nx2kx3vyidgpuzkv02f5jfcdw0ingp 2435869 2435868 2026-07-13T15:43:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host kafka-logging1006.eqiad.wmnet 2435869 wikitext text/x-wiki ecvjs9s3r64d44c1ugw7f6oh9avx0ox 2435870 2435869 2026-07-13T15:43:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host kafka-logging1006.eqiad.wmnet 2435870 wikitext text/x-wiki 6akntitnlk98r6sd21msctj01c5qfv4 2435871 2435870 2026-07-13T15:46:47Z Stashbot 7414 aikochou@deploy2003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2435871 wikitext text/x-wiki j9nazr64qsepjx6z3w5qysrmrh1h7td 2435872 2435871 2026-07-13T15:50:25Z Stashbot 7414 atsukoito: restarting pybal on lvs1020 for https://gerrit.wikimedia.org/r/1310117 2435872 wikitext text/x-wiki ax5j00p29axcnh70vcnwa42h08smb6j 2435874 2435872 2026-07-13T15:55:48Z Stashbot 7414 aikochou@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2435874 wikitext text/x-wiki ksu3259vp8jinatpmv7928ax8f9ixdk 2435875 2435874 2026-07-13T15:59:52Z Stashbot 7414 atsukoito: restarting pybal on lvs1018 for https://gerrit.wikimedia.org/r/1310117 2435875 wikitext text/x-wiki 7qrj5q0q7gfq6amqfsanslw2jrbn2v2 2435877 2435875 2026-07-13T16:17:48Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync 2435877 wikitext text/x-wiki 3hy92wlir3r89yf2udcmdw721qrxf60 2435878 2435877 2026-07-13T16:18:14Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync 2435878 wikitext text/x-wiki cwnfp370ao16n5gotx5mhdybomnhwn0 2435879 2435878 2026-07-13T16:19:15Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync 2435879 wikitext text/x-wiki rawsl943icypci38ovpgxqggmetfev7 2435880 2435879 2026-07-13T16:19:38Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync 2435880 wikitext text/x-wiki 37pr8vdcnza2vn170pq0ndj9hxiqabl 2435881 2435880 2026-07-13T16:21:02Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: sync 2435881 wikitext text/x-wiki lar7bj0cy7l6zydxfgdkafnqs6cucl2 2435882 2435881 2026-07-13T16:21:33Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync 2435882 wikitext text/x-wiki 4fvxovb49kdqqkx4foz5h2zugq0z0jq 2435883 2435882 2026-07-13T16:22:00Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: sync 2435883 wikitext text/x-wiki qs6aroul97bbla3bxxpgd66bnsf5lky 2435884 2435883 2026-07-13T16:22:33Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync 2435884 wikitext text/x-wiki 80lbbkw3avfpjg68gqeimqx8xp4ayhs 2435886 2435884 2026-07-13T16:26:34Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: sync 2435886 wikitext text/x-wiki sxw11r59q3twmrctc8cujxw791goy02 2435887 2435886 2026-07-13T16:26:57Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: sync 2435887 wikitext text/x-wiki nrlr739yil75kd6opv42o4e1sim2g5u 2435888 2435887 2026-07-13T16:27:06Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: sync 2435888 wikitext text/x-wiki jowclto1l963lvcylpbilcx6x95qqzu 2435889 2435888 2026-07-13T16:27:27Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: sync 2435889 wikitext text/x-wiki g2iswo3jp54qcv97jzihljfyjzevark 2435890 2435889 2026-07-13T16:27:54Z Stashbot 7414 dancy@deploy2003: Installing scap version "4.273.0" for 159 host(s) 2435890 wikitext text/x-wiki i6wx3vjpnr8u25chehftkc737dou1gw 2435891 2435890 2026-07-13T16:29:50Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1065.eqiad.wmnet with reason: vacuum overlarge container dbs 2435891 wikitext text/x-wiki h1p0pzdpcnxk14iipwatzu7hxc5ekfn 2435892 2435891 2026-07-13T16:31:55Z Stashbot 7414 dancy@deploy2003: Installation of scap version "4.273.0" completed for 159 hosts 2435892 wikitext text/x-wiki 1earjthb8e84hthuyhko9fsklig99so 2435893 2435892 2026-07-13T16:33:30Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Depool test 2435893 wikitext text/x-wiki qb61p603wrc4upprn1exfolgl1jvq4j 2435894 2435893 2026-07-13T16:33:32Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2435894 wikitext text/x-wiki 7oqvk92w6mxnab2wrrq07u9jvxoci5p 2435895 2435894 2026-07-13T16:33:40Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2435895 wikitext text/x-wiki 0ke6ulozmjnqr8h9tvv4e6xk8ml12jf 2435896 2435895 2026-07-13T16:33:41Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Depool test 2435896 wikitext text/x-wiki 7gw5xtrmxgjlrfye3ohfx4bz7kn16dx 2435898 2435896 2026-07-13T16:34:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Pool test 2435898 wikitext text/x-wiki r3zzw87u6lvigm51xl1uzj6ee36qqi9 2435899 2435898 2026-07-13T16:34:35Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2435899 wikitext text/x-wiki 63tdfxi3c6vpa9wr54vaovk68l33793 2435900 2435899 2026-07-13T16:34:50Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2435900 wikitext text/x-wiki j6wyarrm5q9pchyp4ptmwuxwe69otnt 2435901 2435900 2026-07-13T16:34:51Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Pool test 2435901 wikitext text/x-wiki 6lvqabm5updm08vtcajaogdrhvte60z 2435906 2435901 2026-07-13T16:42:26Z Stashbot 7414 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model damaging --old (T431159) 2435906 wikitext text/x-wiki feujpf2tfbxwxz71qjzt4uc7upsrmxl 2435907 2435906 2026-07-13T16:42:35Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2435907 wikitext text/x-wiki l2kbpivo8ur4q9gnih13646j6651279 2435908 2435907 2026-07-13T16:42:39Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2435908 wikitext text/x-wiki b6tzfuco3m58kjzcspogkryhqw7lxuc 2435909 2435908 2026-07-13T16:42:40Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2435909 wikitext text/x-wiki 1l8eixkf8kvvgfjdwzoe1d4kza2flxl 2435910 2435909 2026-07-13T16:42:42Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2435910 wikitext text/x-wiki 413c49s7kmxysapzmo4jsvh4turofwd 2435911 2435910 2026-07-13T16:45:06Z Stashbot 7414 atsukoito: restarting pybal on lvs1019 to flush IP address for `cirrussearch1122.eqiad.wmnet` after moving the vlan T431311 2435911 wikitext text/x-wiki eiuhzjl5rc89g7nnaaxh74p02hbixb7 2435912 2435911 2026-07-13T16:47:32Z Stashbot 7414 dancy@deploy2003: Started scap sync-world: testing T428971 2435912 wikitext text/x-wiki rpmw9m5lwsjvfpse81jlgu4e36amzmz 2435915 2435912 2026-07-13T16:51:09Z Stashbot 7414 dancy@deploy2003: Finished scap sync-world: testing T428971 (duration: 03m 37s) 2435915 wikitext text/x-wiki 0qstdrkoge65iehvfkydc4vt2oagpba 2435916 2435915 2026-07-13T16:59:20Z Stashbot 7414 dzahn@dns1006: START - running authdns-update 2435916 wikitext text/x-wiki p3uv0mmmi5fhbqby4281xx9e0fy5oml 2435917 2435916 2026-07-13T17:01:14Z Stashbot 7414 dzahn@dns1006: END - running authdns-update 2435917 wikitext text/x-wiki b5u1ljdtz0mpgk6e0ycq56m1zbdlx5p 2435918 2435917 2026-07-13T17:04:28Z Stashbot 7414 dzahn@dns1006: START - running authdns-update 2435918 wikitext text/x-wiki 8bpitcztq782wt3dejyefylcibiw3cs 2435919 2435918 2026-07-13T17:06:21Z Stashbot 7414 dzahn@dns1006: END - running authdns-update 2435919 wikitext text/x-wiki n8qlmjlxomr06w2qjfojfw4xe2i5zfq 2435920 2435919 2026-07-13T17:07:20Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.hosts.remove-downtime for ms-be1065.eqiad.wmnet 2435920 wikitext text/x-wiki 840c2cuymb2mcrpuhylf5l1awluto7c 2435921 2435920 2026-07-13T17:07:21Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1065.eqiad.wmnet 2435921 wikitext text/x-wiki batyyxts5uen7m0tbukmnil61dub9z8 2435922 2435921 2026-07-13T17:13:02Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1071.eqiad.wmnet with reason: vacuum overlarge container dbs 2435922 wikitext text/x-wiki ikb3r0b4mc19qswbt5gz4nuxx7n0efe 2435926 2435922 2026-07-13T17:46:31Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.hosts.remove-downtime for ms-be1071.eqiad.wmnet 2435926 wikitext text/x-wiki o40yzirf29w0nzm3n0l9ed65itki3s0 2435927 2435926 2026-07-13T17:46:33Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1071.eqiad.wmnet 2435927 wikitext text/x-wiki mlv9zoiglt1i4ch6bzz0rr61lhuw88z 2435932 2435927 2026-07-13T18:19:58Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7002.wikimedia.org with OS trixie 2435932 wikitext text/x-wiki bb18p9h86xbmaontjpn4riv1eaaf2o3 2435933 2435932 2026-07-13T18:22:51Z Stashbot 7414 cwhite: lvextend vg0/srv +500g on centrallog hosts 2435933 wikitext text/x-wiki 5wctrcqqjtn65nvlbsgzu8k2k3oy1af 2435941 2435933 2026-07-13T20:04:01Z Stashbot 7414 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1212157|Enable DiscussionTools visual enhancements on enwiki (T409297)]], [[gerrit:1310161|Disable mobile "exit the editor" survey phase 2 (T426135)]], [[gerrit:1304630|Add source tab to ukwikisource's "Архів" (Archive) namespace (T53980)]] 2435941 wikitext text/x-wiki jqnr5w6b3ox2p61fw5l6uag2afh5qs0 2435942 2435941 2026-07-13T20:05:43Z Stashbot 7414 kemayo@deploy2003: soda, esanders, kemayo: Backport for [[gerrit:1212157|Enable DiscussionTools visual enhancements on enwiki (T409297)]], [[gerrit:1310161|Disable mobile "exit the editor" survey phase 2 (T426135)]], [[gerrit:1304630|Add source tab to ukwikisource's "Архів" (Archive) namespace (T53980)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there 2435942 wikitext text/x-wiki 0gm8g35lsckh2beug71360a7n1ruoa8 2435943 2435942 2026-07-13T20:07:56Z Stashbot 7414 kemayo@deploy2003: soda, esanders, kemayo: Continuing with deployment 2435943 wikitext text/x-wiki 79nwzmplgia3hhvfbz8clno8jp6lr2w 2435944 2435943 2026-07-13T20:12:24Z Stashbot 7414 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1212157|Enable DiscussionTools visual enhancements on enwiki (T409297)]], [[gerrit:1310161|Disable mobile "exit the editor" survey phase 2 (T426135)]], [[gerrit:1304630|Add source tab to ukwikisource's "Архів" (Archive) namespace (T53980)]] (duration: 08m 25s) 2435944 wikitext text/x-wiki iquobe40oxcmrdh96jvurqzqclb9mw8 2435945 2435944 2026-07-13T20:16:39Z Stashbot 7414 dancy@deploy2003: Installing scap version "4.274.0" for 3 host(s) 2435945 wikitext text/x-wiki d5zidzcd48cs6zgpc8i7xoupmw7s63w 2435946 2435945 2026-07-13T20:18:32Z Stashbot 7414 dancy@deploy2003: Installation of scap version "4.274.0" completed for 3 hosts 2435946 wikitext text/x-wiki 8j6e9irttuwnt2pco5olmwsbyy1rp5g 2435947 2435946 2026-07-13T20:19:33Z Stashbot 7414 dancy@deploy2003: Started scap sync-world: Testing T431635 2435947 wikitext text/x-wiki n18aoclpuwud1w8uf7uf4sbnbauoled 2435948 2435947 2026-07-13T20:23:09Z Stashbot 7414 dancy@deploy2003: Finished scap sync-world: Testing T431635 (duration: 03m 36s) 2435948 wikitext text/x-wiki dd7n2rusays8m9dfa83w9rkc1bfvq1y 2435949 2435948 2026-07-13T20:26:53Z Stashbot 7414 swfrench-wmf: reprepro include etcd-mirror_0.0.12-1+deb12u1 into main for bookworm-wikimedia - T428495 2435949 wikitext text/x-wiki cmrg7z0gtm1svqzkhgyup2zeqvivpy6 2435950 2435949 2026-07-13T20:28:45Z Stashbot 7414 swfrench-wmf: reprepro include etcd-mirror_0.0.12-1+deb13u1 into main for trixie-wikimedia - T424266 2435950 wikitext text/x-wiki hrqgox5uyi88zrtf7oe1sstxd1j4bj7 2435953 2435950 2026-07-13T21:18:24Z Stashbot 7414 maryum: Deployed security fix for T321092 2435953 wikitext text/x-wiki pha3xit11kkee98lk8e6e9b4mfbo2yo 2435964 2435953 2026-07-13T22:34:54Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1069.eqiad.wmnet with reason: vacuum overlarge container dbs 2435964 wikitext text/x-wiki ng7r2byv810qn5a5h3mxmy9iyxpii3r 2435967 2435964 2026-07-13T23:06:40Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.hosts.remove-downtime for ms-be1069.eqiad.wmnet 2435967 wikitext text/x-wiki sookfzrdqnrpueyg6b5boi59xdxcbbj 2435968 2435967 2026-07-13T23:06:41Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1069.eqiad.wmnet 2435968 wikitext text/x-wiki dj76wfwcxmqqo96xjfhrmdzu004ypto 2435969 2435968 2026-07-13T23:08:20Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1064.eqiad.wmnet with reason: vacuum overlarge container dbs 2435969 wikitext text/x-wiki odwmapz0so6yacxmwyc8dabb3ckh969 2435976 2435969 2026-07-13T23:33:45Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.hosts.remove-downtime for ms-be1064.eqiad.wmnet 2435976 wikitext text/x-wiki t3ji8c1i7seeplgk1xr80u0xfzkl4fr 2435977 2435976 2026-07-13T23:33:46Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1064.eqiad.wmnet 2435977 wikitext text/x-wiki 2riv64gig8t6z6l50iuii8s20mf4twi 2435990 2435977 2026-07-14T02:00:29Z Stashbot 7414 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image 2435990 wikitext text/x-wiki rpju1vpv5yl3nrkqg19moxef0ret48m 2435991 2435990 2026-07-14T02:06:59Z Stashbot 7414 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) 2435991 wikitext text/x-wiki r4320zinjc1b8ti3gf9nm7me1awtrc6 2435993 2435991 2026-07-14T03:03:10Z Stashbot 7414 mwpresync@deploy2003: Started scap sync-world: testwikis to 1.47.0-wmf.11 refs T430830 2435993 wikitext text/x-wiki n22gvcxunop7niyudnmoa3oy8rbh3gh 2435994 2435993 2026-07-14T03:39:11Z Stashbot 7414 mwpresync@deploy2003: Finished scap sync-world: testwikis to 1.47.0-wmf.11 refs T430830 (duration: 36m 01s) 2435994 wikitext text/x-wiki ip0l0db8ostgb2umtjts8xgbl227t9u 2435995 2435994 2026-07-14T04:01:09Z Stashbot 7414 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.8 (duration: 01m 07s) 2435995 wikitext text/x-wiki 4wcwgezokkm8jru3wl269a5guqbnxch 2435996 2435995 2026-07-14T05:07:29Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org 2435996 wikitext text/x-wiki kgog6fla2ss7hr84s1vkg924qz2lzbg 2435997 2435996 2026-07-14T05:13:47Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org 2435997 wikitext text/x-wiki opq54erbgn3zp18qufwc1sapzp6rm92 2435998 2435997 2026-07-14T05:24:34Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2435998 wikitext text/x-wiki cprq19gj0i5elr6znt8lv76pmp92jf7 2435999 2435998 2026-07-14T05:24:40Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2435999 wikitext text/x-wiki 2h4mpbsid0374vldskwxaustqrgk3j4 2436000 2435999 2026-07-14T05:26:32Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2436000 wikitext text/x-wiki 8wwdwc9fk8ytsf1mbowegxlqvrv2gk6 2436001 2436000 2026-07-14T05:39:47Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436001 wikitext text/x-wiki 0zvp028l3q19bthfq0zkt2gzt3eifbl 2436002 2436001 2026-07-14T05:40:22Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436002 wikitext text/x-wiki 30iqzwew8a1ezmhh1jdupwh0x50luho 2436003 2436002 2026-07-14T05:59:21Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436003 wikitext text/x-wiki ibdhukgz09yyixaylvrdwz15rfct8lq 2436004 2436003 2026-07-14T05:59:55Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436004 wikitext text/x-wiki jg8uzm89srdrwjt5r63kttnuiioijv4 2436005 2436004 2026-07-14T06:00:54Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync 2436005 wikitext text/x-wiki 66x9ri23yvy6jb4mru1ofx2plh1f1hj 2436006 2436005 2026-07-14T06:01:21Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync 2436006 wikitext text/x-wiki jc794hb7j7du9cjhjxd9q74wg9il9g1 2436007 2436006 2026-07-14T06:01:37Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync 2436007 wikitext text/x-wiki 3njq2zfnsjn5tg5r72jnlbjl1gkon42 2436008 2436007 2026-07-14T06:02:00Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync 2436008 wikitext text/x-wiki brfmm8k7mvs2u6bic5vbmesm0cdesfw 2436009 2436008 2026-07-14T06:03:18Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: sync 2436009 wikitext text/x-wiki 7racofoduaudv6wbxupbky93q8j4ewe 2436010 2436009 2026-07-14T06:03:51Z Stashbot 7414 javiermonton@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync 2436010 wikitext text/x-wiki ppyxpziaj12ulav3pzo48rqgw7ugo3h 2436011 2436010 2026-07-14T06:04:18Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: sync 2436011 wikitext text/x-wiki j8ekqz726iv5esf6vpa4zteeyk03gnr 2436012 2436011 2026-07-14T06:04:43Z Stashbot 7414 javiermonton@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync 2436012 wikitext text/x-wiki 0xgg4ycb0t3ku3yuo6fe6hc5nfnkbsq 2436013 2436012 2026-07-14T06:17:17Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1026.eqiad.wmnet with reason: reboot 2436013 wikitext text/x-wiki 8p5fhs2wslq7likrhfoa2rw6xrwveyd 2436014 2436013 2026-07-14T06:20:09Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2436014 wikitext text/x-wiki 5aaqjod16wjq4ygw8k8z842t7nxvq70 2436015 2436014 2026-07-14T06:22:02Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2436015 wikitext text/x-wiki 7y3e9p6kk79xpyjutb7xnipx75d0oot 2436016 2436015 2026-07-14T06:23:10Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2436016 wikitext text/x-wiki 7wp4vcraetk425fudp2jxbhkm8rlasz 2436017 2436016 2026-07-14T06:25:04Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2436017 wikitext text/x-wiki gdkznrhs0maffz34xbqmfr5b6hegffv 2436018 2436017 2026-07-14T06:44:10Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org 2436018 wikitext text/x-wiki 1ahsx70sdeomhacsulc85yad8an9gbv 2436019 2436018 2026-07-14T06:50:42Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org 2436019 wikitext text/x-wiki 9kystsw3mpmgzs5tg06h19f6z149d0p 2436020 2436019 2026-07-14T07:23:40Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436020 wikitext text/x-wiki p08qn9outm6zu2664m4xmxfoja6yl6h 2436021 2436020 2026-07-14T07:26:08Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436021 wikitext text/x-wiki 2hszhdqwt82w8h8dnk7a2eq5jkhvjn4 2436022 2436021 2026-07-14T07:26:19Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{aux-k8s-ctrl200*} and (A:aux-master-codfw or A:aux-worker-codfw) 2436022 wikitext text/x-wiki qtzvw35ao07izr2528i2sos3myncidh 2436023 2436022 2026-07-14T07:26:22Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet 2436023 wikitext text/x-wiki m7pdsk80ii84fpscg10mygnsy8618k0 2436024 2436023 2026-07-14T07:26:24Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet 2436024 wikitext text/x-wiki p6uxsdjezkt8ttd5bs22stti2hf0zqj 2436025 2436024 2026-07-14T07:28:16Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2024.codfw.wmnet with OS trixie 2436025 wikitext text/x-wiki sa4fks6dmsib76q9rbpv733jaq2cuwf 2436026 2436025 2026-07-14T07:29:41Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1024.eqiad.wmnet with OS trixie 2436026 wikitext text/x-wiki dznkmelswvieqquv6lzu869onr40hso 2436027 2436026 2026-07-14T07:31:20Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet 2436027 wikitext text/x-wiki 6unxtfa1kdotvstzhfa31rqadaogxxv 2436028 2436027 2026-07-14T07:31:22Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet 2436028 wikitext text/x-wiki f84v1c6zrrrfzu98ojmpc8j18w31nlr 2436029 2436028 2026-07-14T07:31:28Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet 2436029 wikitext text/x-wiki kvo65tq6otncwopi70usy5q2yty8up8 2436030 2436029 2026-07-14T07:31:30Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet 2436030 wikitext text/x-wiki ewrskfgi9bgux9ofdamydjffn2j0jsz 2436031 2436030 2026-07-14T07:34:52Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on P{config-master*} and (A:config-master or A:config-master-eqiad or A:config-master-codfw) 2436031 wikitext text/x-wiki 1bcnqrzzll9e0iaubw7fn4172im9eh5 2436032 2436031 2026-07-14T07:35:13Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors 2436032 wikitext text/x-wiki qnslrbmf9nrrmb234q1g4hk280cme8n 2436033 2436032 2026-07-14T07:35:17Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors 2436033 wikitext text/x-wiki 8rf6cfaha83zpykfqu5qj485rvaf3p5 2436034 2436033 2026-07-14T07:36:34Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet 2436034 wikitext text/x-wiki 3wba9e10li12s1d9yzgqgm5jpmu19mh 2436035 2436034 2026-07-14T07:36:36Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet 2436035 wikitext text/x-wiki kdh3y104il2ukgtjrcp6l3erqj8zaom 2436036 2436035 2026-07-14T07:36:37Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{aux-k8s-ctrl200*} and (A:aux-master-codfw or A:aux-worker-codfw) 2436036 wikitext text/x-wiki 5ubzkntf79n17ffr26a5btbs9lzmzke 2436037 2436036 2026-07-14T07:39:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{aux-k8s-worker2*} and (A:aux-master-codfw or A:aux-worker-codfw) 2436037 wikitext text/x-wiki c1w8malzrqmv3k6ngxejgh526c354mf 2436038 2436037 2026-07-14T07:39:24Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2002.codfw.wmnet 2436038 wikitext text/x-wiki jziadtxeq8czrlahh69bg9a2341bo6i 2436039 2436038 2026-07-14T07:39:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors 2436039 wikitext text/x-wiki rkja8ddwh3it500e6vbjx1gqz5p9u62 2436040 2436039 2026-07-14T07:39:45Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors 2436040 wikitext text/x-wiki 0f9y3uxds2qg30hblzb8io3u59yv0p0 2436041 2436040 2026-07-14T07:39:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2002.codfw.wmnet 2436041 wikitext text/x-wiki hfqb3tvyb7u4elze66mwm7gebl88o8e 2436042 2436041 2026-07-14T07:44:02Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2002.codfw.wmnet 2436042 wikitext text/x-wiki qxax1nl2v7cm72h2redhj6nq5a9kehv 2436043 2436042 2026-07-14T07:44:04Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2002.codfw.wmnet 2436043 wikitext text/x-wiki e2m4ybdx70xkahwbs9u4s610kh206ie 2436044 2436043 2026-07-14T07:44:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on P{config-master*} and (A:config-master or A:config-master-eqiad or A:config-master-codfw) 2436044 wikitext text/x-wiki 62ssx0ffsiuk4lou6j66s9qn34p5den 2436045 2436044 2026-07-14T07:44:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2003.codfw.wmnet 2436045 wikitext text/x-wiki 45hnq5lurwk5zdl2h6cdj9w5ng6gkt7 2436047 2436045 2026-07-14T07:45:02Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436047 wikitext text/x-wiki 149cgcxg32ju5bec16562f72kiiclp1 2436048 2436047 2026-07-14T07:45:03Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2024.codfw.wmnet with reason: host reimage 2436048 wikitext text/x-wiki 8cqruyrs9m7a0p8c4llf4bssad77got 2436049 2436048 2026-07-14T07:45:11Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436049 wikitext text/x-wiki i3byyowxdunc7uc6e6jeyeyxiiyeyfn 2436050 2436049 2026-07-14T07:46:04Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1024.eqiad.wmnet with reason: host reimage 2436050 wikitext text/x-wiki tg8h9i9331tsa741dluyl2560rfcvfc 2436051 2436050 2026-07-14T07:48:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.pki.restart-reboot rolling reboot on P{pki*} and (A:pki) 2436051 wikitext text/x-wiki 5i4jman8ain422xb70nnjal09wr2382 2436052 2436051 2026-07-14T07:48:54Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2024.codfw.wmnet with reason: host reimage 2436052 wikitext text/x-wiki 2qapsf3xm1jneplvpeksfehupdtxv7l 2436053 2436052 2026-07-14T07:49:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors 2436053 wikitext text/x-wiki c5ce66d4mizpabcggc94vcot5pcjvo8 2436054 2436053 2026-07-14T07:49:13Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2003.codfw.wmnet 2436054 wikitext text/x-wiki 6f0pdw229e7lx825m12zd9g9w4u4aez 2436055 2436054 2026-07-14T07:49:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors 2436055 wikitext text/x-wiki nzexdpberxkz1bfibhx4c760yvtre3n 2436056 2436055 2026-07-14T07:53:13Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1024.eqiad.wmnet with reason: host reimage 2436056 wikitext text/x-wiki 73ct24ouvpper4ho3qjij5ma197mzhv 2436057 2436056 2026-07-14T07:53:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2003.codfw.wmnet 2436057 wikitext text/x-wiki rpw1k1ctgqb33jfhg3u47mbhdqyxs67 2436058 2436057 2026-07-14T07:53:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2003.codfw.wmnet 2436058 wikitext text/x-wiki o5zymijjlkdcayi8365fu08gpls54mw 2436059 2436058 2026-07-14T07:53:20Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2004.codfw.wmnet 2436059 wikitext text/x-wiki jq60d0hnsteprkulp9xw99hqtv3bs73 2436060 2436059 2026-07-14T07:53:55Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2004.codfw.wmnet 2436060 wikitext text/x-wiki hzjiwkctwxm4tbdnd4zqr4ky58vj21s 2436061 2436060 2026-07-14T07:54:56Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors 2436061 wikitext text/x-wiki 4b1gxpkcee4u5b8a08ngthq7urrj009 2436062 2436061 2026-07-14T07:54:59Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors 2436062 wikitext text/x-wiki q5jr6er4dfhm69g7ep3vax4q19ov2bb 2436063 2436062 2026-07-14T07:57:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2004.codfw.wmnet 2436063 wikitext text/x-wiki 0ihg2fjj5si4y3ewadfeiznrkgkjns3 2436064 2436063 2026-07-14T07:57:59Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2004.codfw.wmnet 2436064 wikitext text/x-wiki 41zd3zwjo0e9i5lsh59qn8r33yvy3y9 2436065 2436064 2026-07-14T07:58:04Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2005.codfw.wmnet 2436065 wikitext text/x-wiki b565ux5v8uaqkmipvj9lm1e5bnuoq5q 2436067 2436065 2026-07-14T07:58:37Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2005.codfw.wmnet 2436067 wikitext text/x-wiki c78gwbgsve8qn1sn6wrmamv0c7k40lw 2436068 2436067 2026-07-14T08:02:44Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2005.codfw.wmnet 2436068 wikitext text/x-wiki 304la8hb1him71xev9p3deg1wviuepv 2436069 2436068 2026-07-14T08:02:45Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2005.codfw.wmnet 2436069 wikitext text/x-wiki 1vijpp20uthxjs3yeid07m4sxeprcjy 2436070 2436069 2026-07-14T08:02:51Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet 2436070 wikitext text/x-wiki 1hdsum9as1bz3gf0v55j002nisiozh8 2436071 2436070 2026-07-14T08:03:28Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet 2436071 wikitext text/x-wiki d3bk60tqxtsije2mj4kkdoxtt7n46yn 2436072 2436071 2026-07-14T08:08:04Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2024.codfw.wmnet with OS trixie 2436072 wikitext text/x-wiki htwlflh5ez94k8z4e4ut078bz1g76dt 2436073 2436072 2026-07-14T08:08:35Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet 2436073 wikitext text/x-wiki k493vtyjpvvzvze6630f2jexfho98oi 2436074 2436073 2026-07-14T08:08:37Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet 2436074 wikitext text/x-wiki nofo8ejxe9r4f5y29dhs2g7laz1x86a 2436075 2436074 2026-07-14T08:08:43Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet 2436075 wikitext text/x-wiki o22pocffgvup6lj0wmjxncutx7atohs 2436076 2436075 2026-07-14T08:09:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet 2436076 wikitext text/x-wiki eeo8otqisu9gq2lmtpqi0qce1yb40hh 2436077 2436076 2026-07-14T08:10:59Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436077 wikitext text/x-wiki b0525zr03li8gfphbm7cewvo4jboz4g 2436078 2436077 2026-07-14T08:11:08Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2436078 wikitext text/x-wiki kwxel3x7cq3t674t132lykl0m5agtnt 2436079 2436078 2026-07-14T08:11:54Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436079 wikitext text/x-wiki h9sazk1rdmf0jb5dsyp5b7j6gz86um9 2436080 2436079 2026-07-14T08:12:18Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436080 wikitext text/x-wiki 2v54uco3to6u9j8q2z3wa62xu8f6nwt 2436081 2436080 2026-07-14T08:12:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on P{pki*} and (A:pki) 2436081 wikitext text/x-wiki mxbhzfwg2w619hk5l39t6j0r3ouz9mt 2436082 2436081 2026-07-14T08:13:05Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1024.eqiad.wmnet with OS trixie 2436082 wikitext text/x-wiki gczdnj2mtknoc6p6v9tygrkjbn186qp 2436083 2436082 2026-07-14T08:14:23Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet 2436083 wikitext text/x-wiki 8fcg4mo82npkjpyn08zz4kuw7eu0c2r 2436084 2436083 2026-07-14T08:14:25Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet 2436084 wikitext text/x-wiki hqmg6hqbr6u62u3ldp4mpqliqnaolfv 2436085 2436084 2026-07-14T08:14:30Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet 2436085 wikitext text/x-wiki 4lqxco88qunjoep3ic1v1xihm2jlmi8 2436086 2436085 2026-07-14T08:15:04Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet 2436086 wikitext text/x-wiki 1vuh66wm7b4oi50apljcn59wjsxm402 2436087 2436086 2026-07-14T08:20:24Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet 2436087 wikitext text/x-wiki 8haq434ovokm7fw3oyw8yg5s2jdsy6g 2436088 2436087 2026-07-14T08:20:26Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet 2436088 wikitext text/x-wiki 2wn0jcf5g4heug4concwn560tho2gx2 2436089 2436088 2026-07-14T08:20:32Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet 2436089 wikitext text/x-wiki oubsdf7d16tnfs699us3hwlng1z71l6 2436090 2436089 2026-07-14T08:21:00Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436090 wikitext text/x-wiki h7bnrnat4jy8yktd9h9sul67231wkxe 2436091 2436090 2026-07-14T08:21:02Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436091 wikitext text/x-wiki stkj1fp29oeck4zv8vguo5mt4d933a2 2436092 2436091 2026-07-14T08:21:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet 2436092 wikitext text/x-wiki c02134k8cuh3mejlbgtppbhckd54m2w 2436093 2436092 2026-07-14T08:21:11Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436093 wikitext text/x-wiki c86svq6qdv6ngf5m5dnp9tq5886o1a1 2436094 2436093 2026-07-14T08:21:20Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436094 wikitext text/x-wiki qge1h0p3yiykhdxfy8zy6n65ke44pqt 2436096 2436094 2026-07-14T08:23:43Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1027.eqiad.wmnet with reason: reboot 2436096 wikitext text/x-wiki fgfuyq2doybd0u9p4zzi9o5inx4bo0k 2436097 2436096 2026-07-14T08:24:12Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1029.eqiad.wmnet with reason: reboot 2436097 wikitext text/x-wiki iew4axaqiz7369y5i7fd2zrmzz6f9o2 2436098 2436097 2026-07-14T08:24:27Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2023.codfw.wmnet with OS trixie 2436098 wikitext text/x-wiki syj32yrpexid0u55u6aemsh1bkmyy1w 2436099 2436098 2026-07-14T08:24:33Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436099 wikitext text/x-wiki ommmu2zor57sh6n8wbte4hpwtkj2tuf 2436100 2436099 2026-07-14T08:24:44Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436100 wikitext text/x-wiki rddftgerzb0024qhlzh4r89ssrafc2c 2436101 2436100 2026-07-14T08:25:23Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet 2436101 wikitext text/x-wiki mmspdmoge2a108al0ehc0iqbrppqw37 2436102 2436101 2026-07-14T08:25:37Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1023.eqiad.wmnet with OS trixie 2436102 wikitext text/x-wiki fgckt3x447nwhgfbpnaol2ou2ksfvlj 2436103 2436102 2026-07-14T08:26:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet 2436103 wikitext text/x-wiki m8q8iuiqfmayc1lcv9r3kyf02sfhxib 2436104 2436103 2026-07-14T08:26:58Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet 2436104 wikitext text/x-wiki kt53kiiyrrm1vc2w3l4unh32vkgagg3 2436105 2436104 2026-07-14T08:27:00Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{aux-k8s-worker2*} and (A:aux-master-codfw or A:aux-worker-codfw) 2436105 wikitext text/x-wiki hmu6juha9hpq5u5gncq0qpr6xfdstmj 2436106 2436105 2026-07-14T08:27:45Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2436106 wikitext text/x-wiki am7c31lm4r6up13xaihckknd189emfa 2436107 2436106 2026-07-14T08:29:23Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet 2436107 wikitext text/x-wiki bxqryaelmlr3yprp1tki1iitu8fkvki 2436108 2436107 2026-07-14T08:29:38Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2436108 wikitext text/x-wiki g34muj6h72ehldk1k55dtpwulgauwxj 2436109 2436108 2026-07-14T08:34:38Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436109 wikitext text/x-wiki l65alv9mxtxt12f4snyuljt9xhcv6hr 2436110 2436109 2026-07-14T08:34:44Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436110 wikitext text/x-wiki c2n3i9htng1v4x3s2nx7fxqy9686stn 2436113 2436110 2026-07-14T08:41:30Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2023.codfw.wmnet with reason: host reimage 2436113 wikitext text/x-wiki gc0fd45cyix5165byhn5dla9v9s5khx 2436114 2436113 2026-07-14T08:42:07Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1023.eqiad.wmnet with reason: host reimage 2436114 wikitext text/x-wiki ippdx7ez060se90iiluyl7p2xmzxe97 2436115 2436114 2026-07-14T08:44:37Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2023.codfw.wmnet with reason: host reimage 2436115 wikitext text/x-wiki jekeez546lv6p6orovxatmg8u7yp6of 2436116 2436115 2026-07-14T08:45:31Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T430920 2436116 wikitext text/x-wiki 7990ya0mu7e8tzg743rnctzknrbvfqs 2436117 2436116 2026-07-14T08:45:56Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2220 with weight 0 T430920', diff saved to https://phabricator.wikimedia.org/P94809 and previous config saved to /var/cache/conftool/dbconfig/20260714-084553-cwilliams.json 2436117 wikitext text/x-wiki lk8v4yizecs9j5k7ayrzryh9ry4eddo 2436119 2436117 2026-07-14T08:48:27Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1023.eqiad.wmnet with reason: host reimage 2436119 wikitext text/x-wiki c1qjmtg2f8p10wi2526yveey40jc9wx 2436121 2436119 2026-07-14T08:51:49Z Stashbot 7414 cezmunsta: Starting s7 codfw failover from db2159 to db2220 - T430920 2436121 wikitext text/x-wiki 3t2cb7ut6fy667tjowa65ljiz4vc8ar 2436122 2436121 2026-07-14T08:52:43Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2220 to s7 primary T430920', diff saved to https://phabricator.wikimedia.org/P94810 and previous config saved to /var/cache/conftool/dbconfig/20260714-085239-cwilliams.json 2436122 wikitext text/x-wiki kntzovsp1nb6p34zkczd8ejuhhlufmq 2436123 2436122 2026-07-14T08:53:29Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1055.eqiad.wmnet 2436123 wikitext text/x-wiki 6puutpuquotzpa2zrmda2v60u6becan 2436124 2436123 2026-07-14T08:53:32Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet 2436124 wikitext text/x-wiki 5j36z2p7meaoxw4qdfpf4xnx0pnnhx9 2436125 2436124 2026-07-14T08:54:05Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet 2436125 wikitext text/x-wiki mqfh7vlplqhkrmbucl2qc2mrmrsdk5r 2436126 2436125 2026-07-14T08:54:46Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1055.eqiad.wmnet with OS trixie 2436126 wikitext text/x-wiki 2h4qdb0xdmi08jy0ll5nqnkp5rev7bq 2436127 2436126 2026-07-14T08:55:14Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1055 2436127 wikitext text/x-wiki 4onene9c45o2w0wn89mt9jwfeom1ut9 2436128 2436127 2026-07-14T08:55:52Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.netbox 2436128 wikitext text/x-wiki rgtfk78mxgiv9v04vy7kq5azze5dciv 2436129 2436128 2026-07-14T08:56:27Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2159 T430920', diff saved to https://phabricator.wikimedia.org/P94811 and previous config saved to /var/cache/conftool/dbconfig/20260714-085624-cwilliams.json 2436129 wikitext text/x-wiki gnz1yiyxupj0irhquwejcnf62qs8k5a 2436130 2436129 2026-07-14T08:59:07Z Stashbot 7414 a-pizzata@deploy2003: Started deploy [analytics/refinery@ad6e05b] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@ad6e05b8] 2436130 wikitext text/x-wiki 8k5axi4g473u6kon5gde9035n4n762k 2436131 2436130 2026-07-14T09:00:05Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1055 - blake@cumin1003" 2436131 wikitext text/x-wiki jcscnncp6ju3hv9a1r0az003o3405wd 2436132 2436131 2026-07-14T09:00:09Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1055 - blake@cumin1003" 2436132 wikitext text/x-wiki 4p8b78o2fgldv64k5t34wuwk3pu2pet 2436133 2436132 2026-07-14T09:00:10Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2436133 wikitext text/x-wiki 2u6mq2x1ar2n4w8ochql38a56rmu1pd 2436134 2436133 2026-07-14T09:00:11Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1055.eqiad.wmnet 50.32.64.10.in-addr.arpa 0.5.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436134 wikitext text/x-wiki 9ne9lo9fhhn2dppplg40g13upnh8h35 2436135 2436134 2026-07-14T09:00:13Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1055.eqiad.wmnet 50.32.64.10.in-addr.arpa 0.5.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2436135 wikitext text/x-wiki nh1j6so2j7p7u317i7rw46ro4950ii4 2436136 2436135 2026-07-14T09:00:14Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1055 2436136 wikitext text/x-wiki 3dsf47vl66ua55d4kuobxle0dum2t26 2436137 2436136 2026-07-14T09:01:09Z Stashbot 7414 a-pizzata@deploy2003: Finished deploy [analytics/refinery@ad6e05b] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@ad6e05b8] (duration: 02m 01s) 2436137 wikitext text/x-wiki hl4n25g9s3u0kdbglzgumwooq3mrjmj 2436138 2436137 2026-07-14T09:01:53Z Stashbot 7414 a-pizzata@deploy2003: Started deploy [analytics/refinery@ad6e05b]: Regular analytics weekly train [analytics/refinery@ad6e05b8] 2436138 wikitext text/x-wiki gxzflo1t540d5kpr8zx9tqmrj8q2jqu 2436139 2436138 2026-07-14T09:02:59Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2023.codfw.wmnet with OS trixie 2436139 wikitext text/x-wiki omsihtzi9r55x6dlm2cr5ubdh0u2ea6 2436140 2436139 2026-07-14T09:04:44Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2022.codfw.wmnet with OS trixie 2436140 wikitext text/x-wiki mf4yalfg5y7izu5j531flvwslgk9yi3 2436141 2436140 2026-07-14T09:06:41Z Stashbot 7414 a-pizzata@deploy2003: Finished deploy [analytics/refinery@ad6e05b]: Regular analytics weekly train [analytics/refinery@ad6e05b8] (duration: 04m 49s) 2436141 wikitext text/x-wiki rcoi0g7j3qr61479xz6bz40lz606y3d 2436142 2436141 2026-07-14T09:07:10Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1023.eqiad.wmnet with OS trixie 2436142 wikitext text/x-wiki 3aa3vp375gvjkwf0fywvs7ge1u69wik 2436143 2436142 2026-07-14T09:10:38Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1022.eqiad.wmnet with OS trixie 2436143 wikitext text/x-wiki 6q1h3iqk0zhwnsg8x1yfagd4i8354hw 2436144 2436143 2026-07-14T09:11:11Z Stashbot 7414 a-pizzata@deploy2003: Started deploy [analytics/refinery@ad6e05b] (thin): Regular analytics weekly train THIN [analytics/refinery@ad6e05b8] 2436144 wikitext text/x-wiki 7ywlmgjtb5lb6sr9ga3yaqzvjxxptw4 2436145 2436144 2026-07-14T09:13:19Z Stashbot 7414 a-pizzata@deploy2003: Finished deploy [analytics/refinery@ad6e05b] (thin): Regular analytics weekly train THIN [analytics/refinery@ad6e05b8] (duration: 02m 07s) 2436145 wikitext text/x-wiki ctbzqll4qxgqcfdrdbyzhs4dutjojpe 2436146 2436145 2026-07-14T09:16:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db2159.codfw.wmnet 2436146 wikitext text/x-wiki 924et07wcg31aoeeqo77bspcpsrenyf 2436147 2436146 2026-07-14T09:17:55Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2436147 wikitext text/x-wiki ipq7jyohjl2sj4uk2t7i16iq3jo2kzm 2436148 2436147 2026-07-14T09:18:29Z Stashbot 7414 jgiannelos@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2436148 wikitext text/x-wiki 1uz5ai89uvt1ondbmhjp252ako5kmqz 2436149 2436148 2026-07-14T09:18:31Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2436149 wikitext text/x-wiki cgyt7jyy63us40drgv87r6tjwwjx9it 2436150 2436149 2026-07-14T09:18:56Z Stashbot 7414 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2436150 wikitext text/x-wiki ix8wmr7a8n61zwcal89edl2fqwwzxjp 2436151 2436150 2026-07-14T09:20:03Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2159: Rebooting db2159.codfw.wmnet 2436151 wikitext text/x-wiki f0hl68crf68d5kaondmxn8xpmqm68vp 2436152 2436151 2026-07-14T09:20:14Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2159: Rebooting db2159.codfw.wmnet 2436152 wikitext text/x-wiki hxyonh0cna8uwh7hnfud5moazay0vui 2436153 2436152 2026-07-14T09:21:25Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2022.codfw.wmnet with reason: host reimage 2436153 wikitext text/x-wiki dpi5enegp0w5l140c61xtv6ml5rst5g 2436154 2436153 2026-07-14T09:24:37Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2022.codfw.wmnet with reason: host reimage 2436154 wikitext text/x-wiki fvux8wr593h35df7utq2ntwe4xeosi4 2436155 2436154 2026-07-14T09:27:04Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1022.eqiad.wmnet with reason: host reimage 2436155 wikitext text/x-wiki dv1b9jj3ffrbg2xpdy3bxivio7bpm8a 2436156 2436155 2026-07-14T09:27:09Z Stashbot 7414 mvernon@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-fe1022.eqiad.wmnet with reason: host reimage 2436156 wikitext text/x-wiki c0sbxzh1hppbzoc8mpufoiw5vy7u1yg 2436157 2436156 2026-07-14T09:28:34Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1055 2436157 wikitext text/x-wiki sfk4wzfgwu3r58nqwqh9fk4yaqixbcp 2436158 2436157 2026-07-14T09:28:36Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1055 2436158 wikitext text/x-wiki d2tteqrmargne8f0crzzprrujy7jqk2 2436159 2436158 2026-07-14T09:31:34Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db2159.codfw.wmnet 2436159 wikitext text/x-wiki hw1zfre0ljplczfazuxmgcal0kgjmlz 2436160 2436159 2026-07-14T09:42:31Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2022.codfw.wmnet with OS trixie 2436160 wikitext text/x-wiki 0sr8cgtykwzc9cxffrjnwdoptuhx4vg 2436161 2436160 2026-07-14T09:44:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2021.codfw.wmnet with OS trixie 2436161 wikitext text/x-wiki q23qklnkl05ncxtyqna8agdr6ji4hj3 2436162 2436161 2026-07-14T09:44:14Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2159: Repooling after switchover 2436162 wikitext text/x-wiki 3ee2jkqjn0z2b9em49oc8ursrf5208i 2436163 2436162 2026-07-14T09:44:18Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1022.eqiad.wmnet with OS trixie 2436163 wikitext text/x-wiki rmk76lif0doimcdbocvn35yczwnkcou 2436164 2436163 2026-07-14T09:45:44Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1021.eqiad.wmnet with OS trixie 2436164 wikitext text/x-wiki opp3jqr0b6cu5yowj2c54qbnrs1a4qv 2436165 2436164 2026-07-14T09:45:57Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1055.eqiad.wmnet with reason: host reimage 2436165 wikitext text/x-wiki mbtnjyko0wgafrtc45ad4dy8ugemsy2 2436166 2436165 2026-07-14T09:50:14Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1055.eqiad.wmnet with reason: host reimage 2436166 wikitext text/x-wiki 93lg0gus47rlethyuq2bdu5fmwk5vrb 2436167 2436166 2026-07-14T09:58:42Z Stashbot 7414 atsukoito: restarting pybal on lvs1020 for https://gerrit.wikimedia.org/r/1310129 2436167 wikitext text/x-wiki 67tc0xk9ti18fqb6pgb7dteqbeada7x 2436168 2436167 2026-07-14T10:01:09Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2021.codfw.wmnet with reason: host reimage 2436168 wikitext text/x-wiki kccpewrg53kke82a4rb8109w0lq69lp 2436169 2436168 2026-07-14T10:02:14Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1021.eqiad.wmnet with reason: host reimage 2436169 wikitext text/x-wiki chv3yw78z4eud06z4h7ehgltaappwg7 2436170 2436169 2026-07-14T10:03:46Z Stashbot 7414 atsukoito: restarting pybal on lvs1020 for https://gerrit.wikimedia.org/r/1310129 revert 2436170 wikitext text/x-wiki oucr6oe3fz6s41yxuhc7eap3u512pra 2436171 2436170 2026-07-14T10:05:48Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2021.codfw.wmnet with reason: host reimage 2436171 wikitext text/x-wiki 5bk324qk6c3tnwkrz5wlansoixikyza 2436173 2436171 2026-07-14T10:09:33Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1021.eqiad.wmnet with reason: host reimage 2436173 wikitext text/x-wiki 7tgaz5foyywpn8tfwpbj7raxqwofd4i 2436175 2436173 2026-07-14T10:11:37Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1055.eqiad.wmnet with OS trixie 2436175 wikitext text/x-wiki f0yp7j11mrwgu3cn9y23f8ip3w3utbj 2436176 2436175 2026-07-14T10:24:04Z Stashbot 7414 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1307716|extension-list: Add WikimediaAntiAbuse (T431023)]] 2436176 wikitext text/x-wiki fhguzz6yf5mq8hsjxbrccozry1rrs6r 2436177 2436176 2026-07-14T10:24:06Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2021.codfw.wmnet with OS trixie 2436177 wikitext text/x-wiki p11mgiua5gyczrqc59kekoao5qmeovz 2436179 2436177 2026-07-14T10:26:08Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet 2436179 wikitext text/x-wiki j093qfv3zptkhfffi6gtviq0tsx8grb 2436180 2436179 2026-07-14T10:26:10Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet 2436180 wikitext text/x-wiki jc38em76lem5efzu2678vai5aharvzt 2436181 2436180 2026-07-14T10:26:11Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1055.eqiad.wmnet 2436181 wikitext text/x-wiki ip6tui542xhwpxvk8cvozhenipwm0kx 2436182 2436181 2026-07-14T10:26:12Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2020.codfw.wmnet with OS trixie 2436182 wikitext text/x-wiki 293fjpde7s4hb8z3in81d70fuf45wcl 2436183 2436182 2026-07-14T10:26:50Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1067.eqiad.wmnet 2436183 wikitext text/x-wiki 7qagn3enj67dr0ilwxbm8qna4kwacn2 2436184 2436183 2026-07-14T10:26:54Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet 2436184 wikitext text/x-wiki segmq51fpyze582xp1lr0aopxro2fzd 2436185 2436184 2026-07-14T10:27:23Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436185 wikitext text/x-wiki 9uozl8j3umt12m6x4a4h12tnjvvgx2h 2436186 2436185 2026-07-14T10:27:25Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet 2436186 wikitext text/x-wiki tspy6pv93kvrk8pqftmspbezs13degl 2436187 2436186 2026-07-14T10:27:34Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436187 wikitext text/x-wiki 7hkt7nsgt30grrryboin4xm2785uguj 2436188 2436187 2026-07-14T10:27:41Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1067.eqiad.wmnet with OS trixie 2436188 wikitext text/x-wiki n1xy2q6vanfd3mbvrqew4ac8dlemt7h 2436189 2436188 2026-07-14T10:28:14Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1021.eqiad.wmnet with OS trixie 2436189 wikitext text/x-wiki 5ftbecjmmd9yoz5qqp66fsqzk6dw2wg 2436190 2436189 2026-07-14T10:29:40Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1067 2436190 wikitext text/x-wiki 8n1wzebw7c4nblo188imaih640umci8 2436191 2436190 2026-07-14T10:29:45Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2159: Repooling after switchover 2436191 wikitext text/x-wiki 7v2r1pn12fqbynfo0d8olply6gaggxn 2436193 2436191 2026-07-14T10:32:16Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1020.eqiad.wmnet with OS trixie 2436193 wikitext text/x-wiki dkl45m1l5vcf1mf2cxhnwg73rzujt9t 2436195 2436193 2026-07-14T10:42:24Z Stashbot 7414 kharlan@deploy2003: kharlan: Backport for [[gerrit:1307716|extension-list: Add WikimediaAntiAbuse (T431023)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436195 wikitext text/x-wiki jvzn7s8giz4riv6zqotpdnqfazh9siz 2436196 2436195 2026-07-14T10:43:02Z Stashbot 7414 kharlan@deploy2003: kharlan: Continuing with deployment 2436196 wikitext text/x-wiki kmqkmd5j0ivg63ppvxxs0x3mdjangh9 2436197 2436196 2026-07-14T10:45:58Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2020.codfw.wmnet with reason: host reimage 2436197 wikitext text/x-wiki 44zjhc1bw78k7xafb584z9y648tohue 2436198 2436197 2026-07-14T10:48:19Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1020.eqiad.wmnet with reason: host reimage 2436198 wikitext text/x-wiki olc7sdg68dgm99qyej559zzsjuiz9y3 2436199 2436198 2026-07-14T10:49:43Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436199 wikitext text/x-wiki h75l60f8goxalqyehhhpbse6yba61wu 2436200 2436199 2026-07-14T10:49:50Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2020.codfw.wmnet with reason: host reimage 2436200 wikitext text/x-wiki aa1yv1g5fmvjwlnsd8jg7d3ad5yl7z7 2436201 2436200 2026-07-14T10:49:53Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436201 wikitext text/x-wiki 8h77dywpeiyvf8ctaeq6hhvjfz3atlx 2436202 2436201 2026-07-14T10:52:44Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2436202 wikitext text/x-wiki to8dqjtk587vfe5teu76kcjjo30p015 2436203 2436202 2026-07-14T10:52:51Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1020.eqiad.wmnet with reason: host reimage 2436203 wikitext text/x-wiki jx6n88suvw1tj98a5ufucft2xri3s2v 2436204 2436203 2026-07-14T10:55:44Z Stashbot 7414 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1307716|extension-list: Add WikimediaAntiAbuse (T431023)]] (duration: 31m 40s) 2436204 wikitext text/x-wiki ie36zsp8irc1q73fqz11lfl8xdp1h72 2436205 2436204 2026-07-14T10:57:38Z Stashbot 7414 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1307717|WikimediaAntiAbuse: Register wmgUse config and load extension (T431023)]] 2436205 wikitext text/x-wiki djt5j61zmsl7ws83reh90uswtbmrin4 2436206 2436205 2026-07-14T11:01:22Z Stashbot 7414 kharlan@deploy2003: kharlan: Backport for [[gerrit:1307717|WikimediaAntiAbuse: Register wmgUse config and load extension (T431023)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436206 wikitext text/x-wiki tcqjbdgg1u10cwhjkxgsyt4crkqrfk7 2436207 2436206 2026-07-14T11:01:55Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.dns.netbox 2436207 wikitext text/x-wiki moo9rjl121dtjve45qdvhwzqs9954b5 2436208 2436207 2026-07-14T11:03:18Z Stashbot 7414 kharlan@deploy2003: kharlan: Continuing with deployment 2436208 wikitext text/x-wiki tpqzvp8vovisuyz6g7vj5y597orocsk 2436209 2436208 2026-07-14T11:08:59Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2020.codfw.wmnet with OS trixie 2436209 wikitext text/x-wiki rjqn8ppsisjuojck5kfjzs11z13ft2r 2436210 2436209 2026-07-14T11:09:51Z Stashbot 7414 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1307717|WikimediaAntiAbuse: Register wmgUse config and load extension (T431023)]] (duration: 12m 12s) 2436210 wikitext text/x-wiki hlluee1ai6jp6du41ak33yym04cs249 2436211 2436210 2026-07-14T11:10:02Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1067 - blake@cumin1003" 2436211 wikitext text/x-wiki efd1hk14cy9t4turo9ypn2u50msfvpw 2436212 2436211 2026-07-14T11:10:40Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1020.eqiad.wmnet with OS trixie 2436212 wikitext text/x-wiki mf5it6idggz8n75knhe9edd124ps448 2436213 2436212 2026-07-14T11:10:48Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-fe2019.codfw.wmnet with OS trixie 2436213 wikitext text/x-wiki tb4vz4ksj8vp6dgete0haaww4k9it8t 2436216 2436213 2026-07-14T11:12:38Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1019.eqiad.wmnet with OS trixie 2436216 wikitext text/x-wiki l3s6apme85s2krbzo785tjwr2jt8hp1 2436217 2436216 2026-07-14T11:13:07Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436217 wikitext text/x-wiki p3kzgb53mcxwaac6yob2bsqdxtzvo96 2436218 2436217 2026-07-14T11:13:19Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436218 wikitext text/x-wiki ogam21r9ilvdry9m39ev2lahzcg21vg 2436221 2436218 2026-07-14T11:15:02Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet 2436221 wikitext text/x-wiki 0oxi37fexutrh3cvzlwg9jiuzr9khof 2436227 2436221 2026-07-14T11:19:56Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436227 wikitext text/x-wiki hnxmgw0zhkokq8ey9ovnt4r0p4f5ybu 2436228 2436227 2026-07-14T11:20:04Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436228 wikitext text/x-wiki mdy6rcr8qnqcayfx4u48zybr8t7j04c 2436229 2436228 2026-07-14T11:20:11Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet 2436229 wikitext text/x-wiki 294rj987d5ugi9ty7mnvqbap2b1bypb 2436231 2436229 2026-07-14T11:26:47Z Stashbot 7414 jforrester@deploy2003: Started scap sync-world: Backport for [[gerrit:1310530|abstractwiki: Make cacheAbstractContentFragment throttling a global setting (T430898)]] 2436231 wikitext text/x-wiki h2gek0ur5tmhkcjgoyx926zm9vsorf0 2436232 2436231 2026-07-14T11:28:39Z Stashbot 7414 jforrester@deploy2003: jforrester, gengh: Backport for [[gerrit:1310530|abstractwiki: Make cacheAbstractContentFragment throttling a global setting (T430898)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2436232 wikitext text/x-wiki mo3fbi3krwtad7ozs6xcuiquu84jfhh 2436233 2436232 2026-07-14T11:28:54Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1019.eqiad.wmnet with reason: host reimage 2436233 wikitext text/x-wiki tfwxlerkw9u10ps5ex5gcjdf5903vdm 2436234 2436233 2026-07-14T11:29:58Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2019.codfw.wmnet with reason: host reimage 2436234 wikitext text/x-wiki 6fooqyfd9qovoay3sjl3issl1ivta8z 2436236 2436234 2026-07-14T11:32:12Z Stashbot 7414 jforrester@deploy2003: jforrester, gengh: Continuing with deployment 2436236 wikitext text/x-wiki o70cee3fq91jdoubroywh1nrwzlxe4l 2436238 2436236 2026-07-14T11:32:43Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1019.eqiad.wmnet with reason: host reimage 2436238 wikitext text/x-wiki ji8tdt2ckvh6g8kzcma9vf8306cpphg 2436241 2436238 2026-07-14T11:35:47Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436241 wikitext text/x-wiki 4t7pus490a6nilfihuwpwe2v8867878 2436242 2436241 2026-07-14T11:35:50Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436242 wikitext text/x-wiki 84r18vqhfujql5irik0rtigmzspxvjl 2436243 2436242 2026-07-14T11:36:12Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436243 wikitext text/x-wiki mfqkt7nt6iivb6s9i91fyvkcggma2t0 2436244 2436243 2026-07-14T11:36:26Z Stashbot 7414 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply 2436244 wikitext text/x-wiki koa6fzqg2ge0k4sxena1w96pg278k9g 2436245 2436244 2026-07-14T11:36:29Z Stashbot 7414 jforrester@deploy2003: Finished scap sync-world: Backport for [[gerrit:1310530|abstractwiki: Make cacheAbstractContentFragment throttling a global setting (T430898)]] (duration: 09m 41s) 2436245 wikitext text/x-wiki k8qxvreicz52plch64akicb5q1d4dic 2436246 2436245 2026-07-14T11:36:31Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2019.codfw.wmnet with reason: host reimage 2436246 wikitext text/x-wiki ekheowwkzb8nw5dcr9wf1hdi199agku 2436248 2436246 2026-07-14T11:41:49Z Stashbot 7414 aikochou@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: sync 2436248 wikitext text/x-wiki 9sowtrkqqmbudcaovmcr6hk0k0139y3 2436250 2436248 2026-07-14T11:42:30Z Stashbot 7414 aikochou@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync 2436250 wikitext text/x-wiki 0vpxtwei12ump37tcbf73ptya7gh743 2436251 2436250 2026-07-14T11:42:43Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{aux-k8s-ctrl100*} and (A:aux-master-eqiad or A:aux-worker-eqiad) 2436251 wikitext text/x-wiki pyqezo4k5rxaxc5c1k7bstlzdpx4vzz 2436252 2436251 2026-07-14T11:42:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet 2436252 wikitext text/x-wiki jacfpy88eihrkbdiw0bogw6w2lc0wx7 2436253 2436252 2026-07-14T11:42:47Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet 2436253 wikitext text/x-wiki rgl9qrts39fax84f1t0jl6l1r7z4dv3 2436256 2436253 2026-07-14T11:44:36Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on es1046.eqiad.wmnet with reason: Reimage to Trixie 2436256 wikitext text/x-wiki sulq99shefd60r7vpjg8x7evweqys89 2436257 2436256 2026-07-14T11:44:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1046: Reimage to Trixie 2436257 wikitext text/x-wiki sxlhmh769ie25r63nq4xi9jntt9k8t9 2436260 2436257 2026-07-14T11:46:10Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1046: Reimage to Trixie 2436260 wikitext text/x-wiki d4udg51oqdl61jef5zwqd7hpua2b6j6 2436261 2436260 2026-07-14T11:48:20Z Stashbot 7414 atsukoito: restarting pybal on lvs1020 for https://gerrit.wikimedia.org/r/1310535 2436261 wikitext text/x-wiki snt71tv1mga0gxbe60wnfwbr4233ezi 2436262 2436261 2026-07-14T11:48:47Z Stashbot 7414 aikochou@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: sync 2436262 wikitext text/x-wiki dcbxcqz0dck75v5n5zk5h0ku23e6a3h 2436263 2436262 2026-07-14T11:48:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1046.eqiad.wmnet with OS trixie 2436263 wikitext text/x-wiki mw0p2tvwscm6q58pwpl3ujgfj152n4e 2436264 2436263 2026-07-14T11:49:20Z Stashbot 7414 aikochou@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: sync 2436264 wikitext text/x-wiki s3aobh6px7enxa5fhug2hupbyj6ixd9 2436265 2436264 2026-07-14T11:49:40Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet 2436265 wikitext text/x-wiki qucv8ge266vw79ys4pb0s769a1pk237 2436266 2436265 2026-07-14T11:49:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet 2436266 wikitext text/x-wiki mrre5wivb90uv3mbsjvw0lyr1x3w0rv 2436267 2436266 2026-07-14T11:49:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet 2436267 wikitext text/x-wiki 2kc0vedfmkxi9y2rbgihib9bkp3rapa 2436268 2436267 2026-07-14T11:49:47Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet 2436268 wikitext text/x-wiki eb2od3oaepf7ui09w63mmwg15dger06 2436269 2436268 2026-07-14T11:52:22Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1019.eqiad.wmnet with OS trixie 2436269 wikitext text/x-wiki dzqmauonm9dsjwaxkv3egqidoayno8r 2436271 2436269 2026-07-14T11:54:47Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet 2436271 wikitext text/x-wiki swcdm7vlbp6neym4xic0wgjo1ro73pr 2436272 2436271 2026-07-14T11:54:48Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet 2436272 wikitext text/x-wiki ll4yw996iqp2l0pqudw35ixrkrfoyql 2436273 2436272 2026-07-14T11:54:49Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{aux-k8s-ctrl100*} and (A:aux-master-eqiad or A:aux-worker-eqiad) 2436273 wikitext text/x-wiki 6pxp74vp51bghasgolgwymtc88uuc9r 2436275 2436273 2026-07-14T11:56:33Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-fe1018.eqiad.wmnet with OS trixie 2436275 wikitext text/x-wiki oj6r620bwrs3exm3qmhyw3mwlbenuqh 2436276 2436275 2026-07-14T11:56:48Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{aux-k8s-worker100*} and (A:aux-master-eqiad or A:aux-worker-eqiad) 2436276 wikitext text/x-wiki hh7k1wpg4a0v1pvit2gumw4zfqjcmuo 2436277 2436276 2026-07-14T11:56:51Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1002.eqiad.wmnet 2436277 wikitext text/x-wiki mw9d07odwicxrxlz2okw1338vvxhobf 2436278 2436277 2026-07-14T11:57:04Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2019.codfw.wmnet with OS trixie 2436278 wikitext text/x-wiki dl6yjrr64f8q730p74taphtgq9ya1vt 2436279 2436278 2026-07-14T11:57:25Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1002.eqiad.wmnet 2436279 wikitext text/x-wiki pjc0j6gt1ucsmifb164dkjh7clyqx90 2436283 2436279 2026-07-14T11:58:22Z Stashbot 7414 atsukoito: restarting pybal on lvs1018 `high-traffic2` for https://gerrit.wikimedia.org/r/1310535 2436283 wikitext text/x-wiki c9e0uz0s8ef2cf7d41ols58w4v1ec2g 0 7942 2435778 2426422 2026-07-13T12:45:37Z Jynus 38174 /* What needs to be added? */ 2435778 wikitext text/x-wiki seh30i3tykp46ypg2u1673qnggnb0s0 0 17290 2435856 2435358 2026-07-13T15:03:03Z Stashbot 7414 dancy: Updating gitlab-cloud-runners to v19.0.2 2435856 wikitext text/x-wiki 03rz8gj59r4cmdmd1mluahdtgju1gxr 2436111 2435856 2026-07-14T08:37:01Z Stashbot 7414 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud 2436111 wikitext text/x-wiki px3xhgbm2gv6d69pmlca7nt55oufwu2 2436270 2436111 2026-07-14T11:53:55Z Stashbot 7414 James_F: Docker: [commit-message-validator] Update to v3.0.0, for T431799 2436270 wikitext text/x-wiki tsi1c029c59pto29qeqawzfxsmtzv1m 0 19208 2435853 2420390 2026-07-13T14:58:56Z Krinkle 44 improve lede, hoist mw:Phabricator link into a hatnote 2435853 wikitext text/x-wiki j5gjtcgwtojgzi65b4wqsi4bcbcrwre 498 30942 2436280 2434962 2026-07-14T11:57:37Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431682) 2436280 wikitext text/x-wiki siq8wz55uclu79nkwdi4rpcafagkw1c 2436281 2436280 2026-07-14T11:57:42Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431682) 2436281 wikitext text/x-wiki majd1m5rsfn50szhb9vzwg3hiqrs6uj 2436282 2436281 2026-07-14T11:58:13Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048' (T431682) 2436282 wikitext text/x-wiki 21nbw8vzh6qdq99sn00un3sw2or6lyx 0 32278 2436172 2414616 2026-07-14T10:07:59Z FCeratto-WMF 39104 Add sre.mysql.global-read-only 2436172 wikitext text/x-wiki 5polld7molqsf2qddtgwv0ombicrh9o 0 67018 2435859 2433611 2026-07-13T15:31:58Z SFrench-WMF 37362 /* Limitations */ Add note that X-W-D has been temporarily reenabled for REST API 2435859 wikitext text/x-wiki 1xjl3ya294uej5s21mq5siu9xtb4cfh 498 443261 2435819 2434860 2026-07-13T13:56:53Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-db03 2435819 wikitext text/x-wiki gjmaxx9nv27wxl114ensy7jox0v4ujm 2435820 2435819 2026-07-13T13:57:38Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-db03 2435820 wikitext text/x-wiki 8xne67vbac6j7v08yj7957o1yrqpbpd 2435822 2435820 2026-07-13T13:57:58Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-db04 2435822 wikitext text/x-wiki pednfuwzzppnvq0vi5dyxsimhv3prbd 2435824 2435822 2026-07-13T13:58:45Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-db04 2435824 wikitext text/x-wiki hwpaslsd5lfujvj53pa96ayru2y6fkj 498 443628 2436112 2433126 2026-07-14T08:41:20Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29318761866 (https://github.com/cluebotng/component-configs/commits/04593c4734db7ed505ba34446adacc22176c18ed) 2436112 wikitext text/x-wiki 62q3wkpjluvraefjd5nq5whwuidpqvt 0 444162 2436174 2435712 2026-07-14T10:11:04Z FCeratto-WMF 39104 Add docs for sre.mysql.global-read-only 2436174 wikitext text/x-wiki oxmvrdfn2y9clymfvxvr2t6353n8bki 0 444177 2435904 2384657 2026-07-13T16:36:52Z BCornwall-WMF 31521 /* Routing */ Remove configuration (see talk page) 2435904 wikitext text/x-wiki fxd5yaicty58kesz8vccmnkaqjwye7g 0 446215 2436194 2393924 2026-07-14T10:36:42Z FNegri-WMF 32595 /* Blue Green Deployment, creating a new cluster */ add note explaining when to use 2436194 wikitext text/x-wiki il423f2m4mbhtg892jva2etffcxrn5h 0 448732 2435973 2434703 2026-07-13T23:20:00Z WFan (WMF) 38087 2435973 wikitext text/x-wiki fx2tjy70ywnqor31n88obmkvec70uud 0 449160 2435979 2435705 2026-07-14T00:01:27Z Dexbot 30554 Bot: Updating the report 2435979 wikitext text/x-wiki hmnfniawbew8xk7e5ylhresehcqhmg2 2436118 2435979 2026-07-14T08:47:31Z Dexbot 30554 Bot: Updating the report 2436118 wikitext text/x-wiki d6vhp5rndhlipglbdd7ezp0py8p8r49 0 451075 2435951 2431608 2026-07-13T20:37:25Z DAdedoyin (WMF) 39150 Rotation 2435951 wikitext text/x-wiki esnsyep864mscucxou60sgvj9l00pqu 2435952 2435951 2026-07-13T20:48:28Z WFan (WMF) 38087 /* Historical */ 2435952 wikitext text/x-wiki j00yz0qidhbhsuig83gsdg28m2lv02y 2435970 2435952 2026-07-13T23:09:34Z WFan (WMF) 38087 /* 2026-06-30 - 2026-07-13 */ 2435970 wikitext text/x-wiki b5v82h0rfh7a26yfs4cnhgakmf2iu5o 2435971 2435970 2026-07-13T23:10:44Z WFan (WMF) 38087 /* Current */ 2435971 wikitext text/x-wiki 1l81qwxzgphmpa6l19d7jeulbn9m2zg 0 451448 2436249 2432521 2026-07-14T11:42:16Z Jdforrester (WMF) 468 + Abstract Wikipedia wiki steps for folks to copy,. 2436249 wikitext text/x-wiki ihdkvx6n9g4hwpw6v2btyuf9jcicrrg 1 452570 2435905 2435520 2026-07-13T16:37:25Z BCornwall-WMF 31521 /* Removal of Routing configuration section */ Reply 2435905 wikitext text/x-wiki lrdla5pbxf1pjhad9qqns563tobbqvn 116 453426 2435795 2435688 2026-07-13T13:01:34Z Phabbanbot 37210 Camrensssss was disabled by Mainframe98 2435795 wikitext text/x-wiki tkvnk6z6xla9trtoqk6uyqgqtvsdmjh 0 453597 2435832 2407069 2026-07-13T14:10:00Z DCaro (WMF) 18271 /* 2026 */ Added a couple fixes 2435832 wikitext text/x-wiki b6m5csd3scwztq9ds9z1u2p0x03q2hx 498 458787 2435844 2433017 2026-07-13T14:32:05Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29258212925 (https://github.com/cluebotng/component-configs/commits/2d491fa13ec950460297fa6cfae6801b75d17cf4) 2435844 wikitext text/x-wiki 0iz1jqu9zbnif3szzcd8hw13jspcaw8 0 459093 2435803 2414578 2026-07-13T13:27:19Z AYounsi (WMF) 10040 /* Limitations */ 2435803 wikitext text/x-wiki 85ozlxmbwjbpk3qh2sp5w6im4dhajlj 0 459699 2435939 2435479 2026-07-13T19:48:45Z Stashbot 7414 eileen: civicrm upgraded from 263ff140 to 171652fa 2435939 wikitext text/x-wiki i7x4jlwu9xwg24p488w58kqnrcrma8b 2435965 2435939 2026-07-13T22:46:12Z Stashbot 7414 larssandergreen: civicrm upgraded from 171652fa to 015e84b8 2435965 wikitext text/x-wiki r5adhssf3rjwilodkemc2626pej65mn 2435989 2435965 2026-07-14T01:42:53Z Stashbot 7414 larssandergreen: civicrm upgraded from 015e84b8 to 24d6e9a0 2435989 wikitext text/x-wiki rkyouseynpabt5nvfswfesxd180i4jq 2435992 2435989 2026-07-14T02:45:31Z Stashbot 7414 larssandergreen: tools upgraded from 0c0cbb37 to 93a9d78d 2435992 wikitext text/x-wiki 2hul4f483h8w21vofl28el5wl2ococf 2 460039 2436274 2434761 2026-07-14T11:55:39Z Atsuko-WMF 54082 /* Service-related changes */ 2436274 wikitext text/x-wiki rl47vdacgph5sj7qco0f1gp2czke32w 2 460101 2435914 2435517 2026-07-13T16:50:32Z Quiddity 1884 try adding fallback colors 2435914 sanitized-css text/css 3qu7qic4my5orvs8ak5h7vysby3e1g4 0 460310 2435930 2435559 2026-07-13T17:53:39Z Quiddity 1884 fix "here" as link label 2435930 wikitext text/x-wiki rzsuo4v6v268xjn88iu1xni7l29s4yf 0 460375 2435929 2435585 2026-07-13T17:51:51Z Quiddity 1884 lang="text" 2435929 wikitext text/x-wiki pusjbj0hm6c9vqs7rdun8ldd93328hf 2435960 2435929 2026-07-13T22:18:55Z AWesterinen-WMF 54189 /* POINT Decomposition */ 2435960 wikitext text/x-wiki 1nkg4m97j7aho9eadteg8b8fs3fmocz 2435978 2435960 2026-07-13T23:40:18Z Jheald 3726 /* Example */ 2435978 wikitext text/x-wiki avge9brd4uj7pqwhw7xpjjoq435eosq 0 460377 2435897 2433188 2026-07-13T16:34:09Z TBurmeister (WMF) 27417 Add formatting for code samples and to improve readability, and add xrefs to relevant docs 2435897 wikitext text/x-wiki af1skngj5hiku7xjxr20dknx23jfbq6 2435923 2435897 2026-07-13T17:13:30Z AWesterinen-WMF 54189 /* Query Rewrite Pattern */ 2435923 wikitext text/x-wiki f935kf42bc0zfhwmf2vdlopu859iice 2435924 2435923 2026-07-13T17:16:05Z AWesterinen-WMF 54189 /* Use of curl and jq */ 2435924 wikitext text/x-wiki 61ufy0s2etlejl96jcbw9778i5earvz 2435928 2435924 2026-07-13T17:50:53Z Quiddity 1884 lang="text" 2435928 wikitext text/x-wiki efd4fy6izf0a9oz0zwmqa4l71o30nvk 2435931 2435928 2026-07-13T18:01:48Z TBurmeister (WMF) 27417 Fix code sample display 2435931 wikitext text/x-wiki a0lzawfc4yx4hr25svkuhf9szxo76og 2435961 2435931 2026-07-13T22:20:54Z AWesterinen-WMF 54189 /* Use of curl and jq */ 2435961 wikitext text/x-wiki m1o5098yb038i28bo8vtgiw29k70qt8 2435962 2435961 2026-07-13T22:32:59Z AWesterinen-WMF 54189 /* EntitySearch/wbsearchentities */ 2435962 wikitext text/x-wiki ncwejw3yyqwubzyz214ctu3uurre6t6 2435963 2435962 2026-07-13T22:33:58Z AWesterinen-WMF 54189 /* Use of curl and jq */ 2435963 wikitext text/x-wiki dmi3eaa5zyv8ckq44thvsgp2qodj6hg 1 460409 2435966 2434711 2026-07-13T23:04:22Z Jheald 3726 /* non-replacement of label service ? */ Reply 2435966 wikitext text/x-wiki 3dzw976gn4mr4r1i0f6uyuuz6gaalzr 2435972 2435966 2026-07-13T23:10:48Z Jheald 3726 /* named subqueries */ 2435972 wikitext text/x-wiki me7nblkuyoi4p95n3skkwql5tgcq9js 2435974 2435972 2026-07-13T23:28:21Z Jheald 3726 /* bd:slice and bd:sample */ Reply 2435974 wikitext text/x-wiki g7jbrrbaxdpjl7op9v8uycfp2ewh6jt 2435975 2435974 2026-07-13T23:30:23Z Jheald 3726 /* bd:slice and bd:sample */ 2435975 wikitext text/x-wiki gig1zv8yl1d754es924s49x5lwt1x9v 2435981 2435975 2026-07-14T00:14:29Z Jheald 3726 /* non-replacement of label service ? */ Reply 2435981 wikitext text/x-wiki j6f2wwqqkhw2nln2jjc8jeg9vwf0p3j 2 460427 2435873 2434561 2026-07-13T15:53:53Z ArielGlenn 11 /* End of day report */ 2435873 wikitext text/x-wiki limj3sf8765h6sbxn3vmtmywis27da1 2 460445 2435876 2026-07-13T16:16:40Z Silvan Heintze (WMDE) 15753 first draft 2435876 wikitext text/x-wiki 62z9njx21li9se4ow5q2b5pauvh74g9