Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.4 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2420242 2420137 2026-05-27T12:02:05Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2420242 wikitext text/x-wiki 7zm59gtm802m42o7et9y07k3nssdc7h 2420244 2420242 2026-05-27T12:03:30Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2420244 wikitext text/x-wiki m0offg7mlhb2ncteduiztkkgs6mvxjn 2420253 2420244 2026-05-27T12:12:11Z Stashbot 7414 godog: begin roll-restart webservices with new default requests - T420565 2420253 wikitext text/x-wiki seha09sma658phuh7fxl0olh4gef3cw 2420565 2420253 2026-05-27T20:33:46Z Stashbot 7414 andrewbogott: rebuilt pywikibot for v11.3.0 T426547 2420565 wikitext text/x-wiki jmcioorpa7yvwl9dtwh3fqzii4s5bn7 0 4108 2420240 2420216 2026-05-27T12:01:05Z ScheduleDeploymentBot 37566 Add [[gerrit:1294264]] to Wednesday, May 27 UTC afternoon backport window 2420240 wikitext text/x-wiki p23s6keurahubt39v826mbgnucbf7at 2420352 2420240 2026-05-27T14:47:22Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-05-27}} */ UTC-late infra window (extended) 2420352 wikitext text/x-wiki 962tkvc4iio8g7lrong6e7vj9rq9pmk 2420397 2420352 2026-05-27T15:42:40Z TCipriani (WMF) 4895 /* {{Deployment_day|date=2026-06-03}} */ 2420397 wikitext text/x-wiki 97xpp0tfs77acgp8b462tf923hbegsa 2420412 2420397 2026-05-27T16:07:58Z ScheduleDeploymentBot 37566 Add [[gerrit:1294342]] to Wednesday, May 27 UTC late backport window 2420412 wikitext text/x-wiki jg6ovz39q9xgrxqb0c6nkz7q54okatf 2420413 2420412 2026-05-27T16:08:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1294343]] to Wednesday, May 27 UTC late backport window 2420413 wikitext text/x-wiki g44jxaks6u5ll2v854nirv00wcsoban 2420414 2420413 2026-05-27T16:08:37Z ScheduleDeploymentBot 37566 Add [[gerrit:1294344]] to Wednesday, May 27 UTC late backport window 2420414 wikitext text/x-wiki 35t14ibonsoztm8tug89p5jp2k1ff2h 2420426 2420414 2026-05-27T16:27:37Z ScheduleDeploymentBot 37566 Add [[gerrit:1270986]] to Thursday, May 28 UTC morning backport window 2420426 wikitext text/x-wiki mfluomd2irkv1kzkmre91jlnsj7wsd4 2420462 2420426 2026-05-27T17:42:24Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-05-27}} */ Update extended UTC-late infra duration 2420462 wikitext text/x-wiki 9lwbqxp86co5rrr2g8hldk5srwnx46t 2420506 2420462 2026-05-27T18:19:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1294373]] to Wednesday, May 27 UTC late backport window 2420506 wikitext text/x-wiki bj8qdd8ywq7x8tdn9y14bz7gvfejrbd 2420508 2420506 2026-05-27T18:21:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1294374]] to Wednesday, May 27 UTC late backport window 2420508 wikitext text/x-wiki afafh7p2tuuwh54vde9x40jxtn1ywzj 2420533 2420508 2026-05-27T19:30:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1288370]] to Wednesday, May 27 UTC late backport window 2420533 wikitext text/x-wiki niq4u4ynvpk39s9ghyaph9dldegneoz 2420536 2420533 2026-05-27T19:40:59Z ScheduleDeploymentBot 37566 Add [[gerrit:1293819]] to Wednesday, May 27 UTC late backport window 2420536 wikitext text/x-wiki m9pojtu2az9eyjs61w467yco88d0cxn 2420553 2420536 2026-05-27T20:08:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1294393]] to Thursday, May 28 UTC late backport window 2420553 wikitext text/x-wiki a1xstlxgyrnznb2rkzrf1vpfgfw6f1c 2420659 2420553 2026-05-28T06:15:20Z ScheduleDeploymentBot 37566 Add [[gerrit:1294808]] to Thursday, May 28 UTC morning backport window 2420659 wikitext text/x-wiki 0n7ejw3k2e2782akllx5ayqzg2dupv5 2420723 2420659 2026-05-28T07:45:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1294826]] to Thursday, May 28 UTC afternoon backport window 2420723 wikitext text/x-wiki ov1p739upr4trg76iirx4dwue35jocx 2420795 2420723 2026-05-28T09:21:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1294229]] to Thursday, May 28 UTC late backport window 2420795 wikitext text/x-wiki f4bpx0r5r9ulmli6jkey7njdb02n5nb 0 4199 2420658 2390273 2026-05-28T06:15:00Z Quiddity 1884 clarify the mw: link targets with consistent formatting 2420658 wikitext text/x-wiki sj7bwu387eupiqsov8w0pp5r7m8643p 2420667 2420658 2026-05-28T06:25:18Z Quiddity 1884 cleanup sidebar -- (But......... Should this be merged into [[template:gerrit nav]] ? ) 2420667 wikitext text/x-wiki tsirrv2hhw262mqytphicd9iysqsy5j 12 5649 2420643 2387888 2026-05-28T05:31:02Z Vmbgl21 56001 /* */ 2420643 wikitext text/x-wiki 7reccjk9zfmb1lef0578c6k69nrgqow 2420644 2420643 2026-05-28T05:39:25Z JJMC89 7474 Reverted edit by [[Special:Contributions/Vmbgl21|Vmbgl21]] ([[User talk:Vmbgl21|talk]]) to last revision by [[User:JJMC89|JJMC89]] 2384391 wikitext text/x-wiki 186qxu6jok8l7sssajuwpzjnr3brt8w 0 6479 2420358 2417690 2026-05-27T14:54:05Z CGoubert-WMF 32775 Clarify rate limit services 2420358 wikitext text/x-wiki g4b8kys9t8fuc93t5iqtdvzetecmy89 0 7919 2420241 2420239 2026-05-27T12:01:13Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox 2420241 wikitext text/x-wiki t7xmq63831yh4t20qr1zqqg05sndyxu 2420243 2420241 2026-05-27T12:02:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T426633)', diff saved to https://phabricator.wikimedia.org/P93242 and previous config saved to /var/cache/conftool/dbconfig/20260527-120205-fceratto.json 2420243 wikitext text/x-wiki iu2yrpp6mxfqkfv6678fnkbvm3qisbs 2420245 2420243 2026-05-27T12:04:47Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance 2420245 wikitext text/x-wiki ak29fjaqq6nnmg5wm9ebsiw5cid013r 2420246 2420245 2026-05-27T12:04:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2212 (T426633)', diff saved to https://phabricator.wikimedia.org/P93243 and previous config saved to /var/cache/conftool/dbconfig/20260527-120452-fceratto.json 2420246 wikitext text/x-wiki flcbpkfdiklln19uktltrqzxzvl5dz6 2420248 2420246 2026-05-27T12:09:40Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2420248 wikitext text/x-wiki d63jxj8zzhkkaqbrspq83zkladjgjgs 2420249 2420248 2026-05-27T12:09:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1193: Migration of db1193.eqiad.wmnet completed 2420249 wikitext text/x-wiki 1s4wcay9ttu141mg54h18645kt0vq1y 2420251 2420249 2026-05-27T12:09:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2420251 wikitext text/x-wiki 9rx8hprddnpaz486pb9ep9pku9ar7w4 2420252 2420251 2026-05-27T12:11:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test 2420252 wikitext text/x-wiki 9duc647lgt493kb5jxs95dow02yr3zo 2420254 2420252 2026-05-27T12:12:13Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2189: Test 2420254 wikitext text/x-wiki 25g7alrl87ts25ac1kciildgrxki66f 2420255 2420254 2026-05-27T12:14:35Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" 2420255 wikitext text/x-wiki 4qtrva8fy0p8w1tjkmcqail7vvfdwtl 2420256 2420255 2026-05-27T12:14:41Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" 2420256 wikitext text/x-wiki ksnoli0anxw359v83v9oqpcuje6xgqs 2420257 2420256 2026-05-27T12:14:43Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2420257 wikitext text/x-wiki 4b6twf8d6cdns68a6coulgl8dati3xi 2420258 2420257 2026-05-27T12:14:47Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Migration of db2165.codfw.wmnet completed 2420258 wikitext text/x-wiki 3gv9j56kq4g5hepft692p9wwxdvz9gv 2420259 2420258 2026-05-27T12:14:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2420259 wikitext text/x-wiki 28hw58oqhffgbixieb57v75x4e4jv59 2420260 2420259 2026-05-27T12:15:01Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2420260 wikitext text/x-wiki 6kyqfgzpwz3kbv3npat94u0nek47i2e 2420261 2420260 2026-05-27T12:18:41Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420261 wikitext text/x-wiki cb2gtl643w761z5vqtqmkix9kku30ps 2420262 2420261 2026-05-27T12:19:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1192: Upgrading db1192.eqiad.wmnet 2420262 wikitext text/x-wiki fy0ne2pb6ah07nom6pqguhl4rlnf0qa 2420263 2420262 2026-05-27T12:19:35Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" 2420263 wikitext text/x-wiki 69xh5quzgw7tso1yzq7vz5elbwnniz8 2420264 2420263 2026-05-27T12:19:39Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1192: Upgrading db1192.eqiad.wmnet 2420264 wikitext text/x-wiki 6xtbm9umld56u62bk6v4f0mbykcf8rz 2420265 2420264 2026-05-27T12:19:42Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" 2420265 wikitext text/x-wiki qea2a6fei6ef64i48q3at6koj4d4kc3 2420266 2420265 2026-05-27T12:19:44Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2420266 wikitext text/x-wiki 6xut0vioq6vind494lqrcriekbacjj0 2420267 2420266 2026-05-27T12:19:54Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078 2420267 wikitext text/x-wiki rpab2gdfqur779t558xtjo3i0285r1r 2420268 2420267 2026-05-27T12:19:58Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079 2420268 wikitext text/x-wiki epu037yjggxgnnyjchs9zhmh78bqaxk 2420269 2420268 2026-05-27T12:20:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420269 wikitext text/x-wiki t5lv7rmdng2fo56s4qzx7o934ugb3ht 2420270 2420269 2026-05-27T12:20:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2164: Upgrading db2164.codfw.wmnet 2420270 wikitext text/x-wiki 906hlqrl4gwhs44g2jpn43uhri50odn 2420271 2420270 2026-05-27T12:20:14Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1079 2420271 wikitext text/x-wiki gzv2n9bylbdnrz13zjk04x07i39bd6h 2420272 2420271 2026-05-27T12:20:21Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078 2420272 wikitext text/x-wiki 1zc0elfrzg44m3vejhfn9t2ko4k6ded 2420273 2420272 2026-05-27T12:20:26Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080 2420273 wikitext text/x-wiki 2b078nqnz06acvbp4mzfsjtyoxacb5m 2420274 2420273 2026-05-27T12:20:33Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 36692 2420274 wikitext text/x-wiki 7r8gh6wdbhw0wn79y5ibze419tuie9m 2420275 2420274 2026-05-27T12:20:35Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2164: Upgrading db2164.codfw.wmnet 2420275 wikitext text/x-wiki rz1hm3e12eyrx4gvw7flj2p77abqsf8 2420276 2420275 2026-05-27T12:20:38Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077 2420276 wikitext text/x-wiki i1kvb89eoc2leju5lbh24uaghns2iws 2420277 2420276 2026-05-27T12:20:41Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080 2420277 wikitext text/x-wiki 423twwfkxfn8qi4kj5ud89b1qgjx2pc 2420278 2420277 2026-05-27T12:20:47Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1077 2420278 wikitext text/x-wiki ttk8btdy3uusy8fehsnygw3ek9466zi 2420279 2420278 2026-05-27T12:21:08Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1192.eqiad.wmnet with OS trixie 2420279 wikitext text/x-wiki 2j2d0nhoc6gq3bol41lysam2nbheuqf 2420280 2420279 2026-05-27T12:21:47Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 36692 2420280 wikitext text/x-wiki cm2qiagwm5ytz22wne9epg16gric8e5 2420281 2420280 2026-05-27T12:22:30Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2164.codfw.wmnet with OS trixie 2420281 wikitext text/x-wiki bxar6l02qu9di6acy42onp3l9o2pckf 2420282 2420281 2026-05-27T12:28:54Z Stashbot 7414 Amir1: deleting binlogs older than a year 2420282 wikitext text/x-wiki bkbjeni8huj4bmcoq9hffack4oewxa7 2420283 2420282 2026-05-27T12:35:17Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage 2420283 wikitext text/x-wiki aa8gn8ird18mtcf4wv7ue4oxs14yfg7 2420284 2420283 2026-05-27T12:40:24Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2164.codfw.wmnet with reason: host reimage 2420284 wikitext text/x-wiki 1li36l9tlzp36fl5mws64a5u3gh63s0 2420285 2420284 2026-05-27T12:40:39Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage 2420285 wikitext text/x-wiki c6l1ap3n79qn6ntmbx92blgqmz137no 2420286 2420285 2026-05-27T12:44:22Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: host reimage 2420286 wikitext text/x-wiki p3nxd2g42zfna2u6fpm5vltm3my9duy 2420287 2420286 2026-05-27T12:58:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1192.eqiad.wmnet with OS trixie 2420287 wikitext text/x-wiki 9096aco3x7bzo2lhadm3rs44jvo1v39 2420288 2420287 2026-05-27T13:01:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2164.codfw.wmnet with OS trixie 2420288 wikitext text/x-wiki quiiqcq9q8phx7vqs9bjulrsrgta2mw 2420289 2420288 2026-05-27T13:05:50Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1192: Migration of db1192.eqiad.wmnet completed 2420289 wikitext text/x-wiki 1tnjble4ttnicyynpvb9uhe4dl742ti 2420290 2420289 2026-05-27T13:06:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2212.codfw.wmnet with reason: failed to reboot T427388 T426633 2420290 wikitext text/x-wiki 6b7po7a5cduc6gvqtfymqnjdiyjdd8e 2420291 2420290 2026-05-27T13:06:30Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2420291 wikitext text/x-wiki c5vhw11ndj6dyk1ll2hj4lt20dvm5aa 2420292 2420291 2026-05-27T13:08:12Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1290781|mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264|MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] 2420292 wikitext text/x-wiki cadr9483f5gf8ipxnn60i39atx6kmes 2420293 2420292 2026-05-27T13:10:09Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2164: Migration of db2164.codfw.wmnet completed 2420293 wikitext text/x-wiki euga55etsqkcvfblj1fij8vodmg9gp4 2420294 2420293 2026-05-27T13:10:47Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2420294 wikitext text/x-wiki r97ffiu6dux3g7c23a59ixmyvugxbjz 2420295 2420294 2026-05-27T13:13:17Z Stashbot 7414 mlitn@deploy1003: krinkle, mlitn: Backport for [[gerrit:1290781|mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264|MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420295 wikitext text/x-wiki bgixajkwbpdh5ugupdrbn20rqzevcpw 2420296 2420295 2026-05-27T13:15:40Z Stashbot 7414 mlitn@deploy1003: krinkle, mlitn: Continuing with deployment 2420296 wikitext text/x-wiki j3fh6amehqjhqlabmsywpigdp8p8qz5 2420297 2420296 2026-05-27T13:15:43Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test 2420297 wikitext text/x-wiki 9f3ayxq8soz0qrkstz2sucs5ryrr7ym 2420298 2420297 2026-05-27T13:15:58Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: Test 2420298 wikitext text/x-wiki ierin95zjdaa69wztd4hyn0sdlexidd 2420299 2420298 2026-05-27T13:21:35Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290781|mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264|MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] (duration: 13m 23s) 2420299 wikitext text/x-wiki qxxv8efd639jf4qapa24413tpvk3k1w 2420300 2420299 2026-05-27T13:28:53Z Stashbot 7414 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1294217|ext.wikimediaEvents: Add hoisting error detection test (T427092)]] 2420300 wikitext text/x-wiki rsgnfxatnho95x945qk19f6jylqbsew 2420301 2420300 2026-05-27T13:30:48Z Stashbot 7414 phuedx@deploy1003: phuedx: Backport for [[gerrit:1294217|ext.wikimediaEvents: Add hoisting error detection test (T427092)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420301 wikitext text/x-wiki 9qztqnvi2qmi3bona1vxg9p1o6jstev 2420302 2420301 2026-05-27T13:36:15Z Stashbot 7414 phuedx@deploy1003: phuedx: Continuing with deployment 2420302 wikitext text/x-wiki rqrxjtd108rn9pn5u3rssutm8c9tm3c 2420303 2420302 2026-05-27T13:40:28Z Stashbot 7414 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294217|ext.wikimediaEvents: Add hoisting error detection test (T427092)]] (duration: 11m 35s) 2420303 wikitext text/x-wiki psmpse91uen560lpd4uw6nu7ukpg5ye 2420304 2420303 2026-05-27T13:45:33Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1294247|Update Direct Reporting email (T427358)]] 2420304 wikitext text/x-wiki 22m2lt9potbnnkj8glbg2jqun2yybk3 2420305 2420304 2026-05-27T13:51:20Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1192: Migration of db1192.eqiad.wmnet completed 2420305 wikitext text/x-wiki 8j9kk2b1uhxl7vy7r27kmjj4qz0esya 2420306 2420305 2026-05-27T13:51:23Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2420306 wikitext text/x-wiki 6k7rk054nr7jhwwz8fkll6pvtzra3rj 2420307 2420306 2026-05-27T13:52:19Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420307 wikitext text/x-wiki l9f1vr38p3m0xwhpj1ww4r0b58lswev 2420308 2420307 2026-05-27T13:52:25Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420308 wikitext text/x-wiki 1sd6r79dxvxbg8yijzu7sft6qzprei2 2420309 2420308 2026-05-27T13:55:37Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Migration of db2164.codfw.wmnet completed 2420309 wikitext text/x-wiki 6oft76xfc4jyaxg12o126wkf48fqfou 2420310 2420309 2026-05-27T13:55:39Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2420310 wikitext text/x-wiki j0i94d5hadflt4vvmtcfsyp6ed71gf6 2420311 2420310 2026-05-27T13:56:19Z Stashbot 7414 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie 2420311 wikitext text/x-wiki 0v9f2jiy7f84p5zufeicuce6r3sa0wc 2420312 2420311 2026-05-27T14:03:01Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1294247|Update Direct Reporting email (T427358)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420312 wikitext text/x-wiki 82ujwkyqyrnwfl7b8iy2ok4qv7q3jhz 2420313 2420312 2026-05-27T14:06:04Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2420313 wikitext text/x-wiki qxduok7jwsyy6nxt6zl23irkudfdsxv 2420314 2420313 2026-05-27T14:06:08Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420314 wikitext text/x-wiki q3w77cxe9dx52wst9tnsm1srnualpwc 2420315 2420314 2026-05-27T14:06:17Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420315 wikitext text/x-wiki qogif1vbl336ai2gewo8xn6to3gxmdg 2420316 2420315 2026-05-27T14:06:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420316 wikitext text/x-wiki 0df90b4ei6ta3uxjvrwiq8jfa1hf10c 2420317 2420316 2026-05-27T14:07:20Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1178: Upgrading db1178.eqiad.wmnet 2420317 wikitext text/x-wiki t1nf0a0ufr1peti3oqarsze1esgy23s 2420318 2420317 2026-05-27T14:07:59Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1178: Upgrading db1178.eqiad.wmnet 2420318 wikitext text/x-wiki ouz194rqwke2xzj5q8w8neq3p693zp2 2420319 2420318 2026-05-27T14:08:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2420319 wikitext text/x-wiki aw4746qyrxkorrdvmbwd0oknoem951h 2420320 2420319 2026-05-27T14:08:19Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Upgrading db2163.codfw.wmnet 2420320 wikitext text/x-wiki jfi3c4jlgrzlcordcztpw0h537p996m 2420321 2420320 2026-05-27T14:08:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Upgrading db2163.codfw.wmnet 2420321 wikitext text/x-wiki h92fb2wo2qgqykj4s7t9dimtv4q9v0p 2420322 2420321 2026-05-27T14:09:56Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1178.eqiad.wmnet with OS trixie 2420322 wikitext text/x-wiki 1ljkndw31atdvpsi7jp3a9hik4jjiyu 2420323 2420322 2026-05-27T14:10:56Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS trixie 2420323 wikitext text/x-wiki lf0zqdrg0020u3b3voweru7z7qfhz9w 2420324 2420323 2026-05-27T14:18:35Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294247|Update Direct Reporting email (T427358)]] (duration: 33m 01s) 2420324 wikitext text/x-wiki 0wvmgddqpyhduc9h89w6vtmn7nn214d 2420326 2420324 2026-05-27T14:22:28Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1033.eqiad.wmnet with reason: Maintenance 2420326 wikitext text/x-wiki 5o2pzctkz6gn4hwfv62r10wgx90858i 2420327 2420326 2026-05-27T14:22:55Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage 2420327 wikitext text/x-wiki 0jborfn6uy9h961144aslv49i93sfor 2420328 2420327 2026-05-27T14:23:58Z Stashbot 7414 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290924|Make logging of title and page ID optional (T426457)]] 2420328 wikitext text/x-wiki 8g1evmuzgz8r68wpipgnyvpx42wxj69 2420329 2420328 2026-05-27T14:26:27Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420329 wikitext text/x-wiki qju4zi5vwh5dejan9wsv2v7acd690nb 2420330 2420329 2026-05-27T14:26:33Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420330 wikitext text/x-wiki b3bsuigmwkwdpfct86t16opre2xfcz0 2420331 2420330 2026-05-27T14:26:53Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance 2420331 wikitext text/x-wiki 6u715ngvfciojfjsdn9jb5scekg546b 2420332 2420331 2026-05-27T14:27:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1179 (T426633)', diff saved to https://phabricator.wikimedia.org/P93260 and previous config saved to /var/cache/conftool/dbconfig/20260527-142659-fceratto.json 2420332 wikitext text/x-wiki j5b0n78605w84qabswhimnx3tyfb7lo 2420333 2420332 2026-05-27T14:27:43Z Stashbot 7414 aude@deploy1003: aude: Backport for [[gerrit:1290924|Make logging of title and page ID optional (T426457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420333 wikitext text/x-wiki ba9ok91ctx9te8z3bp9uo3khhmx2pc3 2420334 2420333 2026-05-27T14:28:17Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage 2420334 wikitext text/x-wiki 5znjepwvtm9db5i80i7scxxqu10iotj 2420338 2420334 2026-05-27T14:29:26Z Stashbot 7414 aude@deploy1003: aude: Continuing with deployment 2420338 wikitext text/x-wiki pozffc8on1suc8c7svzy4a89opj91vz 2420339 2420338 2026-05-27T14:29:29Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage 2420339 wikitext text/x-wiki jv1mym2hp2fv03p6nbzdeyeef1z4ort 2420341 2420339 2026-05-27T14:33:47Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage 2420341 wikitext text/x-wiki de7as8p0xzjeq1db0d3pmddahort77a 2420342 2420341 2026-05-27T14:34:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T426633)', diff saved to https://phabricator.wikimedia.org/P93262 and previous config saved to /var/cache/conftool/dbconfig/20260527-143416-fceratto.json 2420342 wikitext text/x-wiki 9ygqpkwd5p6zqpzj0jro0c8ss3rmgrv 2420343 2420342 2026-05-27T14:35:28Z Stashbot 7414 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290924|Make logging of title and page ID optional (T426457)]] (duration: 11m 30s) 2420343 wikitext text/x-wiki c4tpwkdw0hpxed88obn0md5yp22t3le 2420344 2420343 2026-05-27T14:38:14Z Stashbot 7414 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290926|Re-enable ReadingLists QuickSurvey (T426781)]] 2420344 wikitext text/x-wiki bpxnyhuctgg1ycpr9c4skscbhnomnd5 2420345 2420344 2026-05-27T14:39:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2189.codfw.wmnet with reason: crashed T427376 2420345 wikitext text/x-wiki d87m4c2hn9b378rwdns640m9imqijzy 2420346 2420345 2026-05-27T14:40:06Z Stashbot 7414 aude@deploy1003: aude: Backport for [[gerrit:1290926|Re-enable ReadingLists QuickSurvey (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420346 wikitext text/x-wiki a7pzg3t8679tbqapxi5qzlang3uy0ea 2420347 2420346 2026-05-27T14:42:36Z Stashbot 7414 aude@deploy1003: aude: Continuing with deployment 2420347 wikitext text/x-wiki bj6h2tzrslart6yubwu1gwzz05hzgi8 2420348 2420347 2026-05-27T14:44:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93263 and previous config saved to /var/cache/conftool/dbconfig/20260527-144423-fceratto.json 2420348 wikitext text/x-wiki ttgm0rtjdbx779xtb5k1pbf8unwllmn 2420349 2420348 2026-05-27T14:44:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1178.eqiad.wmnet with OS trixie 2420349 wikitext text/x-wiki 5czkswr6lzcuuybc77xw13lyhx0c8qd 2420351 2420349 2026-05-27T14:46:46Z Stashbot 7414 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290926|Re-enable ReadingLists QuickSurvey (T426781)]] (duration: 08m 32s) 2420351 wikitext text/x-wiki 8242m9uhq97hfpuj4y0qidn7nq9lxjr 2420353 2420351 2026-05-27T14:50:55Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2420353 wikitext text/x-wiki h25vajt8ymt7c6pyi0wjn5sbuv0iz38 2420354 2420353 2026-05-27T14:51:11Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2420354 wikitext text/x-wiki 5xusfn9xwdfes6nmosk6mgekujtpksv 2420356 2420354 2026-05-27T14:51:38Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS trixie 2420356 wikitext text/x-wiki qno1pvq575hxgth9kk8z4m3ncbjvtao 2420357 2420356 2026-05-27T14:54:03Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2420357 wikitext text/x-wiki tpybjccpttruu4kcxdf48hku88joup4 2420359 2420357 2026-05-27T14:54:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93264 and previous config saved to /var/cache/conftool/dbconfig/20260527-145430-fceratto.json 2420359 wikitext text/x-wiki luk1gahhsmzuvkjvr7kmq4j2nnud8rc 2420364 2420359 2026-05-27T14:59:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed 2420364 wikitext text/x-wiki lump5wo7dbnzyeo57z52er4qj9arhvq 2420365 2420364 2026-05-27T15:04:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T426633)', diff saved to https://phabricator.wikimedia.org/P93267 and previous config saved to /var/cache/conftool/dbconfig/20260527-150438-fceratto.json 2420365 wikitext text/x-wiki mfdhkxlclmdkqa6rlvfxfoqxhf5lssc 2420366 2420365 2026-05-27T15:05:03Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1220.eqiad.wmnet with reason: Maintenance 2420366 wikitext text/x-wiki 01gsw7nillt99y5qopvd4kl0d4rnvq5 2420367 2420366 2026-05-27T15:05:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1220 (T426633)', diff saved to https://phabricator.wikimedia.org/P93268 and previous config saved to /var/cache/conftool/dbconfig/20260527-150508-fceratto.json 2420367 wikitext text/x-wiki 1dpbh1zl21dwvgtobcusdwlrc90s7kn 2420368 2420367 2026-05-27T15:08:04Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420368 wikitext text/x-wiki eqgnnk8ndxbrud7i4sz1wksevfiiw76 2420369 2420368 2026-05-27T15:08:11Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420369 wikitext text/x-wiki gm7k932d1ci5nwz7ag19w8w7qcchrrt 2420370 2420369 2026-05-27T15:08:31Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420370 wikitext text/x-wiki 84nhlxiesyzvpd67olm08rrgchlf5t8 2420371 2420370 2026-05-27T15:08:35Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420371 wikitext text/x-wiki 0w7jr8wnk2xt6sducdrp61c3sqtninb 2420372 2420371 2026-05-27T15:09:07Z Stashbot 7414 cdanis: 💔cdanis@apt1002.wikimedia.org ~ 🕚☕ sudo -i reprepro --component main --restrict cidergrinder update trixie-wikimedia 2420372 wikitext text/x-wiki nspoprimrbiorvjw24skwqzwtcwxl8e 2420373 2420372 2026-05-27T15:09:56Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420373 wikitext text/x-wiki cxperj1uo4i2bjja2owy4v7de68bhxi 2420374 2420373 2026-05-27T15:10:01Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420374 wikitext text/x-wiki clp1uvi5ci3o8fpbi7jqxv649tvflmy 2420375 2420374 2026-05-27T15:10:28Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420375 wikitext text/x-wiki 3rgenvrmdefdkza75otfad3k1kuy83q 2420376 2420375 2026-05-27T15:11:01Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420376 wikitext text/x-wiki a89rh8bt9hmdx6hj35i6q30giwfud3g 2420377 2420376 2026-05-27T15:11:06Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Icinga wait failed during run 2420377 wikitext text/x-wiki slp13l8tw755k8a2fybr835k9c8jwyh 2420378 2420377 2026-05-27T15:11:16Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420378 wikitext text/x-wiki ednal5xr85ahgqpojum5ic5jy72mzyn 2420379 2420378 2026-05-27T15:11:36Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420379 wikitext text/x-wiki 54o1xy6ts9mpml2i6ejh9d8wt4w9kgm 2420380 2420379 2026-05-27T15:12:25Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420380 wikitext text/x-wiki gcpg318lfvzbmmyeecg11btig60afor 2420381 2420380 2026-05-27T15:12:43Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420381 wikitext text/x-wiki 7dy06nn0ve168twuefk53akb3bi8mo5 2420382 2420381 2026-05-27T15:13:21Z Stashbot 7414 cdanis: 💙cdanis@cp5026.eqsin.wmnet ~ 🕚☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym 2420382 wikitext text/x-wiki lxfsqt0j98r2ckqgl5fceem8kisqcze 2420384 2420382 2026-05-27T15:19:25Z Stashbot 7414 cdanis: 💙cdanis@cp4047.ulsfo.wmnet ~ 🕦☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym 2420384 wikitext text/x-wiki kelrejwjv37y3dqmg5vqo4eqdpjr3d4 2420385 2420384 2026-05-27T15:19:55Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie 2420385 wikitext text/x-wiki oq4183bweewp91w7493y2kme6xgx5lf 2420386 2420385 2026-05-27T15:22:36Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1178.eqiad.wmnet 2420386 wikitext text/x-wiki b6octi91eiyx1tmk3ta78wm1t4x7p8o 2420387 2420386 2026-05-27T15:22:39Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1178.eqiad.wmnet 2420387 wikitext text/x-wiki el4207kjgt97nikd7grfg4a9b09parv 2420388 2420387 2026-05-27T15:24:59Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1178: Recovering from failure in cookbook 2420388 wikitext text/x-wiki aeveu0xh06mjm7z7z0eo3lz53cp7grf 2420392 2420388 2026-05-27T15:32:24Z Stashbot 7414 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2163: Migration of db2163.codfw.wmnet completed 2420392 wikitext text/x-wiki 05j7tks1v66eebxp7wb9nq2v54jqtzv 2420393 2420392 2026-05-27T15:32:52Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed 2420393 wikitext text/x-wiki 8afgr44dwzf5c859o4dxm43ydtlcsws 2420394 2420393 2026-05-27T15:33:03Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Migration of db2163.codfw.wmnet completed 2420394 wikitext text/x-wiki 6alvo9e5q95ortg5a61gxsh79tpkyzb 2420395 2420394 2026-05-27T15:33:05Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2420395 wikitext text/x-wiki sqsld2b1d7wfi4r5yitjwi8b9zpdnyr 2420396 2420395 2026-05-27T15:40:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 (T426633)', diff saved to https://phabricator.wikimedia.org/P93274 and previous config saved to /var/cache/conftool/dbconfig/20260527-154011-fceratto.json 2420396 wikitext text/x-wiki ny1eaw6rix152m59xeqvt4w888cbqnn 2420398 2420396 2026-05-27T15:45:10Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420398 wikitext text/x-wiki ob4fov8i8olbb7yljxud9qup876a2rw 2420399 2420398 2026-05-27T15:45:19Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420399 wikitext text/x-wiki 810gb50837y47pdtc1b3kpetla37fkz 2420400 2420399 2026-05-27T15:50:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93276 and previous config saved to /var/cache/conftool/dbconfig/20260527-155019-fceratto.json 2420400 wikitext text/x-wiki smrz50we1m260dzq6dhpj13lwo3lz4t 2420401 2420400 2026-05-27T15:50:52Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Testing cookbook 2420401 wikitext text/x-wiki pymbbbtnlkmeaw90id1l77bah35elyf 2420402 2420401 2026-05-27T15:51:13Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Testing cookbook 2420402 wikitext text/x-wiki 5qvkmgkhw8qc0z4s5n2wv444tk5p16p 2420403 2420402 2026-05-27T15:52:21Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet} and A:cp 2420403 wikitext text/x-wiki avoiukan19j5wec2m3o0dw5922fecib 2420404 2420403 2026-05-27T15:52:24Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Repooling after testing patch 2420404 wikitext text/x-wiki 0xuwteyvwqk269avxpkq2e74vrdpak0 2420405 2420404 2026-05-27T15:52:59Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1017.eqiad.wmnet 2420405 wikitext text/x-wiki gelzutzasi1qy0ktm4cnnkol1arl8z1 2420406 2420405 2026-05-27T15:53:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2163.codfw.wmnet 2420406 wikitext text/x-wiki rpjvsjm6b3xdq3m773naxv05ubpp7a6 2420407 2420406 2026-05-27T15:53:07Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2163.codfw.wmnet 2420407 wikitext text/x-wiki 3rynsx7cafln96zscr9o6u0jwonqqg3 2420408 2420407 2026-05-27T15:59:19Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1017.eqiad.wmnet 2420408 wikitext text/x-wiki l2i41j8xwgj16gmzk8rftok5s21b0tv 2420409 2420408 2026-05-27T16:00:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93280 and previous config saved to /var/cache/conftool/dbconfig/20260527-160027-fceratto.json 2420409 wikitext text/x-wiki m0vwzzhemvx4trs8ltq2523776wthfz 2420410 2420409 2026-05-27T16:03:18Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet 2420410 wikitext text/x-wiki ddshig3opndmbaerja7gd99xfesmgfr 2420411 2420410 2026-05-27T16:05:08Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host durum5003.eqsin.wmnet with OS trixie 2420411 wikitext text/x-wiki qvp8kjg4rygmbkig5tftjdh7ztpna0o 2420415 2420411 2026-05-27T16:10:18Z Stashbot 7414 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2420415 wikitext text/x-wiki jugk524yj98epqqgzokvxb6x2vgq4x2 2420416 2420415 2026-05-27T16:10:26Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1178: Recovering from failure in cookbook 2420416 wikitext text/x-wiki jkybjw8zixrucu9gifqo5x8xn3h22q3 2420417 2420416 2026-05-27T16:10:29Z Stashbot 7414 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2420417 wikitext text/x-wiki 2lw5sfrlb26iyav7f72sdtmrcj6soof 2420418 2420417 2026-05-27T16:10:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 (T426633)', diff saved to https://phabricator.wikimedia.org/P93283 and previous config saved to /var/cache/conftool/dbconfig/20260527-161034-fceratto.json 2420418 wikitext text/x-wiki my3v2p8ukvv616ee1yleezhprfm5w0z 2420419 2420418 2026-05-27T16:10:56Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance 2420419 wikitext text/x-wiki jehcp9feke7m0x35db7993584fbpeez 2420420 2420419 2026-05-27T16:11:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1224 (T426633)', diff saved to https://phabricator.wikimedia.org/P93284 and previous config saved to /var/cache/conftool/dbconfig/20260527-161101-fceratto.json 2420420 wikitext text/x-wiki hr7b0vjsto8777j22a8asexfgkwsn3x 2420421 2420420 2026-05-27T16:12:46Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2420421 wikitext text/x-wiki kskam44godlpr3xge4jpb08o1vklq53 2420422 2420421 2026-05-27T16:13:16Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2420422 wikitext text/x-wiki rrgjqmc2inlrpqetqmk3g1bqe0d3ie2 2420423 2420422 2026-05-27T16:13:23Z Stashbot 7414 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2420423 wikitext text/x-wiki 3bdkjtq5p5qfhcyq6bf501d3efqqtlc 2420424 2420423 2026-05-27T16:14:07Z Stashbot 7414 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2420424 wikitext text/x-wiki 3d25izqtfq2cmo7g97sneukf7dqk2nt 2420425 2420424 2026-05-27T16:17:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T426633)', diff saved to https://phabricator.wikimedia.org/P93285 and previous config saved to /var/cache/conftool/dbconfig/20260527-161753-fceratto.json 2420425 wikitext text/x-wiki 3lzp58ms9iukqsq5eyjw7aeisjrd6gx 2420427 2420425 2026-05-27T16:28:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93287 and previous config saved to /var/cache/conftool/dbconfig/20260527-162800-fceratto.json 2420427 wikitext text/x-wiki ivsx7vs2a492gv84ynxpsqngytzl99o 2420430 2420427 2026-05-27T16:37:53Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Repooling after testing patch 2420430 wikitext text/x-wiki 1yyce3aqg9kv7gvlo7n959w837c3wro 2420431 2420430 2026-05-27T16:38:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93290 and previous config saved to /var/cache/conftool/dbconfig/20260527-163808-fceratto.json 2420431 wikitext text/x-wiki mu8t93yk8j2p8t5hdlbvppecdbj3vie 2420432 2420431 2026-05-27T16:42:01Z Stashbot 7414 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1017.eqiad.wmnet with reason: Setting up 2420432 wikitext text/x-wiki akr3mocqbmrj3gukv1klx7i05ixpjug 2420433 2420432 2026-05-27T16:43:17Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet 2420433 wikitext text/x-wiki gbfgebh28zz98pn7taac3qz9oeywzst 2420434 2420433 2026-05-27T16:48:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T426633)', diff saved to https://phabricator.wikimedia.org/P93291 and previous config saved to /var/cache/conftool/dbconfig/20260527-164815-fceratto.json 2420434 wikitext text/x-wiki 60yrjeukllnyxwftvnmedcss9krc91t 2420435 2420434 2026-05-27T16:48:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance 2420435 wikitext text/x-wiki aq14fkzut0u050cegrjegnjtd5rxmwi 2420436 2420435 2026-05-27T16:48:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 (T426633)', diff saved to https://phabricator.wikimedia.org/P93292 and previous config saved to /var/cache/conftool/dbconfig/20260527-164846-fceratto.json 2420436 wikitext text/x-wiki sq3yi3xv0wczk3gvfx3hhwqx6yjdi7r 2420437 2420436 2026-05-27T16:51:27Z Stashbot 7414 atsuko@dns1004: START - running authdns-update 2420437 wikitext text/x-wiki 37nzwkgy3ji53j09b2sc2bv94z4qqzm 2420438 2420437 2026-05-27T16:53:25Z Stashbot 7414 atsuko@dns1004: END - running authdns-update 2420438 wikitext text/x-wiki b6orgvaoo83zihwuvxezfbsnwo46br9 2420439 2420438 2026-05-27T16:56:26Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293775|ProductionServices: Temporarily use shellbox in eqiad]] 2420439 wikitext text/x-wiki iluj2jm04qukqn2p31mkytbnl3dykiu 2420440 2420439 2026-05-27T16:58:32Z Stashbot 7414 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293775|ProductionServices: Temporarily use shellbox in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420440 wikitext text/x-wiki luk0m84gqo8zirnfnzcot9ipf24wo9k 2420441 2420440 2026-05-27T17:00:52Z Stashbot 7414 swfrench@deploy1003: swfrench: Continuing with deployment 2420441 wikitext text/x-wiki fj20fg2aglhbni0cl4tmuj8xn742utj 2420442 2420441 2026-05-27T17:05:11Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293775|ProductionServices: Temporarily use shellbox in eqiad]] (duration: 08m 44s) 2420442 wikitext text/x-wiki l65tiv4gsvxr5g3bjz5x0uyd2kjck3b 2420444 2420442 2026-05-27T17:13:57Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2420444 wikitext text/x-wiki 4vgu8kqxt8rvci0dskn27w0azktvoqv 2420445 2420444 2026-05-27T17:14:45Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2420445 wikitext text/x-wiki ev6ve8i0dgekyzi83bqe3m8i49fqliv 2420446 2420445 2026-05-27T17:14:48Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2420446 wikitext text/x-wiki fmw2l75zk7w6bpwif31ylptkyef92lz 2420448 2420446 2026-05-27T17:15:40Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2420448 wikitext text/x-wiki hewa6gmc0xzuwc6nftfbtat7bnf2adp 2420449 2420448 2026-05-27T17:15:44Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2420449 wikitext text/x-wiki 19yk4lgairchjuzqmf6aigjslycdp3m 2420451 2420449 2026-05-27T17:16:13Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2420451 wikitext text/x-wiki ojxkgq7corozuu3iik14aaure4puhqt 2420452 2420451 2026-05-27T17:16:16Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2420452 wikitext text/x-wiki i9juuf7c2xskfwfeu9imrmh8skbf0e8 2420453 2420452 2026-05-27T17:16:50Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2420453 wikitext text/x-wiki 4s2n1zrcid1b5tbjxb7t0htf1pn62by 2420454 2420453 2026-05-27T17:16:53Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2420454 wikitext text/x-wiki g7mhek16tlza3s2m6imoifksfst6cdg 2420455 2420454 2026-05-27T17:17:22Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2420455 wikitext text/x-wiki cv3xmhgnpbj7ghfat2qofdd0vs3jem7 2420456 2420455 2026-05-27T17:17:26Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2420456 wikitext text/x-wiki gef7elvgg654q9w3up71sqyvq3vqtca 2420457 2420456 2026-05-27T17:18:10Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2420457 wikitext text/x-wiki 8bdhcsx2jo2u0t880v4483wjf8m51wj 2420458 2420457 2026-05-27T17:25:15Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet 2420458 wikitext text/x-wiki k18wvvgd1py5jfchvv5t3ssjfcj4c4j 2420459 2420458 2026-05-27T17:28:02Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293774|ProductionServices: Temporarily use shellbox in codfw]] 2420459 wikitext text/x-wiki 7hu2imh1w1lqfhgdmsul53ansxpci7x 2420460 2420459 2026-05-27T17:31:44Z Stashbot 7414 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293774|ProductionServices: Temporarily use shellbox in codfw]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420460 wikitext text/x-wiki opnb7ldbxx14zrv3i9c5wsfgzisgzsc 2420461 2420460 2026-05-27T17:38:50Z Stashbot 7414 swfrench@deploy1003: swfrench: Continuing with deployment 2420461 wikitext text/x-wiki lnmupxhgkev0c798tur7arrs622h64n 2420463 2420461 2026-05-27T17:43:03Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293774|ProductionServices: Temporarily use shellbox in codfw]] (duration: 15m 01s) 2420463 wikitext text/x-wiki rrg0ni7sjg8dfeal5fujt5j6kygaxz1 2420464 2420463 2026-05-27T17:49:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 (T426633)', diff saved to https://phabricator.wikimedia.org/P93293 and previous config saved to /var/cache/conftool/dbconfig/20260527-174900-fceratto.json 2420464 wikitext text/x-wiki clvc77ik5m7nr92ggc9697pdn8yp3q8 2420465 2420464 2026-05-27T17:51:30Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply 2420465 wikitext text/x-wiki qjartevr866p5g7qjvr05zx15390x9e 2420466 2420465 2026-05-27T17:52:40Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply 2420466 wikitext text/x-wiki i0s9ipri3ct2a58sul37ya2mswmr23x 2420467 2420466 2026-05-27T17:52:43Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply 2420467 wikitext text/x-wiki fvvt2rh41vyyi0tkdhiq1j4nkv41g0q 2420468 2420467 2026-05-27T17:53:21Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply 2420468 wikitext text/x-wiki dwqebq7r07y62mdwkafxkmk3qsosymu 2420469 2420468 2026-05-27T17:53:23Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply 2420469 wikitext text/x-wiki kh4u7jwmyarc2w4791h2yglp0uhncc2 2420470 2420469 2026-05-27T17:53:47Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply 2420470 wikitext text/x-wiki 2cmczp1y7cdsxdpdlun6ztnhhqvb08l 2420471 2420470 2026-05-27T17:53:50Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply 2420471 wikitext text/x-wiki g3fs2qkjviwhiuj57v3lcbccf0kks4d 2420472 2420471 2026-05-27T17:54:10Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2420472 wikitext text/x-wiki 9bfr65csinhpyhnnik6791r5nb1l2nf 2420473 2420472 2026-05-27T17:54:13Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2420473 wikitext text/x-wiki eefyeoq9vl2zp0e4q7bs7bcu35aq7xc 2420474 2420473 2026-05-27T17:54:40Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2420474 wikitext text/x-wiki cb4pmd8jvum0d7h0its7lkkm2ylkp1n 2420475 2420474 2026-05-27T17:54:43Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2420475 wikitext text/x-wiki fzbuxphz2d6xad45uben956jksou48h 2420476 2420475 2026-05-27T17:55:33Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2420476 wikitext text/x-wiki h8jz3uude4swgaavnkeg75ebbl4kd2f 2420477 2420476 2026-05-27T17:58:41Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293776|ProductionServices: Revert to discovery shellbox listeners]] 2420477 wikitext text/x-wiki 83xwflgvzbj7lzegegf2e0aivm6tmh7 2420478 2420477 2026-05-27T17:59:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93294 and previous config saved to /var/cache/conftool/dbconfig/20260527-175908-fceratto.json 2420478 wikitext text/x-wiki sld6uggq8ikp3ldsykw8r8lewbt6im2 2420479 2420478 2026-05-27T18:00:38Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2420479 wikitext text/x-wiki 7b1zgkensxfwehzkigmzf9dwlxr6kpm 2420480 2420479 2026-05-27T18:00:42Z Stashbot 7414 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293776|ProductionServices: Revert to discovery shellbox listeners]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420480 wikitext text/x-wiki os69bx210mjqcfyrsr54vh1uyf4xvnm 2420481 2420480 2026-05-27T18:01:00Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2420481 wikitext text/x-wiki s2xavw6x19qtk08v8ohvca8oj9x4eit 2420482 2420481 2026-05-27T18:01:03Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2420482 wikitext text/x-wiki r4mv0e4knoc6szk2uymvdrg6trr2cv9 2420483 2420482 2026-05-27T18:01:14Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2420483 wikitext text/x-wiki efvgi76acvssdw3qm9xdorda4w62nlm 2420484 2420483 2026-05-27T18:01:17Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2420484 wikitext text/x-wiki 6r37b35mysz4zqdyg8bl5oxlxto4wyn 2420485 2420484 2026-05-27T18:01:28Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2420485 wikitext text/x-wiki q6hc9ty0snccmwfna37wafvsq5r5rnx 2420486 2420485 2026-05-27T18:01:31Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2420486 wikitext text/x-wiki p58v4b54qbtq14l3es43wbih1grlf6f 2420487 2420486 2026-05-27T18:01:42Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2420487 wikitext text/x-wiki 00bx008ae7u3g5gv1j0jcnknwwogaui 2420488 2420487 2026-05-27T18:01:44Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2420488 wikitext text/x-wiki jhb0d227erkioqddurjq8fw68pr0ew9 2420489 2420488 2026-05-27T18:02:02Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2420489 wikitext text/x-wiki evdc78fopfakmq52mq41fqnqfi6ya9a 2420490 2420489 2026-05-27T18:02:04Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2420490 wikitext text/x-wiki rkrh9tvh9hxh96oxl5ki005uxespcfw 2420491 2420490 2026-05-27T18:02:12Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2420491 wikitext text/x-wiki 2v548w4mt9zy65p6ww1d2ow7o3f2j1j 2420492 2420491 2026-05-27T18:03:32Z Stashbot 7414 swfrench@deploy1003: swfrench: Continuing with deployment 2420492 wikitext text/x-wiki sjgypkrbcjqd7pvywmbgv2hxnt5l388 2420493 2420492 2026-05-27T18:07:46Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet 2420493 wikitext text/x-wiki sxqb4m4oc0h0t6ew4407if7lt1aey2f 2420494 2420493 2026-05-27T18:08:28Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1017.eqiad.wmnet 2420494 wikitext text/x-wiki imgnjem463tuujczjnlzsvz19vg1u3d 2420495 2420494 2026-05-27T18:08:31Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1017.eqiad.wmnet 2420495 wikitext text/x-wiki qg9aoidagyq9e8ifivg4ngqm43do9qf 2420496 2420495 2026-05-27T18:09:05Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293776|ProductionServices: Revert to discovery shellbox listeners]] (duration: 10m 24s) 2420496 wikitext text/x-wiki 0v1qyozsllyetlsdw8nl6liciyoc40t 2420497 2420496 2026-05-27T18:09:09Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply 2420497 wikitext text/x-wiki 9re3tb6a5bpds9doufy2w0paff7jliz 2420498 2420497 2026-05-27T18:09:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93295 and previous config saved to /var/cache/conftool/dbconfig/20260527-180915-fceratto.json 2420498 wikitext text/x-wiki oxuqlu7hxvxcqcuaygr80iqjvyn7tsm 2420499 2420498 2026-05-27T18:10:06Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply 2420499 wikitext text/x-wiki 0asl5ia0816az4fxv6xra3ew67qwjls 2420500 2420499 2026-05-27T18:10:24Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply 2420500 wikitext text/x-wiki 8poq1fdixf2dr2vt61m36772c0l8xow 2420501 2420500 2026-05-27T18:11:18Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply 2420501 wikitext text/x-wiki c9ep2a11qsy9aa2sb9v3i2a6szemex3 2420502 2420501 2026-05-27T18:11:51Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2420502 wikitext text/x-wiki 8bkmqmdgn2d1i079a7dr44nxi6mlk2o 2420503 2420502 2026-05-27T18:12:30Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2420503 wikitext text/x-wiki beupcjijqwk1w27o4q14dl8o8hq8xfv 2420504 2420503 2026-05-27T18:12:40Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2420504 wikitext text/x-wiki 093v7hchz6muma2nbr0ax9cojgvbr5q 2420505 2420504 2026-05-27T18:13:15Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2420505 wikitext text/x-wiki 4utvru97q83jqbtdym8egjigdhyfsow 2420507 2420505 2026-05-27T18:19:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 (T426633)', diff saved to https://phabricator.wikimedia.org/P93296 and previous config saved to /var/cache/conftool/dbconfig/20260527-181923-fceratto.json 2420507 wikitext text/x-wiki qnjt34zsxhgbmnnpiz1albgq59yrhji 2420509 2420507 2026-05-27T18:23:11Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment to clean up unused mesh listeners 2420509 wikitext text/x-wiki hvwfi9esljb4m2m6hcuvzfp8gqsos3g 2420510 2420509 2026-05-27T18:24:15Z Stashbot 7414 swfrench@deploy1003: swfrench: Helmfile-only deployment to clean up unused mesh listeners synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420510 wikitext text/x-wiki c2neod1z6f01xw0aqwmqjvs7rpj8mk7 2420511 2420510 2026-05-27T18:25:21Z Stashbot 7414 swfrench@deploy1003: swfrench: Continuing with deployment 2420511 wikitext text/x-wiki 61z1l2kg2ycu8isv9k80o0qcblr3onw 2420512 2420511 2026-05-27T18:29:24Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Helmfile-only deployment to clean up unused mesh listeners (duration: 06m 12s) 2420512 wikitext text/x-wiki c6bvjdm62h32jc3xkg1dwrc14b1mqut 2420514 2420512 2026-05-27T18:35:10Z Stashbot 7414 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] 2420514 wikitext text/x-wiki 3gaoxtggevsn3fhx96mw3x2831pl3fz 2420515 2420514 2026-05-27T18:37:14Z Stashbot 7414 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] (duration: 02m 04s) 2420515 wikitext text/x-wiki e4m4lx0fhdaw3dxvknan46hnljoqsbr 2420516 2420515 2026-05-27T18:39:10Z Stashbot 7414 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] 2420516 wikitext text/x-wiki 0deuw4f5fajcqbix5wtwz2l8lvf0ool 2420517 2420516 2026-05-27T18:40:15Z Stashbot 7414 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 01m 05s) 2420517 wikitext text/x-wiki r5xu5985pkeuds4lad8xmry69lf02ea 2420518 2420517 2026-05-27T18:45:57Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294376|Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375|auth: Mark the hidden token field used for reauth as skippable (T427398)]] 2420518 wikitext text/x-wiki s0037xvodn5s8o8jnaigy12wf73kmi7 2420519 2420518 2026-05-27T18:47:48Z Stashbot 7414 catrope@deploy1003: catrope: Backport for [[gerrit:1294376|Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375|auth: Mark the hidden token field used for reauth as skippable (T427398)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420519 wikitext text/x-wiki tldv6xx9nz730jf9s79ts4rc1t3kunw 2420520 2420519 2026-05-27T18:49:31Z Stashbot 7414 catrope@deploy1003: catrope: Continuing with deployment 2420520 wikitext text/x-wiki sycsomfloto35y726dgitq01zwb8qji 2420521 2420520 2026-05-27T18:49:55Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet 2420521 wikitext text/x-wiki odibqvqoyoov8j3xmhb47ycsh9e2x67 2420522 2420521 2026-05-27T18:53:39Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294376|Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375|auth: Mark the hidden token field used for reauth as skippable (T427398)]] (duration: 07m 41s) 2420522 wikitext text/x-wiki dbtwueh2wfdhhij5n2bl1ers6imgug7 2420523 2420522 2026-05-27T18:53:43Z Stashbot 7414 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] 2420523 wikitext text/x-wiki 588tsiymscr2w79vz3aneh141hcogcq 2420524 2420523 2026-05-27T18:58:43Z Stashbot 7414 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 05m 01s) 2420524 wikitext text/x-wiki azbd0t00vbnfcz5u88nt80f5fausij0 2420525 2420524 2026-05-27T18:59:21Z Stashbot 7414 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] 2420525 wikitext text/x-wiki lmor2wivg0axt1m10pg5vryhgj9shzz 2420526 2420525 2026-05-27T19:01:32Z Stashbot 7414 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] (duration: 02m 08s) 2420526 wikitext text/x-wiki lzvasbd0cphn9j3mhhdp7v0uvo1tb75 2420529 2420526 2026-05-27T19:20:18Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2420529 wikitext text/x-wiki bfk30yzbznmpf7pyxoyog8y6du9iaqs 2420530 2420529 2026-05-27T19:20:35Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2420530 wikitext text/x-wiki ljxrax2knarc3usus40v9oio89br4dq 2420534 2420530 2026-05-27T19:32:10Z Stashbot 7414 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet 2420534 wikitext text/x-wiki ki0mhnn15498drpzxbq5ak91gbsjqnt 2420535 2420534 2026-05-27T19:32:12Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet} and A:cp 2420535 wikitext text/x-wiki ahs5l5kymxh10dpt59rr34icvk0upd9 2420537 2420535 2026-05-27T19:45:48Z Stashbot 7414 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply 2420537 wikitext text/x-wiki tboponkire5a4e57hhbmgoe8zb19mrq 2420538 2420537 2026-05-27T19:46:01Z Stashbot 7414 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply 2420538 wikitext text/x-wiki 1usxwigdox1xh4kcfac61a7yig0rcgj 2420539 2420538 2026-05-27T19:46:03Z Stashbot 7414 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2420539 wikitext text/x-wiki e54szwlubg2273llocxbszepaid3x0o 2420540 2420539 2026-05-27T19:46:14Z Stashbot 7414 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2420540 wikitext text/x-wiki ieq7bzx6q3e8lfygp4z3rcnkn9e29as 2420541 2420540 2026-05-27T19:46:17Z Stashbot 7414 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2420541 wikitext text/x-wiki hvudh7qp9cb87dutsi30zb001pz04g6 2420542 2420541 2026-05-27T19:46:30Z Stashbot 7414 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2420542 wikitext text/x-wiki 2oxlfwg305xt74hycb62qcspkjqk5xl 2420543 2420542 2026-05-27T19:48:08Z Stashbot 7414 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply 2420543 wikitext text/x-wiki ar5fj8c3gcu2d8n2h244s7xeiwnkpev 2420544 2420543 2026-05-27T19:48:20Z Stashbot 7414 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply 2420544 wikitext text/x-wiki k8xiwuhbifmv5rmy17pykgh4l2o0597 2420545 2420544 2026-05-27T19:48:23Z Stashbot 7414 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2420545 wikitext text/x-wiki mhf2o2glb9cw0ax6a3hwofj4bh97691 2420546 2420545 2026-05-27T19:48:37Z Stashbot 7414 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2420546 wikitext text/x-wiki 1cytvoslffpyto05rat5rkonfk6lsij 2420547 2420546 2026-05-27T19:48:40Z Stashbot 7414 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2420547 wikitext text/x-wiki 4okweuiibu7z6jrtv2n2q3rtagw9ybw 2420548 2420547 2026-05-27T19:48:58Z Stashbot 7414 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2420548 wikitext text/x-wiki tunvusipb5521bzr5ci5atenxd4go3a 2420549 2420548 2026-05-27T19:51:26Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye 2420549 wikitext text/x-wiki smjxyr5uik8iomem2choyicq52w1tb9 2420551 2420549 2026-05-27T20:04:36Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 12355 2420551 wikitext text/x-wiki nghyot7m2y5s1zidq59hbt1yynyc3v0 2420552 2420551 2026-05-27T20:05:51Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12355 2420552 wikitext text/x-wiki cbi0oh9ql1j94xqs89tygxwjhjdubrp 2420555 2420552 2026-05-27T20:14:10Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet 2420555 wikitext text/x-wiki qb7by4gdcp7qssuiu53w3qjd0wvj2rn 2420556 2420555 2026-05-27T20:17:25Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294342|Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343|Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344|frwiki: restrict Article Guidance experiment to junior editors (T426871)]] 2420556 wikitext text/x-wiki cbw82brcbgrdanyks947n2o8o7r7273 2420557 2420556 2026-05-27T20:19:17Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1294342|Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343|Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344|frwiki: restrict Article Guidance experiment to junior editors (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v 2420557 wikitext text/x-wiki kbtcmre7jrlulr3eyec0bz7deqtgevj 2420558 2420557 2026-05-27T20:20:56Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1020.eqiad.wmnet 2420558 wikitext text/x-wiki 3ttzsuknusub4zuxaf559nfqy92mypc 2420559 2420558 2026-05-27T20:21:21Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2420559 wikitext text/x-wiki eg8wddvay3hhg2o1ndy8vmge72zh8gj 2420560 2420559 2026-05-27T20:21:45Z Stashbot 7414 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye 2420560 wikitext text/x-wiki fzqbedo6n3fqzfxv2i2bzn8cq6lnsto 2420561 2420560 2026-05-27T20:25:37Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294342|Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343|Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344|frwiki: restrict Article Guidance experiment to junior editors (T426871)]] (duration: 08m 11s) 2420561 wikitext text/x-wiki dzo9y7z4ambtu49kfswjbl4juw8p3y6 2420562 2420561 2026-05-27T20:25:41Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs1016.eqiad.wmnet 2420562 wikitext text/x-wiki k8tnrsx7kswbi0k8be25a51yw09cm6p 2420563 2420562 2026-05-27T20:27:16Z Stashbot 7414 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf12u2 into component/php83 for bookworm-wikimedia - T427312 2420563 wikitext text/x-wiki ok22spsqy2zcwejtyve1zf631wf3t3o 2420564 2420563 2026-05-27T20:31:11Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.dns.netbox 2420564 wikitext text/x-wiki 6ahv0mag49h7f1o7zufi02x38jrzu3s 2420566 2420564 2026-05-27T20:37:13Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" 2420566 wikitext text/x-wiki lrn5biqhbcbi5dmyhba9pvocc9tuzoo 2420567 2420566 2026-05-27T20:38:23Z Stashbot 7414 swfrench-wmf: reprepro include php-defaults_94+wmf12u1 into component/php83 for bookworm-wikimedia - T427312 2420567 wikitext text/x-wiki 2hbvo2dexygot7whqr9v64purq5wpqb 2420568 2420567 2026-05-27T20:39:08Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" 2420568 wikitext text/x-wiki by3qrig9fspkj2z81zguldzvtjbekil 2420569 2420568 2026-05-27T20:39:11Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2420569 wikitext text/x-wiki ih0dj1b2iaarr3mpv8dxqssnqp4q0cs 2420570 2420569 2026-05-27T20:39:14Z Stashbot 7414 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lvs1016.eqiad.wmnet 2420570 wikitext text/x-wiki c4ft7sfaung2o0me55i9c11pvhtuwgt 2420571 2420570 2026-05-27T20:43:59Z Stashbot 7414 swfrench-wmf: reprepro include dh-php_5.5+wmf12u1 into component/php83 for bookworm-wikimedia - T427312 2420571 wikitext text/x-wiki 5cgq1v46szhj3p4ujcqg6jd7w0re3ej 2420572 2420571 2026-05-27T20:44:17Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294373|identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374|Revert^2 "cirrus: AB test query suggester variants" (T407432)]] 2420572 wikitext text/x-wiki qt7sttohvodlro0ajvrb4b3mfx0z97w 2420573 2420572 2026-05-27T20:46:15Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1294373|identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374|Revert^2 "cirrus: AB test query suggester variants" (T407432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420573 wikitext text/x-wiki rtmtyf5c8a5r1y6hjl4kgfwqtqgoq8d 2420574 2420573 2026-05-27T20:47:35Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Continuing with deployment 2420574 wikitext text/x-wiki g9h8af52wg7tkk79tkct0jqwk4xh9yq 2420575 2420574 2026-05-27T20:51:48Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294373|identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374|Revert^2 "cirrus: AB test query suggester variants" (T407432)]] (duration: 07m 30s) 2420575 wikitext text/x-wiki dl0zm3m4z35wy4j1l6411ab9q8st2kd 2420576 2420575 2026-05-27T20:56:30Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1288370|Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819|Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] 2420576 wikitext text/x-wiki p0uiehl55951ati43wom5l1bjpuf2yd 2420577 2420576 2026-05-27T20:58:30Z Stashbot 7414 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Backport for [[gerrit:1288370|Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819|Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420577 wikitext text/x-wiki 0to597h329hti5trhwdtamxpk62dxit 2420578 2420577 2026-05-27T20:59:52Z Stashbot 7414 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Continuing with deployment 2420578 wikitext text/x-wiki seg4kgx9o21hg2aboplkdigmi2k5hj6 2420579 2420578 2026-05-27T21:04:08Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288370|Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819|Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] (duration: 07m 38s) 2420579 wikitext text/x-wiki gapjajrfnr3ynq9nu98b414fct7v566 2420582 2420579 2026-05-27T21:19:57Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2420582 wikitext text/x-wiki rdldu0awvqzwmtu48tfk3aovrdafuwr 2420583 2420582 2026-05-27T21:20:23Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2420583 wikitext text/x-wiki bjatefyryui3cwy9t7z1wlk6b42wlpf 2420584 2420583 2026-05-27T21:20:26Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2420584 wikitext text/x-wiki 11ly6o8r1wi6h7pmnt9f7qomebacjwl 2420585 2420584 2026-05-27T21:20:50Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2420585 wikitext text/x-wiki sd95y55m0dbnnxn7n6eqa867qzft9aq 2420587 2420585 2026-05-27T21:37:43Z Stashbot 7414 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on relforge[1008-1010].eqiad.wmnet with reason: non-production environment 2420587 wikitext text/x-wiki dc7z9cuyfo4mdbl0k0hr8fcp6unv2vz 2420594 2420587 2026-05-27T22:03:44Z Stashbot 7414 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1294370|Carousel only on articles (T427336)]] 2420594 wikitext text/x-wiki 0de42bvl0ob0xxaygvuaoubfqvaq0z7 2420595 2420594 2026-05-27T22:05:40Z Stashbot 7414 egardner@deploy1003: egardner: Backport for [[gerrit:1294370|Carousel only on articles (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420595 wikitext text/x-wiki k7aq2hjuzagfxfnoxvmrtw9b5n1kzee 2420596 2420595 2026-05-27T22:09:34Z Stashbot 7414 egardner@deploy1003: egardner: Continuing with deployment 2420596 wikitext text/x-wiki n1e9pini4m275l1ghl7q57gw70pf2ae 2420597 2420596 2026-05-27T22:13:44Z Stashbot 7414 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294370|Carousel only on articles (T427336)]] (duration: 10m 00s) 2420597 wikitext text/x-wiki bqimbfrufxgon7nf4jlzuu4s8luzphz 2420598 2420597 2026-05-27T22:33:22Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Add conduct.wikimedia.org (T426984) 2420598 wikitext text/x-wiki h9s1z4xmrd5olbpmvximmrrqurxouub 2420599 2420598 2026-05-27T22:34:19Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Add conduct.wikimedia.org (T426984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420599 wikitext text/x-wiki cf74qxm5kk7zlrzkkgq7pd25qo1cm28 2420600 2420599 2026-05-27T22:35:27Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2420600 wikitext text/x-wiki ldxy4rrhxa49j5t6atkk1jmmepw3qcp 2420601 2420600 2026-05-27T22:39:18Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Add conduct.wikimedia.org (T426984) (duration: 07m 16s) 2420601 wikitext text/x-wiki gl70ufvht4b0teavdm0kae1ir88yl4l 2420602 2420601 2026-05-27T22:39:39Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294360|Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322|Thumbnails are not being optimized in large mode (T427237)]] 2420602 wikitext text/x-wiki jvk3gt6jitq7dhqs9l83fozogk3cz3s 2420603 2420602 2026-05-27T22:40:08Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart 2420603 wikitext text/x-wiki 1tgcwru5ye39xup6v4240r76vnj4uyt 2420604 2420603 2026-05-27T22:40:10Z Stashbot 7414 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) 2420604 wikitext text/x-wiki 9j94j90jtc1zwz5kr0rpryslqwl8yvi 2420605 2420604 2026-05-27T22:40:20Z Stashbot 7414 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart 2420605 wikitext text/x-wiki 9ypsleg6ax0qehw7p2vqqhsqnxfi9xj 2420606 2420605 2026-05-27T22:41:32Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1294360|Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322|Thumbnails are not being optimized in large mode (T427237)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420606 wikitext text/x-wiki elh0l4h42cynjgwnij9hwxk1q74gmcn 2420607 2420606 2026-05-27T22:42:26Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2420607 wikitext text/x-wiki iaf5o7g2lqflnk7pxi24bq5riszgwkq 2420608 2420607 2026-05-27T22:46:33Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294360|Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322|Thumbnails are not being optimized in large mode (T427237)]] (duration: 06m 54s) 2420608 wikitext text/x-wiki qjvqgx9sdxuzmzd2721vaqezr7t8m6z 2420610 2420608 2026-05-27T22:50:50Z Stashbot 7414 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294435|passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] 2420610 wikitext text/x-wiki 47hq2hofp7tvzy4v7nnevo00hvv7rl1 2420611 2420610 2026-05-27T22:52:42Z Stashbot 7414 catrope@deploy1003: catrope: Backport for [[gerrit:1294435|passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420611 wikitext text/x-wiki o096pn0l47fp954ugomsw1wa4a1zqpz 2420612 2420611 2026-05-27T22:54:29Z Stashbot 7414 catrope@deploy1003: catrope: Continuing with deployment 2420612 wikitext text/x-wiki ntavjbqpqtj4p0opvqoe51xqa9nz6sx 2420613 2420612 2026-05-27T22:55:03Z Stashbot 7414 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) 2420613 wikitext text/x-wiki se5lman0lvab8s6cor2zi2fhs7zh28x 2420614 2420613 2026-05-27T22:58:39Z Stashbot 7414 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294435|passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] (duration: 07m 49s) 2420614 wikitext text/x-wiki aihpun2shkesuyyuosvybv0g4lnqklm 2420615 2420614 2026-05-27T23:04:47Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294432|Exclude more content from selection (T426308)]], [[gerrit:1285523|Remove MinervaNightMode config after skin cleanup (T426689)]] 2420615 wikitext text/x-wiki 882ebonapo8eukx3qwuembc9c28q4vt 2420616 2420615 2026-05-27T23:06:42Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Backport for [[gerrit:1294432|Exclude more content from selection (T426308)]], [[gerrit:1285523|Remove MinervaNightMode config after skin cleanup (T426689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420616 wikitext text/x-wiki p0rr2nltckd5f5mtz5rrklk4lzblacu 2420617 2420616 2026-05-27T23:09:13Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Continuing with deployment 2420617 wikitext text/x-wiki nl5bhtsioyfl2aspif7u7yk67baurmi 2420618 2420617 2026-05-27T23:13:28Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294432|Exclude more content from selection (T426308)]], [[gerrit:1285523|Remove MinervaNightMode config after skin cleanup (T426689)]] (duration: 08m 42s) 2420618 wikitext text/x-wiki 1zswnh9a9gckihgk74sj7btdqx7zc9d 2420620 2420618 2026-05-28T00:04:05Z Stashbot 7414 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf12u1 into component/php83 for bookworm-wikimedia - T427312 2420620 wikitext text/x-wiki 2n8xdcvaw7dmvvml7ap7b57lcup0mcy 2420621 2420620 2026-05-28T00:04:55Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294438|Init conductwiki (T426984)]] 2420621 wikitext text/x-wiki 9to4bjsp2xlehptnk02ktdvb3dccngx 2420623 2420621 2026-05-28T00:06:53Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294438|Init conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420623 wikitext text/x-wiki t6oack56n5ppof7p1q71100qpqhsa32 2420624 2420623 2026-05-28T00:08:09Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2420624 wikitext text/x-wiki 6m4a4n4yrr3nhjpf7qbbqbr1quz79if 2420625 2420624 2026-05-28T00:08:42Z Stashbot 7414 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf12u1 into component/php83 for bookworm-wikimedia - T427312 2420625 wikitext text/x-wiki 8bgh12oydoepj9fzpl5g22h95z18v45 2420626 2420625 2026-05-28T00:09:43Z Stashbot 7414 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - T427312 2420626 wikitext text/x-wiki hr2wmn9smx8h4ob8n7o2dyqfkbg8hkh 2420627 2420626 2026-05-28T00:12:20Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294438|Init conductwiki (T426984)]] (duration: 07m 25s) 2420627 wikitext text/x-wiki acda2l5rbrjk576d6dcfvygh0x5ys4d 2420628 2420627 2026-05-28T00:18:38Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294470|Activate conductwiki (T426984)]] 2420628 wikitext text/x-wiki axpdc7dvodax22jt3zugzniiv0jw1v8 2420629 2420628 2026-05-28T00:20:36Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294470|Activate conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420629 wikitext text/x-wiki 3vg5rc9f11msdwnxobeny6gfk13oze5 2420630 2420629 2026-05-28T00:21:43Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2420630 wikitext text/x-wiki 1mj5i1md30kft4j3wniwnya1qtgdh2v 2420631 2420630 2026-05-28T00:25:50Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294470|Activate conductwiki (T426984)]] (duration: 07m 12s) 2420631 wikitext text/x-wiki cujar1u6euybksx9ze0cpe397py16gi 2420632 2420631 2026-05-28T00:49:16Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.dns.netbox 2420632 wikitext text/x-wiki 9txcpe4g5kdtatfqw86dg8x79u492nn 2420633 2420632 2026-05-28T00:53:17Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" 2420633 wikitext text/x-wiki p5oya71pl6y243bcs6u0do7xs6fdowq 2420634 2420633 2026-05-28T00:53:47Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" 2420634 wikitext text/x-wiki oirfb1q2ivbx2au2v07yx823lzekdb1 2420635 2420634 2026-05-28T00:53:50Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2420635 wikitext text/x-wiki 608871ubliwexeduymuimkviu8x94za 2420636 2420635 2026-05-28T02:00:54Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2420636 wikitext text/x-wiki 8h5rizod5h3n457c3vh7z5qd26fs0p4 2420637 2420636 2026-05-28T02:07:36Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) 2420637 wikitext text/x-wiki 693ceb4wgs5f4m37hrwgg4uc29rkj39 2420651 2420637 2026-05-28T06:04:06Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 T426095 2420651 wikitext text/x-wiki qcasttp61ict4febgvowfopc47oct9x 2420652 2420651 2026-05-28T06:04:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1193 with weight 0 T426095', diff saved to https://phabricator.wikimedia.org/P93298 and previous config saved to /var/cache/conftool/dbconfig/20260528-060412-fceratto.json 2420652 wikitext text/x-wiki osfrhnxe0turj5q8x2da6zqjcqet85x 2420654 2420652 2026-05-28T06:10:27Z Stashbot 7414 federico3: Starting s8 eqiad failover from db1209 to db1193 - T426095 2420654 wikitext text/x-wiki kzljhtfkfe16uy4a0fbf263qtckb971 2420655 2420654 2026-05-28T06:10:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - T426095', diff saved to https://phabricator.wikimedia.org/P93299 and previous config saved to /var/cache/conftool/dbconfig/20260528-061048-fceratto.json 2420655 wikitext text/x-wiki 7wwi8lg0wzh9uq7v8fe0g0a46fzmz8w 2420656 2420655 2026-05-28T06:11:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1193 to s8 primary and set section read-write T426095', diff saved to https://phabricator.wikimedia.org/P93300 and previous config saved to /var/cache/conftool/dbconfig/20260528-061138-fceratto.json 2420656 wikitext text/x-wiki coxi8zxptuu00ts1ozmd96k9qbmqrwt 2420657 2420656 2026-05-28T06:14:48Z Stashbot 7414 fceratto@dns1005: START - running authdns-update 2420657 wikitext text/x-wiki tb3siz56bnqqywp41g5iwbqpswh7oln 2420660 2420657 2026-05-28T06:16:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1209 T426095', diff saved to https://phabricator.wikimedia.org/P93301 and previous config saved to /var/cache/conftool/dbconfig/20260528-061609-fceratto.json 2420660 wikitext text/x-wiki oggyalfaty9wmuwaxr9d1z3t3vab2c7 2420661 2420660 2026-05-28T06:16:32Z Stashbot 7414 fceratto@dns1005: END - running authdns-update 2420661 wikitext text/x-wiki nv8n8mqoyttdyzdi78p88f2qcopdv3o 2420668 2420661 2026-05-28T06:25:30Z Stashbot 7414 hashar: Restarting CI Jenkins for plugins upgrades 2420668 wikitext text/x-wiki sc590anpw8ycdq72p3bpktwm8m4i3cn 2420671 2420668 2026-05-28T06:33:44Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance 2420671 wikitext text/x-wiki qdzdnsrv5n3c2u6zbcl83nx25o061yo 2420672 2420671 2026-05-28T06:33:51Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2420672 wikitext text/x-wiki 6nsouy7tvroy4zc9hjnn7frbk1u3sn7 2420673 2420672 2026-05-28T06:33:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 (T426633)', diff saved to https://phabricator.wikimedia.org/P93302 and previous config saved to /var/cache/conftool/dbconfig/20260528-063357-fceratto.json 2420673 wikitext text/x-wiki h39lgf8t8a7de6jj7qotkr8bqoip5j3 2420677 2420673 2026-05-28T06:42:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T426633)', diff saved to https://phabricator.wikimedia.org/P93303 and previous config saved to /var/cache/conftool/dbconfig/20260528-064217-fceratto.json 2420677 wikitext text/x-wiki tttplx6r6l0g6pc6j4rheofdtilww7a 2420678 2420677 2026-05-28T06:43:49Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Reboot completed 2420678 wikitext text/x-wiki t1r0axhbbizguq16iyl856u2yqupse4 2420692 2420678 2026-05-28T07:04:06Z Stashbot 7414 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1289898|Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] 2420692 wikitext text/x-wiki 0m514rxu6bs2sd9r0leldwpo0g7tmei 2420697 2420692 2026-05-28T07:06:11Z Stashbot 7414 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Backport for [[gerrit:1289898|Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420697 wikitext text/x-wiki 9itfx50sdh3qtie4v0a9t4f1r2inxir 2420700 2420697 2026-05-28T07:07:09Z Stashbot 7414 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Continuing with deployment 2420700 wikitext text/x-wiki cuk8e85eaqxwqhswtcvxl2ngqmxq8nk 2420701 2420700 2026-05-28T07:11:22Z Stashbot 7414 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289898|Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] (duration: 07m 15s) 2420701 wikitext text/x-wiki si4ugx352kbg0v4jxrllqt77h4gtwh1 2420702 2420701 2026-05-28T07:13:46Z Stashbot 7414 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1270986|Update wikimania wordmark for 2026 (T413331)]] 2420702 wikitext text/x-wiki rah33fitnsm8bhvyjrtwckl70ah6s53 2420703 2420702 2026-05-28T07:15:43Z Stashbot 7414 wmde-fisch@deploy1003: wmde-fisch, robertsky: Backport for [[gerrit:1270986|Update wikimania wordmark for 2026 (T413331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420703 wikitext text/x-wiki 6486su4vgrr87n04ms0vfe7ec2l4s0y 2420705 2420703 2026-05-28T07:16:27Z Stashbot 7414 wmde-fisch@deploy1003: wmde-fisch, robertsky: Continuing with deployment 2420705 wikitext text/x-wiki h13m73o6uxl445hm1r63hyxxdigyiac 2420706 2420705 2026-05-28T07:16:34Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Reboot completed 2420706 wikitext text/x-wiki c6eiecocxm8ebvf94epggvobzlu5yjy 2420707 2420706 2026-05-28T07:18:30Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance 2420707 wikitext text/x-wiki 2phr1urdp8rv3woiqumkxphb7au4wav 2420708 2420707 2026-05-28T07:18:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 (T426633)', diff saved to https://phabricator.wikimedia.org/P93307 and previous config saved to /var/cache/conftool/dbconfig/20260528-071836-fceratto.json 2420708 wikitext text/x-wiki aat44dybfsuyorz5ef1uh08b65066j0 2420710 2420708 2026-05-28T07:20:40Z Stashbot 7414 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270986|Update wikimania wordmark for 2026 (T413331)]] (duration: 06m 54s) 2420710 wikitext text/x-wiki 58s08ktmzplj2j5ksma81erjd58tegd 2420711 2420710 2026-05-28T07:22:57Z Stashbot 7414 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294808|Don't run the click intent experiment on mobile (T426743)]] 2420711 wikitext text/x-wiki e4vt61i7hppr3an15c9ttiv2xb73wxg 2420712 2420711 2026-05-28T07:23:30Z Stashbot 7414 tgr@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwikisource --logwiki=metawiki Ioed Renamed_user_4232d41570b9e8f46ef150e5e360e446 # T427459 2420712 wikitext text/x-wiki gko8ok6937pxceifpj84tedwwqiani6 2420713 2420712 2026-05-28T07:24:41Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica 2420713 wikitext text/x-wiki k5eoqebo7dzzab9d012ykip7b66dgsf 2420714 2420713 2026-05-28T07:24:52Z Stashbot 7414 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Backport for [[gerrit:1294808|Don't run the click intent experiment on mobile (T426743)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420714 wikitext text/x-wiki 62jes29rlf7yd3ddt9205lk82b06b9r 2420715 2420714 2026-05-28T07:25:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 (T426633)', diff saved to https://phabricator.wikimedia.org/P93308 and previous config saved to /var/cache/conftool/dbconfig/20260528-072458-fceratto.json 2420715 wikitext text/x-wiki lllamhpx0a1rsuigpbe5l62dkmkyqn2 2420716 2420715 2026-05-28T07:25:15Z Stashbot 7414 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Continuing with deployment 2420716 wikitext text/x-wiki p3fudyzxjarvsyck63360hm66itx2ib 2420717 2420716 2026-05-28T07:29:25Z Stashbot 7414 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294808|Don't run the click intent experiment on mobile (T426743)]] (duration: 06m 29s) 2420717 wikitext text/x-wiki cd5icf986746dkw9w8o0e0kdmfqzmuv 2420718 2420717 2026-05-28T07:34:40Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica 2420718 wikitext text/x-wiki 9z03lo895rpb3hz50iq2tbqxikrz2hv 2420719 2420718 2026-05-28T07:35:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93309 and previous config saved to /var/cache/conftool/dbconfig/20260528-073506-fceratto.json 2420719 wikitext text/x-wiki prhp0nae3qugrc54jdybzp1rf9t4fsv 2420720 2420719 2026-05-28T07:36:57Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica 2420720 wikitext text/x-wiki 2e00y9ar42ieifdiqn1rntcv7ff12pu 2420721 2420720 2026-05-28T07:37:52Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2189: repool after crash 2420721 wikitext text/x-wiki esxbhwiq3nuvpl54hnxkixzev1iygq9 2420722 2420721 2026-05-28T07:45:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93311 and previous config saved to /var/cache/conftool/dbconfig/20260528-074513-fceratto.json 2420722 wikitext text/x-wiki qxvebrfiw7fyu5ro1a8bkzxefh9vdlj 2420724 2420722 2026-05-28T07:47:05Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica 2420724 wikitext text/x-wiki qvqojognt66m1c20a0if6dmqkhdsukj 2420725 2420724 2026-05-28T07:55:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 (T426633)', diff saved to https://phabricator.wikimedia.org/P93313 and previous config saved to /var/cache/conftool/dbconfig/20260528-075521-fceratto.json 2420725 wikitext text/x-wiki t6v29t2m09zjcj6ll2ml4q4kwr7tywp 2420726 2420725 2026-05-28T07:56:05Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance 2420726 wikitext text/x-wiki 1vzgi1x68pvykl569lhwx38e7cholgw 2420727 2420726 2026-05-28T07:56:25Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020,1022-1023].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2420727 wikitext text/x-wiki 5pjmulzrcfemmscshgnaa505r2zlzwk 2420728 2420727 2026-05-28T07:56:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1211 (T426633)', diff saved to https://phabricator.wikimedia.org/P93314 and previous config saved to /var/cache/conftool/dbconfig/20260528-075631-fceratto.json 2420728 wikitext text/x-wiki 0d9i6tgieu9ohhk1utw9fdsx3nvq4j6 2420730 2420728 2026-05-28T08:03:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T426633)', diff saved to https://phabricator.wikimedia.org/P93315 and previous config saved to /var/cache/conftool/dbconfig/20260528-080309-fceratto.json 2420730 wikitext text/x-wiki ihg2kxyr8sizhza982kcid0uqfwiefd 2420731 2420730 2026-05-28T08:05:25Z Stashbot 7414 hashar@deploy1003: Started deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 2420731 wikitext text/x-wiki avhggcng8dd69t8es7lx3kvy53tl39n 2420732 2420731 2026-05-28T08:05:39Z Stashbot 7414 hashar@deploy1003: Finished deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 (duration: 00m 13s) 2420732 wikitext text/x-wiki 6kuoogxkq3iyo71ebgmn1lg5h6k8g8p 2420733 2420732 2026-05-28T08:09:05Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Test 2420733 wikitext text/x-wiki gkgokm7mdr91dt3sgorcs6jnvjp17bq 2420734 2420733 2026-05-28T08:10:51Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.4 refs T423913 2420734 wikitext text/x-wiki lqdxiadrkk3ntzuxy38r6d60exaxu1l 2420735 2420734 2026-05-28T08:13:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93318 and previous config saved to /var/cache/conftool/dbconfig/20260528-081316-fceratto.json 2420735 wikitext text/x-wiki 7idd2ct5fuys9wklibnc6m9rmzwcnr0 2420736 2420735 2026-05-28T08:16:04Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2420736 wikitext text/x-wiki f9g4n9d60qbd0trrty81dvcrfk6pfdl 2420737 2420736 2026-05-28T08:17:49Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2420737 wikitext text/x-wiki n09w3puek20pci018domnb934rdpgxo 2420738 2420737 2026-05-28T08:23:17Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: repool after crash 2420738 wikitext text/x-wiki tp5gcufl99mcy0ry1gh56wztmzre2hr 2420739 2420738 2026-05-28T08:23:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93320 and previous config saved to /var/cache/conftool/dbconfig/20260528-082324-fceratto.json 2420739 wikitext text/x-wiki a33td9d23b2saeyvnacywg54z2ceodf 2420740 2420739 2026-05-28T08:24:21Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Test 2420740 wikitext text/x-wiki 06qdmvl4lpo5dhcwa7s6zadur3fp00d 2420741 2420740 2026-05-28T08:33:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T426633)', diff saved to https://phabricator.wikimedia.org/P93322 and previous config saved to /var/cache/conftool/dbconfig/20260528-083331-fceratto.json 2420741 wikitext text/x-wiki j6q8c78wvinjl0wxttay0of2hfly78q 2420742 2420741 2026-05-28T08:34:39Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance 2420742 wikitext text/x-wiki hlmva1z6uj0rojjmdija6z8jbykaan8 2420743 2420742 2026-05-28T08:34:59Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance 2420743 wikitext text/x-wiki g5euom9zbmcxug6t1s4ydymthcp7r1u 2420744 2420743 2026-05-28T08:35:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 (T426633)', diff saved to https://phabricator.wikimedia.org/P93323 and previous config saved to /var/cache/conftool/dbconfig/20260528-083504-fceratto.json 2420744 wikitext text/x-wiki 0cf8vus69ddquajqbz4e1oxcwor6a1n 2420745 2420744 2026-05-28T08:37:17Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet 2420745 wikitext text/x-wiki qm3tcib85v84q63aos4rodfeeanbkgg 2420747 2420745 2026-05-28T08:39:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet 2420747 wikitext text/x-wiki 7cubs2a0nzas6bd5h78rqgt9xdccudg 2420748 2420747 2026-05-28T08:40:11Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294925|hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] 2420748 wikitext text/x-wiki 887r58xk9j7pgh4ajip57uncewznubw 2420749 2420748 2026-05-28T08:41:52Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294925|hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420749 wikitext text/x-wiki 9525v8vyje8eonllq2e3g2amk13p5tv 2420750 2420749 2026-05-28T08:41:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T426633)', diff saved to https://phabricator.wikimedia.org/P93324 and previous config saved to /var/cache/conftool/dbconfig/20260528-084149-fceratto.json 2420750 wikitext text/x-wiki iwxlvcn4wnkrxn5n4dk6fvovaacogle 2420751 2420750 2026-05-28T08:45:15Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2420751 wikitext text/x-wiki qfdmzjvs9m848nnmaw67359v652quzx 2420752 2420751 2026-05-28T08:46:29Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2420752 wikitext text/x-wiki hhfjnj4utcidtx1anyfngyhz59z4ffx 2420753 2420752 2026-05-28T08:47:07Z Stashbot 7414 slyngs: Upgrade IDP to CAS 7.3.7.1 2420753 wikitext text/x-wiki a2uwt8y0o6am1uja2tmzf2z65cwaoba 2420754 2420753 2026-05-28T08:47:18Z Stashbot 7414 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] 2420754 wikitext text/x-wiki b5j4dgcqr879yyq7bhi2v7w0hku9b3j 2420755 2420754 2026-05-28T08:47:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1165: Reboot completed 2420755 wikitext text/x-wiki 7njwa8dpb47hrx0jxul8piudpf0v8pv 2420756 2420755 2026-05-28T08:48:13Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2420756 wikitext text/x-wiki q3h0boiplx806x0vbauel9fo1tl7zxs 2420757 2420756 2026-05-28T08:49:01Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance 2420757 wikitext text/x-wiki mx2u9fol1xrs5rxlyhcrgdgn7lld7xj 2420758 2420757 2026-05-28T08:49:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1209 (T419635)', diff saved to https://phabricator.wikimedia.org/P93326 and previous config saved to /var/cache/conftool/dbconfig/20260528-084906-fceratto.json 2420758 wikitext text/x-wiki g85nge5xpmwz8733uz5ek4s025c6nku 2420759 2420758 2026-05-28T08:49:19Z Stashbot 7414 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] (duration: 02m 00s) 2420759 wikitext text/x-wiki 59uytmmla613fki1yjgdjsvaswmozqf 2420760 2420759 2026-05-28T08:49:29Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294925|hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] (duration: 09m 20s) 2420760 wikitext text/x-wiki g6bjptagp44d4gmx508ygg5x5om4fxt 2420761 2420760 2026-05-28T08:49:37Z Stashbot 7414 joal@deploy1003: Started deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] 2420761 wikitext text/x-wiki 4inoxi97wvgu3kte1d8pn8igqoepkv8 2420763 2420761 2026-05-28T08:50:13Z Stashbot 7414 XioNoX: cr1-codfw# delete protocols bgp group fundraising family inet6 - T423384 2420763 wikitext text/x-wiki gv7gx06269jrurp6msrqzqkdww1qxrd 2420764 2420763 2026-05-28T08:52:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T419635)', diff saved to https://phabricator.wikimedia.org/P93327 and previous config saved to /var/cache/conftool/dbconfig/20260528-085216-fceratto.json 2420764 wikitext text/x-wiki 7gsmmmorq8ck5aminbp04n6tehwofw1 2420765 2420764 2026-05-28T08:56:32Z Stashbot 7414 joal@deploy1003: Finished deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] (duration: 06m 54s) 2420765 wikitext text/x-wiki 8ndj7m8c54glb2pzlfw1ciwhu171swv 2420766 2420765 2026-05-28T08:56:48Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): T427406 Testing on backup host 2420766 wikitext text/x-wiki 7mg56k6x9moxme2qn9o95chq7evx0o1 2420767 2420766 2026-05-28T08:57:11Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): T427406 Testing on backup host (duration: 00m 53s) 2420767 wikitext text/x-wiki omu1szabdw8d6ws5em09enqoucndzjt 2420768 2420767 2026-05-28T08:58:45Z Stashbot 7414 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] 2420768 wikitext text/x-wiki 5yzbimse390ve57caa79s0svuantv83 2420769 2420768 2026-05-28T08:59:39Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): T427406 Deploying to prod 2420769 wikitext text/x-wiki o10ya1zgxqd1nl5xeszw4r39nvq7vx1 2420770 2420769 2026-05-28T09:00:54Z Stashbot 7414 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] (duration: 02m 08s) 2420770 wikitext text/x-wiki dex3l6l3zqiw98j2mtnhtt0d8wtmtq6 2420771 2420770 2026-05-28T09:01:09Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance 2420771 wikitext text/x-wiki jbdm37ahchzcrjt779bmkaadwdys3xo 2420772 2420771 2026-05-28T09:01:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2216 (T426633)', diff saved to https://phabricator.wikimedia.org/P93328 and previous config saved to /var/cache/conftool/dbconfig/20260528-090114-fceratto.json 2420772 wikitext text/x-wiki gxpwwpz9siwg89joaj6gwhixjc93uvx 2420774 2420772 2026-05-28T09:02:03Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): T427406 Deploying to prod (duration: 02m 31s) 2420774 wikitext text/x-wiki irhzq80g3xp3xkm5zcp4m4gnj8q1x60 2420775 2420774 2026-05-28T09:02:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93329 and previous config saved to /var/cache/conftool/dbconfig/20260528-090224-fceratto.json 2420775 wikitext text/x-wiki jbyiktyvjodfmabl98bjxgudg1xzywu 2420777 2420775 2026-05-28T09:08:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T426633)', diff saved to https://phabricator.wikimedia.org/P93331 and previous config saved to /var/cache/conftool/dbconfig/20260528-090826-fceratto.json 2420777 wikitext text/x-wiki egqa9y9ph5w0r5xrsmx1wf7j85327mr 2420778 2420777 2026-05-28T09:09:13Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420778 wikitext text/x-wiki 15roklpdrqtp6huvdwa327pprq564xg 2420779 2420778 2026-05-28T09:09:43Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420779 wikitext text/x-wiki skxrpg9k2hgto1eteza4ovlnyfigu02 2420780 2420779 2026-05-28T09:12:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93332 and previous config saved to /var/cache/conftool/dbconfig/20260528-091231-fceratto.json 2420780 wikitext text/x-wiki hco4cl2vo8usq68br5s1na03t3jd995 2420781 2420780 2026-05-28T09:13:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.decommission for hosts pki-root1001.eqiad.wmnet 2420781 wikitext text/x-wiki k9bd0m1xvfloig00rrkc5gr1kt7sm3u 2420782 2420781 2026-05-28T09:14:01Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2420782 wikitext text/x-wiki gkcb0lc3efs2gc12uzj4i95h8osrwxu 2420783 2420782 2026-05-28T09:14:08Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2420783 wikitext text/x-wiki ilpcngwdek8devw0djtc322w76sss9u 2420784 2420783 2026-05-28T09:17:51Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.netbox 2420784 wikitext text/x-wiki m3imidlm7zey59hzt138f77hwwxhv2c 2420785 2420784 2026-05-28T09:17:54Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2420785 wikitext text/x-wiki k4cixyalafq4ls6bsg8mahp91p01ulc 2420786 2420785 2026-05-28T09:17:57Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Reboot completed 2420786 wikitext text/x-wiki c0s00hy1m6cgyxlz4k8dg2tlfy81yz8 2420787 2420786 2026-05-28T09:18:10Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2420787 wikitext text/x-wiki i6mdmhsem8u14jzpagtz3z5rbahrwt6 2420788 2420787 2026-05-28T09:18:23Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2420788 wikitext text/x-wiki fjrngtogxnvi9g948xg2ppozpw0ygbl 2420789 2420788 2026-05-28T09:18:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93334 and previous config saved to /var/cache/conftool/dbconfig/20260528-091834-fceratto.json 2420789 wikitext text/x-wiki ozjkrgj4meaex6usiye395wmcx8v8ff 2420790 2420789 2026-05-28T09:18:49Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2420790 wikitext text/x-wiki gnzkxaa8fk917m72dmdzyh8re6leboh 2420791 2420790 2026-05-28T09:18:59Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420791 wikitext text/x-wiki ek03yae8kjd7ahajfrbjux4pkazlt9u 2420792 2420791 2026-05-28T09:19:08Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420792 wikitext text/x-wiki nmylzdf80bx8dsix52ydddz3e97elmq 2420796 2420792 2026-05-28T09:22:08Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" 2420796 wikitext text/x-wiki 54l4jqyxlhoam5ecdk4sekwgj8hdv4d 2420797 2420796 2026-05-28T09:22:39Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" 2420797 wikitext text/x-wiki 07uxnbzzle7r5x3m5jfkdputffh8htj 2420798 2420797 2026-05-28T09:22:42Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2420798 wikitext text/x-wiki 5y1kg2mid1eczmoe8bfbmpgcy8lekom 2420799 2420798 2026-05-28T09:22:45Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki-root1001.eqiad.wmnet 2420799 wikitext text/x-wiki cv80rq4h868qfrp43l9tp0dkh3iwb34 2420800 2420799 2026-05-28T09:22:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T419635)', diff saved to https://phabricator.wikimedia.org/P93335 and previous config saved to /var/cache/conftool/dbconfig/20260528-092239-fceratto.json 2420800 wikitext text/x-wiki k86giqps4h6ag0ouqmvxbdkbqy1nr3k 2420801 2420800 2026-05-28T09:22:59Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance 2420801 wikitext text/x-wiki fgvtx21t01m6btdit46lwi54xve4tjt 2420802 2420801 2026-05-28T09:28:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93336 and previous config saved to /var/cache/conftool/dbconfig/20260528-092842-fceratto.json 2420802 wikitext text/x-wiki foyoqcbjfm8yifm8tkxmi6v21v7r4vm 2420807 2420802 2026-05-28T09:38:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T426633)', diff saved to https://phabricator.wikimedia.org/P93337 and previous config saved to /var/cache/conftool/dbconfig/20260528-093849-fceratto.json 2420807 wikitext text/x-wiki 8lcxtoctw8584dvorgkhqaunizwk9bo 2420808 2420807 2026-05-28T09:39:15Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance 2420808 wikitext text/x-wiki 7cl6lkq6hsie5io6219m1j9n0s2uq80 2420809 2420808 2026-05-28T09:39:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1169 (T426633)', diff saved to https://phabricator.wikimedia.org/P93338 and previous config saved to /var/cache/conftool/dbconfig/20260528-093920-fceratto.json 2420809 wikitext text/x-wiki smu24g73foszgl07qupfncsdu7wb8o7 2420810 2420809 2026-05-28T09:40:52Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294243|Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937|CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] 2420810 wikitext text/x-wiki 0w6giu09gc0dyl8py1wqansh28daoow 2420812 2420810 2026-05-28T09:42:36Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, stran: Backport for [[gerrit:1294243|Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937|CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420812 wikitext text/x-wiki b7qdw6akzdt6frrzrsluy5244k5i1yf 2420814 2420812 2026-05-28T09:43:42Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420814 wikitext text/x-wiki 0udzyuw5n3cepr8t3vm0e2smg1k7q1b 2420815 2420814 2026-05-28T09:43:54Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420815 wikitext text/x-wiki rb77eozibgcs21r14r1txowsiiy3ruu 2420816 2420815 2026-05-28T09:43:57Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420816 wikitext text/x-wiki 00cy8xmlfjznydqiif89bqjozh4cx9i 2420817 2420816 2026-05-28T09:44:07Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2420817 wikitext text/x-wiki q03p7zmz0hz0eey55grss89w32uqh4j 2420818 2420817 2026-05-28T09:44:20Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, stran: Continuing with deployment 2420818 wikitext text/x-wiki oznzfuzgovuyghfu8x2wdyy9ett972w 2420819 2420818 2026-05-28T09:48:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T426633)', diff saved to https://phabricator.wikimedia.org/P93339 and previous config saved to /var/cache/conftool/dbconfig/20260528-094807-fceratto.json 2420819 wikitext text/x-wiki axmqz4l3zrfmdtltqf2j9abgolcqvdx 2420820 2420819 2026-05-28T09:48:29Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294243|Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937|CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] (duration: 07m 37s) 2420820 wikitext text/x-wiki 1vhlnfqmkyxw5d529k0udjcy632wwk3 2420821 2420820 2026-05-28T09:52:45Z Stashbot 7414 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1290687|stream: webrequest.page_view (T426092 T426091)]] 2420821 wikitext text/x-wiki t817wtul500cuh567o3t3r69e3vof8i 2420822 2420821 2026-05-28T09:54:29Z Stashbot 7414 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1290687|stream: webrequest.page_view (T426092 T426091)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2420822 wikitext text/x-wiki acd5l8dk098pwlmz8w0osh7lua2jr2u 2420823 2420822 2026-05-28T09:55:16Z Stashbot 7414 javiermonton@deploy1003: javiermonton: Continuing with deployment 2420823 wikitext text/x-wiki md8k19a51uph4858mivw5ppz6uro3lv 2420824 2420823 2026-05-28T09:58:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93340 and previous config saved to /var/cache/conftool/dbconfig/20260528-095814-fceratto.json 2420824 wikitext text/x-wiki 3l7ym3ogl89sqmi17iva0w23l1tx7is 2420825 2420824 2026-05-28T09:59:26Z Stashbot 7414 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290687|stream: webrequest.page_view (T426092 T426091)]] (duration: 06m 41s) 2420825 wikitext text/x-wiki 07vnb7jbjpydr828vzrgnbnhnca2gzm 2420827 2420825 2026-05-28T10:08:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93341 and previous config saved to /var/cache/conftool/dbconfig/20260528-100822-fceratto.json 2420827 wikitext text/x-wiki tmnijdrs6ekmd7o2qrdzep59gj41573 2420828 2420827 2026-05-28T10:18:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T426633)', diff saved to https://phabricator.wikimedia.org/P93342 and previous config saved to /var/cache/conftool/dbconfig/20260528-101829-fceratto.json 2420828 wikitext text/x-wiki t724a9ec6ni7nrkfyukq517lupraef6 2420829 2420828 2026-05-28T10:18:55Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance 2420829 wikitext text/x-wiki kaqnhed0prh5zyd6jccfh2lzgh199bf 2420830 2420829 2026-05-28T10:19:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1186 (T426633)', diff saved to https://phabricator.wikimedia.org/P93343 and previous config saved to /var/cache/conftool/dbconfig/20260528-101900-fceratto.json 2420830 wikitext text/x-wiki 8jamdynz0lnp7t0es0ex5q97ja9a3hk 2420831 2420830 2026-05-28T10:27:00Z Stashbot 7414 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P1748 # T422392 2420831 wikitext text/x-wiki of3v5ajtbqncedmdcy0rm8yfo3uzk4e 2420832 2420831 2026-05-28T10:27:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T426633)', diff saved to https://phabricator.wikimedia.org/P93344 and previous config saved to /var/cache/conftool/dbconfig/20260528-102730-fceratto.json 2420832 wikitext text/x-wiki rsx5g4kfhcgzpk2ij40yl9u4m38dsm8 2420833 2420832 2026-05-28T10:28:42Z Stashbot 7414 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P14223 # T422264 2420833 wikitext text/x-wiki 757pao2t0er4g14ng8xs8hnxe3p3vkv 2420834 2420833 2026-05-28T10:29:48Z Stashbot 7414 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P13724 # T406971 2420834 wikitext text/x-wiki 2r0gp6wyoavgirtahlb2gs4v8tz8bdr 2420835 2420834 2026-05-28T10:37:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93345 and previous config saved to /var/cache/conftool/dbconfig/20260528-103738-fceratto.json 2420835 wikitext text/x-wiki cpy9b6x9yky5fh7fpkp1nqy67996j0e 2420836 2420835 2026-05-28T10:47:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93346 and previous config saved to /var/cache/conftool/dbconfig/20260528-104745-fceratto.json 2420836 wikitext text/x-wiki 3gzus7pt4aiwiei32dqsxgghrhzvx1w 2420837 2420836 2026-05-28T10:50:25Z Stashbot 7414 moritzm: update trixie netboot image for 13.5 point release T427072 2420837 wikitext text/x-wiki qmibh2yd1pxaycrh6q2x10y03o24goh 2420838 2420837 2026-05-28T10:55:43Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2420838 wikitext text/x-wiki meulk5c8pcvvfe79a22y5df8ityp0l7 2420839 2420838 2026-05-28T10:55:51Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2420839 wikitext text/x-wiki 1dyhev7isb68ubmn4dnjm4xw5eo8wdn 2420840 2420839 2026-05-28T10:55:58Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2420840 wikitext text/x-wiki duar25xlnr23i8y880fh7y0a1sryqkl 2420841 2420840 2026-05-28T10:56:05Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2420841 wikitext text/x-wiki g5hoo2q36ua6zxwtb6zzbib8r7fdcq5 2420842 2420841 2026-05-28T10:57:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T426633)', diff saved to https://phabricator.wikimedia.org/P93347 and previous config saved to /var/cache/conftool/dbconfig/20260528-105753-fceratto.json 2420842 wikitext text/x-wiki fwklabu6e40agqdzl3se458ayzyaoqp 2420843 2420842 2026-05-28T10:58:15Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance 2420843 wikitext text/x-wiki tpm10wfvnzc2hlsk4tfjui96d5iy3y0 2420844 2420843 2026-05-28T10:58:20Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie 2420844 wikitext text/x-wiki 5fwszpiz87n6y6ujk4hnv0sql8872vd 2420845 2420844 2026-05-28T10:58:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1195 (T426633)', diff saved to https://phabricator.wikimedia.org/P93348 and previous config saved to /var/cache/conftool/dbconfig/20260528-105820-fceratto.json 2420845 wikitext text/x-wiki 9awhs8nj5fgity0qk620zd0vvuhfhor 2420847 2420845 2026-05-28T11:05:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T426633)', diff saved to https://phabricator.wikimedia.org/P93349 and previous config saved to /var/cache/conftool/dbconfig/20260528-110536-fceratto.json 2420847 wikitext text/x-wiki pf1kam2l0foggem3uquscfop4bqfl8p 2420848 2420847 2026-05-28T11:15:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93350 and previous config saved to /var/cache/conftool/dbconfig/20260528-111543-fceratto.json 2420848 wikitext text/x-wiki efiqxdypb5yh5bioppw3bhp7k6g66ym 2420850 2420848 2026-05-28T11:25:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93351 and previous config saved to /var/cache/conftool/dbconfig/20260528-112551-fceratto.json 2420850 wikitext text/x-wiki tlcndasawsg5cxm9ial7p7r15wawcrp 2420851 2420850 2026-05-28T11:36:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T426633)', diff saved to https://phabricator.wikimedia.org/P93352 and previous config saved to /var/cache/conftool/dbconfig/20260528-113559-fceratto.json 2420851 wikitext text/x-wiki ow4fzolokjicqkds2l57hh18fs1zk3i 2420852 2420851 2026-05-28T11:36:21Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance 2420852 wikitext text/x-wiki rrrgt7iv4qt4t3mrqoiihhp4przvxoj 2420853 2420852 2026-05-28T11:36:30Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2420853 wikitext text/x-wiki nnp5f3eiavg8c8mljpaak3teshsqxwx 2420854 2420853 2026-05-28T11:36:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1196 (T426633)', diff saved to https://phabricator.wikimedia.org/P93353 and previous config saved to /var/cache/conftool/dbconfig/20260528-113635-fceratto.json 2420854 wikitext text/x-wiki 8mterm5h0wcg7wbeecg4slb4mnhwnfa 2420856 2420854 2026-05-28T11:45:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T426633)', diff saved to https://phabricator.wikimedia.org/P93354 and previous config saved to /var/cache/conftool/dbconfig/20260528-114500-fceratto.json 2420856 wikitext text/x-wiki jgei1rkq3hva951eg0kbmq1wfs1hqtc 2420858 2420856 2026-05-28T11:55:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93355 and previous config saved to /var/cache/conftool/dbconfig/20260528-115508-fceratto.json 2420858 wikitext text/x-wiki dlhygo5c3lr2ukuwgvddwit1t0dj64r 0 17290 2420580 2420168 2026-05-27T21:07:36Z Stashbot 7414 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1294425 2420580 wikitext text/x-wiki 70e64ge8bhi7viiszowzdbubzft61ia 2420670 2420580 2026-05-28T06:29:49Z Stashbot 7414 hashar: integration: delete integration-castor05 agent from Jenkins, replaced by integration-castor06 # T421114 2420670 wikitext text/x-wiki qtb8g4f7pkc74ugmgfa48bfrwgbsvuw 2420695 2420670 2026-05-28T07:05:32Z Stashbot 7414 hashar: integration: delete deployment-deploy04 agent from Jenkins controller. Batch job has been migrated to a systemd driven script # T256168 2420695 wikitext text/x-wiki guu3yne3db8uq6yjljubnqercwer9te 0 19208 2420389 2403162 2026-05-27T15:25:38Z AKlapper (WMF) 421 +Antivandalism extension 2420389 wikitext text/x-wiki jbk4tcbw577uyxf9xc21a59eywbrc3u 2420390 2420389 2026-05-27T15:27:47Z AKlapper (WMF) 421 2420390 wikitext text/x-wiki qqauiqyhcxrn44dvspejork2ntydaxy 0 20225 2420527 2387878 2026-05-27T19:04:11Z JAllemandou (WMF) 37709 Update hdfs deployment to make it clearer 2420527 wikitext text/x-wiki cycu3qny5fzbrjoya0vevj3u09vr27g 2420528 2420527 2026-05-27T19:04:44Z JAllemandou (WMF) 37709 Minor cosmetic adjustment 2420528 wikitext text/x-wiki rmk8p9s8m4wdeqmpgh5be83l3urtb71 0 33444 2420554 2368744 2026-05-27T20:10:09Z Cybercezaryc13 54582 2420554 wikitext text/x-wiki is5a3f3692jv11xoledk5l13g2xsxhp 0 46350 2420846 2375503 2026-05-28T11:01:47Z Ricert511 56005 /* */ 2420846 wikitext text/x-wiki celoh1n3ol67y2xza67z813twje8de9 3 220136 2420581 811859 2026-05-27T21:14:30Z ماهر سامي 49535 /* Request for JarBot source code under Toolforge Right to fork policy */ new section 2420581 wikitext text/x-wiki i1t2o93mgbp1fhfix2fe3yeul85kd9z 2420588 2420581 2026-05-27T21:40:55Z ماهر سامي 49535 /* Request for JarBot source code under Toolforge Right to fork policy */ Reply 2420588 wikitext text/x-wiki 67eo3geomllgqf0tpx9wxztyh6ew37x 0 440900 2420622 2420227 2026-05-28T00:06:40Z BryanDavis 1604 typo 2420622 wikitext text/x-wiki 74y6o3rwsx7exsx1y2yw5uj3b5k4tum 498 443946 2420513 2418802 2026-05-27T18:34:51Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 73fcbaa464 (extract mwoauth2 module) 2420513 wikitext text/x-wiki n8p6l1rw3fyo8skq5wr7ok58etq93g9 2420586 2420513 2026-05-27T21:34:28Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 994cfa51b3 (make mwoauth2 strictly typed) 2420586 wikitext text/x-wiki ic4bg2707tfbvcnel2zzth94n4ctoua 0 444380 2420250 2387411 2026-05-27T12:09:52Z Ottomata 88 /* No Capital letters - Use snake_case */ 2420250 wikitext text/x-wiki r1d25z7gzor7ob0cyvgvjtrhhhngokq 0 444636 2420646 2201611 2026-05-28T05:57:10Z Quiddity 1884 Quiddity moved page [[Data Platform/Data quality/Entrophy alarms]] to [[Data Platform/Data quality/Entropy alarms]]: Misspelled title 2201611 wikitext text/x-wiki 23u91lnwtkg30buxrvl13k2f6dctcao 2420648 2420646 2026-05-28T05:58:01Z Quiddity 1884 fix heading 2420648 wikitext text/x-wiki 0c5zq7cdrd7dmvae3uz8kw9mfx741yq 12 446155 2420649 2249394 2026-05-28T06:01:27Z Santhosh.thottingal 141 /* Tools using Rust */ 2420649 wikitext text/x-wiki tfjhncvi2lswim7io7868zwli7vxrbs 0 446855 2420794 2198275 2026-05-28T09:21:05Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Data Platform/Data quality/Entropy alarms]] 2420794 wikitext text/x-wiki fipqnjwv6pg6526jk41x3qt6zlf40gz 10 447289 2420675 1924490 2026-05-28T06:36:45Z Quiddity 1884 fix colors in dark-mode 2420675 wikitext text/x-wiki png9qq2r7lk5mpcso16m8eri0yny6ln 0 447343 2420650 2410931 2026-05-28T06:02:21Z Quiddity 1884 {{hatnote}} for meta-context 2420650 wikitext text/x-wiki jcrkg2e157hthi3c66dikvkaz9acppe 2420680 2420650 2026-05-28T06:46:01Z Quiddity 1884 use {{Navigation sidebar}} for better accessibility (enables font-scaling, and dark-mode) 2420680 wikitext text/x-wiki ma1b26aujpzc9g7c3gb99myl4ysgddh 2420681 2420680 2026-05-28T06:46:29Z Quiddity 1884 rm redundant title from navbox 2420681 wikitext text/x-wiki eow9rjkjd9l7jwhezhkuxjpyxb84shy 0 447760 2420443 2408669 2026-05-27T17:09:18Z TCipriani (WMF) 4895 2420443 wikitext text/x-wiki ofkl13sj9xfszhi3yt6ro52gzyaor0f 2420450 2420443 2026-05-27T17:15:46Z TCipriani (WMF) 4895 2420450 wikitext text/x-wiki a9lz7n07bwrup03z36713k56occ6qsd 0 447761 2420447 2414330 2026-05-27T17:15:28Z TCipriani (WMF) 4895 /* JVM */ 2420447 wikitext text/x-wiki 7ync59m3nz4ze059poge34ejcn3fwsg 0 448236 2420684 2349176 2026-05-28T06:55:11Z Quiddity 1884 fix heading levels 2420684 wikitext text/x-wiki gsalxyb51ayrieyz91nhcdyvc94eh2e 0 448240 2420679 2361584 2026-05-28T06:44:31Z Quiddity 1884 rm gap 2420679 wikitext text/x-wiki kflg4werefuovnbi6jnym5pw8dagyqt 2420682 2420679 2026-05-28T06:52:42Z Quiddity 1884 fix heading levels 2420682 wikitext text/x-wiki 7wssmj2ka4wzkwieq50eyra3gyac9ca 0 448262 2420696 2322523 2026-05-28T07:05:58Z Quiddity 1884 fix heading levels 2420696 wikitext text/x-wiki oarmaam8ealls8uzwrtwyr7kbduu02x 0 448359 2420686 2361583 2026-05-28T06:56:50Z Quiddity 1884 fix heading levels 2420686 wikitext text/x-wiki nn42zje3iuf7nm3amhkxgr1b3xf7yun 2 448673 2420247 2374479 2026-05-27T12:09:14Z Ottomata 88 /* Capital letters in data keys is a really bad idea | snake_case > camelCase */ 2420247 wikitext text/x-wiki q2jwcmjuthvxyld8wtaq1nwgoraatfl 0 449080 2420691 2333994 2026-05-28T07:03:57Z Quiddity 1884 fix heading levels 2420691 wikitext text/x-wiki h7fac2d07ankbl19uy1if069179gioj 0 449081 2420693 2321945 2026-05-28T07:04:35Z Quiddity 1884 fix heading levels 2420693 wikitext text/x-wiki bntm59bsjvkzjb3equ7122wgpvp5mua 0 449082 2420688 2177977 2026-05-28T07:00:05Z Quiddity 1884 fix heading levels 2420688 wikitext text/x-wiki l3kb4m9azmondrsd247lm5qfrni1nt7 2420689 2420688 2026-05-28T07:02:12Z Quiddity 1884 fix list 2420689 wikitext text/x-wiki me0chmlbn1hibmf71thf1e0hp5y527z 0 449083 2420690 2250539 2026-05-28T07:03:08Z Quiddity 1884 fix heading levels 2420690 wikitext text/x-wiki 6gm113b8kzlfax6znrh8s9a6o2vkwq8 0 449084 2420687 2163617 2026-05-28T06:57:57Z Quiddity 1884 fix heading levels 2420687 wikitext text/x-wiki mcvg1ctt9qjxfu0uq8o1u9wut50yg4l 0 449138 2420699 2323555 2026-05-28T07:06:53Z Quiddity 1884 fix heading levels 2420699 wikitext text/x-wiki sskely9muq1155f9ihn444rvuo68mrp 0 449160 2420337 2420076 2026-05-27T14:29:13Z Dexbot 30554 Bot: Updating the report 2420337 wikitext text/x-wiki 7o98vpksel9x25b48l23ok0vyw5bo6s 2420619 2420337 2026-05-28T00:00:32Z Dexbot 30554 Bot: Updating the report 2420619 wikitext text/x-wiki ipusmstuui39x3i7r2r9jl06r838k6i 2420653 2420619 2026-05-28T06:04:20Z Dexbot 30554 Bot: Updating the report 2420653 wikitext text/x-wiki kpahyndx98063vszcfki5bn0nsiafzl 2420674 2420653 2026-05-28T06:34:25Z Dexbot 30554 Bot: Updating the report 2420674 wikitext text/x-wiki l3u4z1wmkatmuu1ftrhi11tmgwp78ro 2420709 2420674 2026-05-28T07:19:31Z Dexbot 30554 Bot: Updating the report 2420709 wikitext text/x-wiki ndejw3jep505fd5lc3erzyc3bm43cwh 2420729 2420709 2026-05-28T07:58:38Z Dexbot 30554 Bot: Updating the report 2420729 wikitext text/x-wiki ma9nm42q8q75rt7hdi3j4abmnnw2u08 2420746 2420729 2026-05-28T08:37:44Z Dexbot 30554 Bot: Updating the report 2420746 wikitext text/x-wiki 8s1z275vih85074gcvkatul6vfsl7ou 2420762 2420746 2026-05-28T08:49:46Z Dexbot 30554 Bot: Updating the report 2420762 wikitext text/x-wiki 0996llik51ci77krlkt4sl58dik33uo 2420773 2420762 2026-05-28T09:01:49Z Dexbot 30554 Bot: Updating the report 2420773 wikitext text/x-wiki qys4yiu4ll0bmsmc5b9rvum20xm8ynn 2420811 2420773 2026-05-28T09:40:55Z Dexbot 30554 Bot: Updating the report 2420811 wikitext text/x-wiki cd6x2wpgz834hki44nwfm42qo2f5vkd 10 451506 2420676 2029070 2026-05-28T06:36:53Z Quiddity 1884 fix colors in dark-mode 2420676 wikitext text/x-wiki ewqdv3ae0qpzbd397sa4c1fxyipfwzl 0 452405 2420664 2361058 2026-05-28T06:18:16Z Quiddity 1884 add {{gerrit nav}} 2420664 wikitext text/x-wiki r9nkdjy517umg1y2eanwm2wrygu51im 0 452963 2420698 2180741 2026-05-28T07:06:26Z Quiddity 1884 fix heading levels 2420698 wikitext text/x-wiki hvfub3ijubs11wyb7ywwjlwm5qc9bw6 10 453139 2420669 2094967 2026-05-28T06:28:31Z Quiddity 1884 fix colors in dark-mode 2420669 wikitext text/x-wiki hz11rwjwl1iekgf7jbx4gtzkmhff7ly 116 453906 2420340 2419302 2026-05-27T14:33:24Z Gitlabaccountapprovalbot 37332 @thilio was approved. 2420340 wikitext text/x-wiki 4b3hqlffujipihoj93x7wr2fyf5siln 2420428 2420340 2026-05-27T16:30:19Z Gitlabaccountapprovalbot 37332 codeurluce was rejected. 2420428 wikitext text/x-wiki ixwdkt20h17z72znbp4idulzu6c0itw 2420550 2420428 2026-05-27T20:00:25Z Gitlabaccountapprovalbot 37332 vinitha was rejected. 2420550 wikitext text/x-wiki e8gn28i99as0aljmn70bobpihn8jkg1 0 454010 2420531 2386982 2026-05-27T19:25:33Z ADancy (WMF) 17694 /* Testing */ Mention T324164 2420531 wikitext text/x-wiki riw207w491pucuk7z6qv7rscsbr1uuy 0 454209 2420662 2156961 2026-05-28T06:17:01Z Quiddity 1884 add {{gerrit nav}} 2420662 wikitext text/x-wiki 1ysvf2cl1qafg9fmcl2zhkycdgpz8qn 0 454512 2420665 2180394 2026-05-28T06:18:26Z Quiddity 1884 add {{gerrit nav}} 2420665 wikitext text/x-wiki h1yz8kbt0q9f27w3vyazuzili2io5md 0 454595 2420335 2401073 2026-05-27T14:28:20Z CGoubert-WMF 32775 Clarify ambiguity between rest-gateway and internal ratelimit services 2420335 wikitext text/x-wiki 5gku6zmbyukxtgpjybrtdp9ttv7fubg 2420350 2420335 2026-05-27T14:44:56Z CGoubert-WMF 32775 CGoubert-WMF moved page [[Ratelimit]] to [[Internal ratelimit]] without leaving a redirect: Creating a disambiguation page for rate limits 2420335 wikitext text/x-wiki 5gku6zmbyukxtgpjybrtdp9ttv7fubg 0 455088 2420793 2194734 2026-05-28T09:21:03Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Data Platform/Data quality/Entropy alarms]] 2420793 wikitext text/x-wiki fipqnjwv6pg6526jk41x3qt6zlf40gz 0 455709 2420683 2352127 2026-05-28T06:53:48Z Quiddity 1884 fix heading levels 2420683 wikitext text/x-wiki farhemzceapxtxdu2fy0ulbb3k1be78 0 456448 2420666 2394349 2026-05-28T06:19:02Z Quiddity 1884 add {{gerrit nav}}; fix listgap 2420666 wikitext text/x-wiki eshkspuaaekh9dkspjaxkg5vphelfp7 0 458675 2420813 2412117 2026-05-28T09:43:07Z AYounsi (WMF) 10040 /* Show BGP routes rejected */ 2420813 wikitext text/x-wiki 1akczg4i8rngdwc5y9447kjaqouvk22 0 458775 2420704 2402207 2026-05-28T07:15:59Z Adam Wight (WMDE) 3064 /* Query templates */ Mention that the SQL flavor is Spark 2420704 wikitext text/x-wiki no5kjl5kb6z00juj9tmp7434vad5w8g 498 458787 2420609 2412685 2026-05-27T22:47:52Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26543153283 (https://github.com/cluebotng/component-configs/commits/b9f54eeee581a4f5a3788d3e16acbc76c7fa6fbb) 2420609 wikitext text/x-wiki faswmzyy9e8qnyb948yrn7obrn7tmrt 0 459037 2420694 2355239 2026-05-28T07:05:17Z Quiddity 1884 fix heading levels 2420694 wikitext text/x-wiki trn93vviyiyggkaqqwbx7767r87q546 498 459243 2420826 2418379 2026-05-28T10:07:20Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26568183905 (https://github.com/cluebotng/component-configs/commits/21b5bddbae9516620318fc278d5c3d1e3f295511) 2420826 wikitext text/x-wiki nozr1mz3vbyviz2l4ktlhsvpknnvmih 2420849 2420826 2026-05-28T11:16:57Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26571205214 (https://github.com/cluebotng/component-configs/commits/8f023304162d9582d5736568d9d7078c10b9041d) 2420849 wikitext text/x-wiki ssws2cics4gxmsvpq5i3bbuebqc1773 498 459244 2420855 2418285 2026-05-28T11:44:08Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26572448836 (https://github.com/cluebotng/component-configs/commits/fdea8d6284d14cd9ed835ab73506d0750af62de2) 2420855 wikitext text/x-wiki mdb9uhhspoch5d7275dbph34455mdc4 0 459290 2420532 2419198 2026-05-27T19:26:42Z JAllemandou (WMF) 37709 Add 2026-05-27 deployment 2420532 wikitext text/x-wiki 4jz4q2c6bvdyfo5jf8cvirn5tpygubc 2420685 2420532 2026-05-28T06:55:14Z KCVelaga 26729 /* Wednesday, May 27, 2026 */ fix change ID 2420685 wikitext text/x-wiki c9uihtxu2z3a5z4gd7pobic0ip0zznj 2420803 2420685 2026-05-28T09:31:09Z APizzata-WMF 47941 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2420803 wikitext text/x-wiki pa14pozlr0rb1xrqyosisfrilrbm4wp 2420804 2420803 2026-05-28T09:31:36Z APizzata-WMF 47941 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2420804 wikitext text/x-wiki pi74pclj5f0a5yx68aksu4z5olxdug3 2420805 2420804 2026-05-28T09:31:47Z APizzata-WMF 47941 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2420805 wikitext text/x-wiki kzoocdz5reabwrkbeuin3blrh9pfu92 2420806 2420805 2026-05-28T09:33:07Z APizzata-WMF 47941 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2420806 wikitext text/x-wiki mo06o075og5b0jmm06pd7d4ud7xxe1p 0 459608 2420360 2370105 2026-05-27T14:55:00Z CGoubert-WMF 32775 2420360 wikitext text/x-wiki gkhint5ihvt35gg0n041794ky7p0lud 2420361 2420360 2026-05-27T14:55:27Z CGoubert-WMF 32775 2420361 wikitext text/x-wiki ossdkhlwzbyzn97iaagzorgicoohymc 0 459609 2420325 2390817 2026-05-27T14:21:44Z CGoubert-WMF 32775 API gateway has been deprecated and taken out of the path 2420325 wikitext text/x-wiki 6pj0nqzbhjnhg7oxebhkhhj4pl3y4r5 2420429 2420325 2026-05-27T16:37:35Z DKinzler (WMF) 156 [[CDN#Rate-limiting]] 2420429 wikitext text/x-wiki 6g2tj35f49atwczzt3j7inh585l3j1k 0 459783 2420336 2412037 2026-05-27T14:28:49Z CGoubert-WMF 32775 Clarify ambiguity between rest-gateway and internal ratelimit services and add sequence diagram 2420336 wikitext text/x-wiki 18u02ol3sk1raqvkm4bfk61anous4g2 2420362 2420336 2026-05-27T14:55:48Z CGoubert-WMF 32775 2420362 wikitext text/x-wiki kficyqm6nw2akf7uzyewwy14cgnskzs 2420363 2420362 2026-05-27T14:59:20Z CGoubert-WMF 32775 2420363 wikitext text/x-wiki cle25hzpc0lq5r1k8dr74t9199nay7v 0 459928 2420663 2394325 2026-05-28T06:17:12Z Quiddity 1884 add {{gerrit nav}} 2420663 wikitext text/x-wiki rwzc9fir4p48g8u1ea8x1vb31qubq1r 0 460059 2420589 2414025 2026-05-27T21:50:03Z AHoelzl-WMF 39487 Added section about dbt docs generation 2420589 wikitext text/x-wiki c8fpvgyrlya8tc9mt9j2liv8xqq0co1 2420590 2420589 2026-05-27T21:51:12Z AHoelzl-WMF 39487 Changed section order, move documentation section up 2420590 wikitext text/x-wiki fwn69277xae1pma36b4mnqgo8c2vt2s 2420591 2420590 2026-05-27T21:52:42Z AHoelzl-WMF 39487 Language, clarifications 2420591 wikitext text/x-wiki i67872wde7bhp9cdcj5cnqaoe18v8ec 2420592 2420591 2026-05-27T21:53:28Z AHoelzl-WMF 39487 port clarification in the provided example 2420592 wikitext text/x-wiki cuwjru3atmyfx864zcgqrz32awq11ux 2420593 2420592 2026-05-27T21:54:28Z AHoelzl-WMF 39487 fix of prior change 2420593 wikitext text/x-wiki bg3vu2e2lbwiu5r0ls0esl8jy82swxg 2 460100 2420641 2416847 2026-05-28T05:15:39Z Quiddity 1884 rm {{DISPLAYTITLE}} - it only changes the letter-case per [[mw:Category:Pages with ignored display titles]] 2420641 wikitext text/x-wiki 7mpqvt1fuu6nc5rgg2lpktborgokml1 2 460101 2420645 2415568 2026-05-28T05:46:04Z Quiddity 1884 change colors to use Codex tokens, for automatic dark-mode compatibility 2420645 sanitized-css text/css q460p1k5fb7e8ej43k49w2uieffk5yd 0 460178 2420639 2416789 2026-05-28T05:14:53Z Quiddity 1884 rm {{DISPLAYTITLE}} - it only changes the letter-case per [[mw:Category:Pages with ignored display titles]] 2420639 wikitext text/x-wiki 6ugxpxmo5vp4qdy6jejxhr6qg4wxldo 2420642 2420639 2026-05-28T05:17:41Z Quiddity 1884 rm gap? 2420642 wikitext text/x-wiki brrnwl54ahtlrimqkk2mwd28g86w5lf 0 460192 2420638 2417075 2026-05-28T05:14:26Z Quiddity 1884 rm {{DISPLAYTITLE}} - it only changes the letter-case per [[mw:Category:Pages with ignored display titles]] 2420638 wikitext text/x-wiki tcuhpm54x6geatjb7onu4357hdanegh 0 460206 2420640 2417045 2026-05-28T05:15:13Z Quiddity 1884 rm {{DISPLAYTITLE}} - it only changes the letter-case per [[mw:Category:Pages with ignored display titles]] 2420640 wikitext text/x-wiki s77vvq49bbm9ahfve9sk222ya8itvi9 0 460226 2420355 2026-05-27T14:51:21Z CGoubert-WMF 32775 Create disambiguation page for Ratelimit 2420355 wikitext text/x-wiki bf8nczbl3aair72p20vcajwxe5loas9 2420383 2420355 2026-05-27T15:19:09Z CGoubert-WMF 32775 2420383 wikitext text/x-wiki 7gzs0632vt60yhnrd8kttru25s9ro8e 3 460227 2420391 2026-05-27T15:28:59Z StrikerBot 8475 Welcome to Toolforge! 2420391 wikitext text/x-wiki oi5388js6u8t62hbdmrvw4zdm8ju79o 0 460228 2420647 2026-05-28T05:57:10Z Quiddity 1884 Quiddity moved page [[Data Platform/Data quality/Entrophy alarms]] to [[Data Platform/Data quality/Entropy alarms]]: Misspelled title 2420647 wikitext text/x-wiki fipqnjwv6pg6526jk41x3qt6zlf40gz 1 460229 2420776 2026-05-28T09:03:17Z TobiasNx 55975 /* Version Control/Git in PAWS? */ new section 2420776 wikitext text/x-wiki jmsvgm75gudjttc9sikidn5hfttosxd 2 460230 2420857 2026-05-28T11:51:55Z SLyngshede-WMF 31501 Created page with "'''Notes'''" 2420857 wikitext text/x-wiki ie7sc9hvw84nwc3qhobedch6bvujq7f