Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.3 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 0 4108 2418745 2418726 2026-05-25T12:48:34Z ScheduleDeploymentBot 37566 Add [[gerrit:1290813]] to Monday, May 25 UTC afternoon backport window 2418745 wikitext text/x-wiki t3svggc9nwhtl2p8iikf4fe7wk7xj2m 2418777 2418745 2026-05-25T13:32:26Z ScheduleDeploymentBot 37566 Add [[gerrit:1293119]] to Monday, May 25 UTC afternoon backport window 2418777 wikitext text/x-wiki cnkblg8soa35cd178a7zbbe36g2u10h 2418819 2418777 2026-05-25T14:15:34Z ScheduleDeploymentBot 37566 Add [[gerrit:1285523]] to Monday, May 25 UTC late backport window 2418819 wikitext text/x-wiki sd2kz4xfn0is1d6z91p2tcva1h60ccs 2418913 2418819 2026-05-25T20:13:58Z ScheduleDeploymentBot 37566 Add [[gerrit:1293161]] to Tuesday, May 26 UTC afternoon backport window 2418913 wikitext text/x-wiki 1bdxe8mboedbfljd8qbt9oy8xxunzbk 2418914 2418913 2026-05-25T20:14:38Z ScheduleDeploymentBot 37566 Add [[gerrit:1293161]] to Tuesday, May 26 UTC late backport window 2418914 wikitext text/x-wiki o0ipu2oqojnb7x6i6ckhuccsbtnmvlz 2418917 2418914 2026-05-25T20:29:34Z ASanford-WMF 52089 /* Tuesday, May 26 */ 2418917 wikitext text/x-wiki 5otvne5oca1i55wr3xiht9xfn0ilps3 2418926 2418917 2026-05-25T23:51:17Z ScheduleDeploymentBot 37566 Add [[gerrit:1293177]] to Tuesday, May 26 UTC afternoon backport window 2418926 wikitext text/x-wiki 9cu3hfde4jqrek8plopxiv5eiw6y1ue 2419138 2418926 2026-05-26T09:15:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1293662]] to Tuesday, May 26 UTC afternoon backport window 2419138 wikitext text/x-wiki rb9fs760of8n02gveqmo1faif7qlukr 0 7919 2418731 2418730 2026-05-25T12:01:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T426633)', diff saved to https://phabricator.wikimedia.org/P92862 and previous config saved to /var/cache/conftool/dbconfig/20260525-120145-fceratto.json 2418731 wikitext text/x-wiki qh8no2ub8xvo56z4p77t5k2dlcef33r 2418732 2418731 2026-05-25T12:10:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418732 wikitext text/x-wiki apchxj26rakncolgpq1dldv47368iyd 2418733 2418732 2026-05-25T12:11:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92863 and previous config saved to /var/cache/conftool/dbconfig/20260525-121153-fceratto.json 2418733 wikitext text/x-wiki tn9hwuhkbvj300h1e9e0vlp6zgc2ngj 2418734 2418733 2026-05-25T12:17:10Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418734 wikitext text/x-wiki 6gr267psts8y3lr1pslh1gs27p8yogg 2418735 2418734 2026-05-25T12:17:13Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418735 wikitext text/x-wiki 23mqepcyarb8tbrwetpmmmoyc2yhfgw 2418736 2418735 2026-05-25T12:17:16Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1037.eqiad.wmnet 2418736 wikitext text/x-wiki fvtsnfjvlyfg9yiy371jbrapeh7m0s4 2418737 2418736 2026-05-25T12:22:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92864 and previous config saved to /var/cache/conftool/dbconfig/20260525-122201-fceratto.json 2418737 wikitext text/x-wiki f71zyl7p2bbglu7qkms2iu12uger31j 2418738 2418737 2026-05-25T12:32:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T426633)', diff saved to https://phabricator.wikimedia.org/P92865 and previous config saved to /var/cache/conftool/dbconfig/20260525-123208-fceratto.json 2418738 wikitext text/x-wiki 8sh7ne2flan36dv5i18iibjcfltt8eo 2418739 2418738 2026-05-25T12:32:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance 2418739 wikitext text/x-wiki 6f3m5vp31ey7quav5i7zsv9cbnlloz3 2418740 2418739 2026-05-25T12:32:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 (T426633)', diff saved to https://phabricator.wikimedia.org/P92866 and previous config saved to /var/cache/conftool/dbconfig/20260525-123239-fceratto.json 2418740 wikitext text/x-wiki a2ryuxncinqnq7in66848139jzvpyts 2418741 2418740 2026-05-25T12:39:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T426633)', diff saved to https://phabricator.wikimedia.org/P92867 and previous config saved to /var/cache/conftool/dbconfig/20260525-123927-fceratto.json 2418741 wikitext text/x-wiki 7i0f32ewljzjw3h9h6e9g8tnu179fss 2418742 2418741 2026-05-25T12:39:45Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1038.eqiad.wmnet 2418742 wikitext text/x-wiki 6decjdf33cq0wr5ehnb2xesvvnetllt 2418743 2418742 2026-05-25T12:39:54Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply 2418743 wikitext text/x-wiki q3jm41oz9b74it08z8410kg812ojdud 2418744 2418743 2026-05-25T12:40:16Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply 2418744 wikitext text/x-wiki 41c7wxhhzs55e87atwlybbhtttskn3v 2418746 2418744 2026-05-25T12:49:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92868 and previous config saved to /var/cache/conftool/dbconfig/20260525-124934-fceratto.json 2418746 wikitext text/x-wiki 0hyjhu8aovd61uuieon4ax8450z9jhm 2418747 2418746 2026-05-25T12:49:57Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Reboot 2418747 wikitext text/x-wiki o08wqg6v0esa96cn5gvvq2iyifzod44 2418748 2418747 2026-05-25T12:53:46Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply 2418748 wikitext text/x-wiki b438zmadncj04bwwpr02lss7b4ylc21 2418749 2418748 2026-05-25T12:54:20Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply 2418749 wikitext text/x-wiki r899cgopij9u8d86n95x1yv9b49i64s 2418750 2418749 2026-05-25T12:54:29Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot 2418750 wikitext text/x-wiki mog3ra0saszaoxm89y9kq0oys7d737q 2418751 2418750 2026-05-25T12:54:36Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db1162: Reboot 2418751 wikitext text/x-wiki pzszggeog500wytaz7kwd4n9yalhezg 2418752 2418751 2026-05-25T12:56:13Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply 2418752 wikitext text/x-wiki qk8lxsn2um18m81f5y0q1mnwg4xmm6l 2418753 2418752 2026-05-25T12:56:46Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply 2418753 wikitext text/x-wiki hhvm5sizxmkh4hjufsogb259s6hnf00 2418754 2418753 2026-05-25T12:58:56Z Stashbot 7414 kart_: Updated cxserver to 2026-05-24-103047-production (T426808, T373418) 2418754 wikitext text/x-wiki 4nsa17ny66qqi1t6ghqry8wg9ocmiax 2418755 2418754 2026-05-25T12:59:04Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot 2418755 wikitext text/x-wiki 2nw5cm1b5fh6a1uq6x8lc704yjixkef 2418756 2418755 2026-05-25T12:59:11Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reboot 2418756 wikitext text/x-wiki bhncnocazog5btc3l2qcy1dd2l49o08 2418757 2418756 2026-05-25T12:59:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92870 and previous config saved to /var/cache/conftool/dbconfig/20260525-125942-fceratto.json 2418757 wikitext text/x-wiki 8roh9tefmvewcuul39ffnmmveo6x13o 2418758 2418757 2026-05-25T12:59:48Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1162: Reboot 2418758 wikitext text/x-wiki rvmob5ildkz25iyriw7fsncf9r8of3n 2418759 2418758 2026-05-25T13:03:32Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1291966|Modify various configurations for English Wikibooks (T426992)]] 2418759 wikitext text/x-wiki bpm178hsde8sh6nahtqm2o7eumww5ze 2418760 2418759 2026-05-25T13:07:42Z Stashbot 7414 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1291966|Modify various configurations for English Wikibooks (T426992)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2418760 wikitext text/x-wiki az7lcoh2676qja8cj4zd6trmfrhlwj6 2418761 2418760 2026-05-25T13:09:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T426633)', diff saved to https://phabricator.wikimedia.org/P92872 and previous config saved to /var/cache/conftool/dbconfig/20260525-130950-fceratto.json 2418761 wikitext text/x-wiki dfkxp0rzzglb0jbcnnv3v0rsl1llqcy 2418762 2418761 2026-05-25T13:10:18Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance 2418762 wikitext text/x-wiki muecwoc08qrhuke3zube5oe7l4lxnoe 2418763 2418762 2026-05-25T13:10:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 (T426633)', diff saved to https://phabricator.wikimedia.org/P92873 and previous config saved to /var/cache/conftool/dbconfig/20260525-131023-fceratto.json 2418763 wikitext text/x-wiki obxr63zif3k1ri6mnknub8v9zwf6v33 2418764 2418763 2026-05-25T13:12:30Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418764 wikitext text/x-wiki bbevr7f833c3jcl2qodfi1756pa713t 2418765 2418764 2026-05-25T13:12:48Z Stashbot 7414 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment 2418765 wikitext text/x-wiki azlabqsl6y7c3hzmqh6olh9dz3iee3c 2418766 2418765 2026-05-25T13:17:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T426633)', diff saved to https://phabricator.wikimedia.org/P92875 and previous config saved to /var/cache/conftool/dbconfig/20260525-131714-fceratto.json 2418766 wikitext text/x-wiki 57jid7wyohc35595njb12nv3k46wi08 2418767 2418766 2026-05-25T13:19:25Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291966|Modify various configurations for English Wikibooks (T426992)]] (duration: 15m 53s) 2418767 wikitext text/x-wiki 3xjg9g4pat9vvtqr9oawnpooakf4ki9 2418768 2418767 2026-05-25T13:20:12Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293094|Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953|Update logo, wordmark and tagline for zghwiki (T426406)]] 2418768 wikitext text/x-wiki 9dqy6wkq8f3r2vh863z8c8rt61lj85o 2418769 2418768 2026-05-25T13:20:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418769 wikitext text/x-wiki irwiglkt6qj56b3b4nlse2bhul82tly 2418770 2418769 2026-05-25T13:21:54Z Stashbot 7414 mszwarc@deploy1003: mszwarc, jhsoby: Backport for [[gerrit:1293094|Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953|Update logo, wordmark and tagline for zghwiki (T426406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2418770 wikitext text/x-wiki i9lnprkgubl8utup9ui8vlnvvfvmbo8 2418771 2418770 2026-05-25T13:23:38Z Stashbot 7414 mszwarc@deploy1003: mszwarc, jhsoby: Continuing with deployment 2418771 wikitext text/x-wiki a5435bgrwltllp694o8ohyidl1cse3i 2418773 2418771 2026-05-25T13:27:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92876 and previous config saved to /var/cache/conftool/dbconfig/20260525-132722-fceratto.json 2418773 wikitext text/x-wiki fqsgy9x18vpz03cuuimrbspsvmqvrbh 2418774 2418773 2026-05-25T13:27:50Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293094|Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953|Update logo, wordmark and tagline for zghwiki (T426406)]] (duration: 07m 43s) 2418774 wikitext text/x-wiki dnszh8m2vdzz3i0dc32m2rmpum9bn80 2418775 2418774 2026-05-25T13:30:13Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290813|Article Guidance: enable experiment on phase 2 wikis (T426871)]] 2418775 wikitext text/x-wiki 8c6bz1eoabtbzj5gc1zinz83mxq5o4h 2418776 2418775 2026-05-25T13:31:55Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290813|Article Guidance: enable experiment on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2418776 wikitext text/x-wiki girjofx4ijq5vd8hmj9nyrl3bqmmucc 2418778 2418776 2026-05-25T13:32:42Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418778 wikitext text/x-wiki g6lrg46rekwb2zw856r9zwob8t8xxq9 2418779 2418778 2026-05-25T13:32:45Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418779 wikitext text/x-wiki bjwwhjcax9izge9oekfu54ega80916s 2418780 2418779 2026-05-25T13:32:47Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1038.eqiad.wmnet 2418780 wikitext text/x-wiki q3lwxmdcpk78qf598cca49qbf3bl25f 2418781 2418780 2026-05-25T13:33:15Z Stashbot 7414 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2418781 wikitext text/x-wiki 9od9rxkd43e6aa843fi1cekw2ncwuiu 2418782 2418781 2026-05-25T13:34:16Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2418782 wikitext text/x-wiki mb3gri9vby76ky1dt9a1bpdb8p8brz8 2418783 2418782 2026-05-25T13:37:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92878 and previous config saved to /var/cache/conftool/dbconfig/20260525-133729-fceratto.json 2418783 wikitext text/x-wiki md0vime4yggxz0nj7cso67tvtfukugz 2418784 2418783 2026-05-25T13:38:06Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin 2418784 wikitext text/x-wiki emcyq4gjikqk8va84eu016c3z18i6gg 2418785 2418784 2026-05-25T13:38:20Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin 2418785 wikitext text/x-wiki 65zwl7tdgixhc9ppx1gtxdden0yzmy9 2418786 2418785 2026-05-25T13:38:27Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290813|Article Guidance: enable experiment on phase 2 wikis (T426871)]] (duration: 08m 14s) 2418786 wikitext text/x-wiki kah4umzw6jle42049s6oozw2433a458 2418787 2418786 2026-05-25T13:40:00Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad 2418787 wikitext text/x-wiki o1r9273rttniio3x6rvkemfdvi4no3k 2418788 2418787 2026-05-25T13:40:03Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad 2418788 wikitext text/x-wiki locbe10ouxo4fozsthltnhbg4a3yvyb 2418789 2418788 2026-05-25T13:43:49Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293119|Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] 2418789 wikitext text/x-wiki 0wv8knlg4kjvy6yz6sdqmgpdgepv5x2 2418790 2418789 2026-05-25T13:45:15Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Reboot 2418790 wikitext text/x-wiki 8hhpol7f8o7vhzkzsxlccl9hwfqg9ld 2418791 2418790 2026-05-25T13:45:34Z Stashbot 7414 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1293119|Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2418791 wikitext text/x-wiki 47qk0mp9lgy3ec7pr68hfyh6gnxd4zu 2418792 2418791 2026-05-25T13:47:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T426633)', diff saved to https://phabricator.wikimedia.org/P92880 and previous config saved to /var/cache/conftool/dbconfig/20260525-134737-fceratto.json 2418792 wikitext text/x-wiki br84tqeusvdbefqc0bboj47bcaxmld0 2418793 2418792 2026-05-25T13:47:43Z Stashbot 7414 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2418793 wikitext text/x-wiki kni6cnlhhrd7oi3ewixj410i0rc7ouy 2418794 2418793 2026-05-25T13:47:56Z Stashbot 7414 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment 2418794 wikitext text/x-wiki p0wy2lgazqngjfvienp7go412v7glmt 2418795 2418794 2026-05-25T13:48:01Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance 2418795 wikitext text/x-wiki ev09v1ssj9jb9wmy4wcys4d4vl6lkjq 2418796 2418795 2026-05-25T13:48:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 (T426633)', diff saved to https://phabricator.wikimedia.org/P92881 and previous config saved to /var/cache/conftool/dbconfig/20260525-134807-fceratto.json 2418796 wikitext text/x-wiki qqh7nuyvlg9ic9qq1wosl5nb6cqr8gs 2418797 2418796 2026-05-25T13:49:01Z Stashbot 7414 kart_: Updated Recommendation API to 2026-05-21-044522-production 2418797 wikitext text/x-wiki 434dru6oz79up56pezcoc9uru07a0dh 2418800 2418797 2026-05-25T13:50:24Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet 2418800 wikitext text/x-wiki taxztxyrwwr0nxmbjgf944asnx0suca 2418801 2418800 2026-05-25T13:50:34Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet 2418801 wikitext text/x-wiki hq2a41lh0gzzna0dwudrpajf1p2khwj 2418803 2418801 2026-05-25T13:51:37Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet 2418803 wikitext text/x-wiki 9firca5zebbx62fr0endxyfinn5euks 2418804 2418803 2026-05-25T13:51:43Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet 2418804 wikitext text/x-wiki 5ftwtzswlqu1pttdejqjvi2frt0y40c 2418805 2418804 2026-05-25T13:52:06Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293119|Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] (duration: 09m 43s) 2418805 wikitext text/x-wiki r4ohqt2bb2kmxw8huma7w36zavp4q59 2418806 2418805 2026-05-25T13:52:28Z Stashbot 7414 Msz2001: Everything deployed, UTC afternoon config+backport window done 2418806 wikitext text/x-wiki aflng05bzy4nzirq69duhl5z3kjgzlp 2418807 2418806 2026-05-25T13:55:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T426633)', diff saved to https://phabricator.wikimedia.org/P92882 and previous config saved to /var/cache/conftool/dbconfig/20260525-135458-fceratto.json 2418807 wikitext text/x-wiki hmeq9ub2ylydmy6q5erca44820saqsg 2418808 2418807 2026-05-25T13:58:02Z Stashbot 7414 sukhe: sudo cumin 'A:lvs and A:eqiad' 'run-puppet-agent --enable "adding new ml-serve (grpc) T424049": NOOP change, since service is codfw only 2418808 wikitext text/x-wiki 9kurympldbmebcvi596bk9y1zhmatk4 2418809 2418808 2026-05-25T13:59:23Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1039.eqiad.wmnet 2418809 wikitext text/x-wiki 5ebpxjgc38wikcdrwcbt6sz9t5t9uqx 2418810 2418809 2026-05-25T14:00:19Z Stashbot 7414 sukhe: sudo cumin 'A:lvs and A:lvs-secondary-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) T424049"' 2418810 wikitext text/x-wiki 9f9ltf9xf85fvecfd4yl1lo94co89ac 2418811 2418810 2026-05-25T14:02:06Z Stashbot 7414 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service 2418811 wikitext text/x-wiki pcpoh1eyxywm4a6vlougibj4zr3j1ca 2418812 2418811 2026-05-25T14:02:21Z Stashbot 7414 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service": T424049 2418812 wikitext text/x-wiki 0tsvi9kdbu5zrq6v2lblpz7poinq196 2418813 2418812 2026-05-25T14:03:56Z Stashbot 7414 sukhe: sudo cumin 'A:lvs and A:lvs-low-traffic-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) T424049"' 2418813 wikitext text/x-wiki 4m7jai3cvkf9vfru07ot11w5lr7civ5 2418814 2418813 2026-05-25T14:05:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92884 and previous config saved to /var/cache/conftool/dbconfig/20260525-140505-fceratto.json 2418814 wikitext text/x-wiki hryee5c3t1i5lk4ozlseoj9nktlgxlr 2418815 2418814 2026-05-25T14:05:13Z Stashbot 7414 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service: T424049 2418815 wikitext text/x-wiki dfh2fix3a02bp08xgdaxogtcu08q70r 2418816 2418815 2026-05-25T14:06:18Z Stashbot 7414 sukhe: curl localhost:9090/pools/inference-staging-grpc_30051 shows ml-staging200[1-3].codfw.wmnet as enabled and pooled: T424049 2418816 wikitext text/x-wiki oj8lkzpas44lczsvte7hivg3qz48wve 2418817 2418816 2026-05-25T14:12:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418817 wikitext text/x-wiki j23di6sdc6bbsartu8fn15lmb2ee0s1 2418818 2418817 2026-05-25T14:15:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92885 and previous config saved to /var/cache/conftool/dbconfig/20260525-141513-fceratto.json 2418818 wikitext text/x-wiki 65m88jzhd9pds1uf5ajvoldwouij1um 2418820 2418818 2026-05-25T14:25:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T426633)', diff saved to https://phabricator.wikimedia.org/P92887 and previous config saved to /var/cache/conftool/dbconfig/20260525-142520-fceratto.json 2418820 wikitext text/x-wiki juxikib4ux6no0txpbo84duh86u3b5b 2418821 2418820 2026-05-25T14:25:45Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance 2418821 wikitext text/x-wiki kbg5an40mivqmyjdoqdkusy8j27z345 2418822 2418821 2026-05-25T14:25:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 (T426633)', diff saved to https://phabricator.wikimedia.org/P92888 and previous config saved to /var/cache/conftool/dbconfig/20260525-142551-fceratto.json 2418822 wikitext text/x-wiki 1pibke0h32imhm1xa1v0q30tf13m1xu 2418823 2418822 2026-05-25T14:31:39Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet 2418823 wikitext text/x-wiki gtae1wyuz4281u7hjggpiumtv73jk7k 2418824 2418823 2026-05-25T14:32:38Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet 2418824 wikitext text/x-wiki pybr0lwkf78ey51sm485ivsmlvi5eri 2418825 2418824 2026-05-25T14:32:47Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet 2418825 wikitext text/x-wiki qzqiovmjdywfme5et0qgteclqi6s1dj 2418826 2418825 2026-05-25T14:32:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T426633)', diff saved to https://phabricator.wikimedia.org/P92889 and previous config saved to /var/cache/conftool/dbconfig/20260525-143246-fceratto.json 2418826 wikitext text/x-wiki gbks6vjef4ljt2aaygry25sj3y4ba59 2418827 2418826 2026-05-25T14:33:16Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet 2418827 wikitext text/x-wiki 7c1aaekn32tckd4wj06j35lxmtrjqih 2418829 2418827 2026-05-25T14:42:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92890 and previous config saved to /var/cache/conftool/dbconfig/20260525-144253-fceratto.json 2418829 wikitext text/x-wiki jyg94db7dfcgqw1at0m20mlsi5hv3x7 2418831 2418829 2026-05-25T14:53:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92891 and previous config saved to /var/cache/conftool/dbconfig/20260525-145301-fceratto.json 2418831 wikitext text/x-wiki sns5gkib4bse3cc77uh04qa7aud23mm 2418832 2418831 2026-05-25T15:03:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T426633)', diff saved to https://phabricator.wikimedia.org/P92892 and previous config saved to /var/cache/conftool/dbconfig/20260525-150309-fceratto.json 2418832 wikitext text/x-wiki 9lryt79n7dm1lfvmlbjuj08dlg4zl3g 2418833 2418832 2026-05-25T15:11:28Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet 2418833 wikitext text/x-wiki g7im8h3svlqsyqlolbdtlxqj3amfw5g 2418834 2418833 2026-05-25T15:13:00Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet 2418834 wikitext text/x-wiki 4b05ywym7hyvsw2o38yp4glaca4qhh2 2418836 2418834 2026-05-25T15:15:01Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet 2418836 wikitext text/x-wiki m70jiphto57f5cfmfpah8zgts2aavyl 2418837 2418836 2026-05-25T15:15:03Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet 2418837 wikitext text/x-wiki ciztucsji0t6l6w2qayw8avmeg5m3tr 2418838 2418837 2026-05-25T15:17:22Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1013 from dbctl T427190', diff saved to https://phabricator.wikimedia.org/P92893 and previous config saved to /var/cache/conftool/dbconfig/20260525-151718-marostegui.json 2418838 wikitext text/x-wiki 5lkao5vhusgrnj3jl3lhwsxta5vh656 2418839 2418838 2026-05-25T15:27:51Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418839 wikitext text/x-wiki l2l59w58gzjeilkuomltom1j0hpwduw 2418840 2418839 2026-05-25T15:29:05Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418840 wikitext text/x-wiki 10lj9fnhl9rz8c5bsvae1pmascf91qh 2418841 2418840 2026-05-25T15:29:08Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418841 wikitext text/x-wiki dnchw2prnlqxbfg8b8jlvi51n6jp113 2418842 2418841 2026-05-25T15:29:10Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1039.eqiad.wmnet 2418842 wikitext text/x-wiki l0ghpyeifnz167e0s8qpuhrz35gs7pq 2418843 2418842 2026-05-25T15:29:30Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1040.eqiad.wmnet 2418843 wikitext text/x-wiki nz0p1400afsb1iwb3gzpgp9sa6iaxld 2418844 2418843 2026-05-25T15:51:12Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet 2418844 wikitext text/x-wiki ftip3x3eezrt3mv7ju16kv9adix5wgd 2418845 2418844 2026-05-25T15:52:46Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet 2418845 wikitext text/x-wiki oson7glctdjvkogw0vxillzsadarnv6 2418846 2418845 2026-05-25T15:57:24Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet 2418846 wikitext text/x-wiki 9hwtt34aeatea6r6uj67bguugmhnno7 2418847 2418846 2026-05-25T15:57:30Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet 2418847 wikitext text/x-wiki l9bnfxspgh334bfgi5a11gbh9yimjzm 2418848 2418847 2026-05-25T15:59:24Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2249.codfw.wmnet with reason: Maintenance 2418848 wikitext text/x-wiki r2o30jc83qvooxhow7tamdewatfkcwa 2418849 2418848 2026-05-25T15:59:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2249 (T426633)', diff saved to https://phabricator.wikimedia.org/P92894 and previous config saved to /var/cache/conftool/dbconfig/20260525-155930-fceratto.json 2418849 wikitext text/x-wiki ir2iisob08jxtr7mwf1rn851eo4dxn8 2418850 2418849 2026-05-25T16:02:12Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418850 wikitext text/x-wiki rrf3jrto7wg9c8vra1kd6w6v0qnaq9i 2418851 2418850 2026-05-25T16:04:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 (T426633)', diff saved to https://phabricator.wikimedia.org/P92895 and previous config saved to /var/cache/conftool/dbconfig/20260525-160450-fceratto.json 2418851 wikitext text/x-wiki 99tua5c96ali1y1psg0dmna81x88kh3 2418852 2418851 2026-05-25T16:14:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92896 and previous config saved to /var/cache/conftool/dbconfig/20260525-161457-fceratto.json 2418852 wikitext text/x-wiki en2g3ynqw932ckqzkmexvhgp4s2d2ae 2418853 2418852 2026-05-25T16:16:37Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418853 wikitext text/x-wiki m354wkx8k29t8imtc38h5hje7kcdw2h 2418854 2418853 2026-05-25T16:20:21Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418854 wikitext text/x-wiki 08d1hfximodgw5ay34yzh333d7w0v65 2418855 2418854 2026-05-25T16:20:24Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418855 wikitext text/x-wiki e9pso0eelkbnl295kq40173383wci9t 2418856 2418855 2026-05-25T16:20:26Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1040.eqiad.wmnet 2418856 wikitext text/x-wiki lq4hv68a4tt3r2onbuwphn4ka3j7jcw 2418857 2418856 2026-05-25T16:20:36Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1041.eqiad.wmnet 2418857 wikitext text/x-wiki bss4lwcgdfwtlm8kvk1y31u9v30y435 2418858 2418857 2026-05-25T16:25:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92897 and previous config saved to /var/cache/conftool/dbconfig/20260525-162505-fceratto.json 2418858 wikitext text/x-wiki 3if70ql2i706rjtonorv1wvuoxbi7et 2418859 2418858 2026-05-25T16:26:38Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418859 wikitext text/x-wiki 049jmfajantqihxz5wnwln69b57a272 2418860 2418859 2026-05-25T16:31:00Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet 2418860 wikitext text/x-wiki 6iskgy0cv42sik6bqvaay3t3lxitkt2 2418861 2418860 2026-05-25T16:34:32Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet 2418861 wikitext text/x-wiki 3zum9umarr6ji3qq20e5nxiaqxxry8l 2418862 2418861 2026-05-25T16:35:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 (T426633)', diff saved to https://phabricator.wikimedia.org/P92898 and previous config saved to /var/cache/conftool/dbconfig/20260525-163512-fceratto.json 2418862 wikitext text/x-wiki 9iou65i428suwbtdgxu79s5ums1me0u 2418863 2418862 2026-05-25T16:35:54Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance 2418863 wikitext text/x-wiki q49a9b919a973dru2cw9niktj8uwor6 2418864 2418863 2026-05-25T16:36:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2158 (T426633)', diff saved to https://phabricator.wikimedia.org/P92899 and previous config saved to /var/cache/conftool/dbconfig/20260525-163559-fceratto.json 2418864 wikitext text/x-wiki bbkq8ue4g3tr7nacevct657ndtlkt35 2418866 2418864 2026-05-25T16:39:31Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet 2418866 wikitext text/x-wiki p7799vkhu1ofgx7yx5y5j73df78n1b4 2418867 2418866 2026-05-25T16:40:10Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet 2418867 wikitext text/x-wiki ly34z7wv4pvd8k026w6u8se8iw6fcf3 2418868 2418867 2026-05-25T16:41:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418868 wikitext text/x-wiki n2n1fb7wkoa0u4l7ddoxgsme2vte851 2418869 2418868 2026-05-25T16:42:17Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418869 wikitext text/x-wiki l6sbux0optbp7a5typdbz1v0lxh33wf 2418870 2418869 2026-05-25T16:42:19Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418870 wikitext text/x-wiki 589cn98p1y4z3uih2w8wqclnzaucyet 2418871 2418870 2026-05-25T16:42:22Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1041.eqiad.wmnet 2418871 wikitext text/x-wiki hws28t1590ffbq7nfscf5vse9ohe432 2418872 2418871 2026-05-25T16:42:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T426633)', diff saved to https://phabricator.wikimedia.org/P92900 and previous config saved to /var/cache/conftool/dbconfig/20260525-164247-fceratto.json 2418872 wikitext text/x-wiki 5nb8vj8u6g7tmplzrglrkohexmx3pg5 2418873 2418872 2026-05-25T16:51:15Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1042.eqiad.wmnet 2418873 wikitext text/x-wiki ao5o11t598t6khhc88p2g0ij0m6500b 2418874 2418873 2026-05-25T16:52:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92901 and previous config saved to /var/cache/conftool/dbconfig/20260525-165255-fceratto.json 2418874 wikitext text/x-wiki 5uavd625q24zesjttmuhv5hcusdrnvv 2418875 2418874 2026-05-25T17:03:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92902 and previous config saved to /var/cache/conftool/dbconfig/20260525-170302-fceratto.json 2418875 wikitext text/x-wiki sxwzfjyrdbkz0mqm7t6ri4rmp2w0hjm 2418876 2418875 2026-05-25T17:06:40Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418876 wikitext text/x-wiki 407jzv6afpnv0i8225y44ihxbl21vdt 2418877 2418876 2026-05-25T17:11:04Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418877 wikitext text/x-wiki 8fgfvtn1js5xpemp1but15yfsuigxt4 2418878 2418877 2026-05-25T17:13:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T426633)', diff saved to https://phabricator.wikimedia.org/P92903 and previous config saved to /var/cache/conftool/dbconfig/20260525-171310-fceratto.json 2418878 wikitext text/x-wiki sbi1ejzhuqzgxkmd17gf1sbbmkp71wf 2418879 2418878 2026-05-25T17:28:38Z Stashbot 7414 sukhe: sukhe@alert1002:~$ sudo systemctl restart icinga.service 2418879 wikitext text/x-wiki a12ihco41vbbbvnu639uei47g6aos2q 2418880 2418879 2026-05-25T17:29:43Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet 2418880 wikitext text/x-wiki hd7cr8tstcr03s3zvqztw2ufn4ia9tq 2418881 2418880 2026-05-25T17:30:22Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418881 wikitext text/x-wiki 2t0y0ikvntcypgm7pqpj28f5p7de5nt 2418882 2418881 2026-05-25T17:30:24Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418882 wikitext text/x-wiki gsli2a1mde9hp7vacwuwjbntksb9m0h 2418883 2418882 2026-05-25T17:30:27Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1042.eqiad.wmnet 2418883 wikitext text/x-wiki 7247rdhxojx7zr77ic96blxts89qsn9 2418884 2418883 2026-05-25T17:31:09Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1110.eqiad.wmnet [reason: manually pooling after reboot as icinga was down] 2418884 wikitext text/x-wiki 616b7ddvi8o2j4gfn9psswti2o52vv5 2418885 2418884 2026-05-25T17:35:19Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1043.eqiad.wmnet 2418885 wikitext text/x-wiki 5ns5ispj02olu727jyvtsslsipppvug 2418886 2418885 2026-05-25T18:00:53Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqsin 2418886 wikitext text/x-wiki i0ey0dmtboptfjbkttva5promo49njb 2418887 2418886 2026-05-25T18:01:08Z Stashbot 7414 sukhe: sre.cdn.roll-reboot cookbooks stalled due to icinga reboot 2418887 wikitext text/x-wiki qnpfu12mpe2qr4xje0zkw4ogou404d8 2418888 2418887 2026-05-25T18:01:13Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_eqsin 2418888 wikitext text/x-wiki l2xz1x8vgu0t7k9dlnqafzhbjrgumtq 2418889 2418888 2026-05-25T18:01:52Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqiad 2418889 wikitext text/x-wiki qe7l2yip8qa06xfd2n72pwe16a6u6z7 2418890 2418889 2026-05-25T18:02:43Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp5023*} and A:cp 2418890 wikitext text/x-wiki jojuran65ahy3oxwz9uo5xb8bsq5yxf 2418891 2418890 2026-05-25T18:03:16Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp1113*} and A:cp 2418891 wikitext text/x-wiki 7pqlndbe3xhhbtuy83pkm9l7uzjqahk 2418892 2418891 2026-05-25T18:09:24Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet 2418892 wikitext text/x-wiki fjoa9qz842nie1e6utjxf3gwysqt76g 2418893 2418892 2026-05-25T18:09:26Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet 2418893 wikitext text/x-wiki 18fja2p88timkjomalul974dkoei0wm 2418894 2418893 2026-05-25T18:09:28Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp1113*} and A:cp 2418894 wikitext text/x-wiki to496d2yh9a7sbgl6fiap5vm04e8fce 2418895 2418894 2026-05-25T18:10:27Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp5030*} and A:cp 2418895 wikitext text/x-wiki 7cum97920n7shvorzc8stig6mjhjato 2418896 2418895 2026-05-25T18:10:51Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418896 wikitext text/x-wiki 0azdi0154ds2ciigzc0xxf907hqaudh 2418897 2418896 2026-05-25T18:15:09Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet 2418897 wikitext text/x-wiki elrt29vbe8dtxjdm0yu55cru9twbn2c 2418898 2418897 2026-05-25T18:15:11Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5023*} and A:cp 2418898 wikitext text/x-wiki qou9ywerx9gw59wfocjxs5b5sqb4fsd 2418899 2418898 2026-05-25T18:22:40Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet 2418899 wikitext text/x-wiki mitqk45iy0vade51rruk2zvoxuiptxv 2418900 2418899 2026-05-25T18:22:43Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5030*} and A:cp 2418900 wikitext text/x-wiki brxn58fs1rqjleo7leu9mj5o0k109hx 2418901 2418900 2026-05-25T18:34:00Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] 2418901 wikitext text/x-wiki 5ediha0b2sxvsqc8yhhwje8r4fly9cr 2418902 2418901 2026-05-25T18:34:08Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] 2418902 wikitext text/x-wiki m65k9p399ln6syr8xu2vvu5f9bazfxg 2418903 2418902 2026-05-25T18:49:19Z Stashbot 7414 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet 2418903 wikitext text/x-wiki o1zjf7sqzzla0h92z14gn3z8so2xm9h 2418904 2418903 2026-05-25T18:49:22Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad 2418904 wikitext text/x-wiki gm8ju901q7xvbkf4l0gl8euhlhfhoag 2418905 2418904 2026-05-25T19:22:14Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418905 wikitext text/x-wiki ky37g4fx6zm78f4egsatwcvf5rl4tzz 2418906 2418905 2026-05-25T19:25:34Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418906 wikitext text/x-wiki exhqxegs039vdsw7kpj3gx1davzm5h8 2418907 2418906 2026-05-25T19:25:36Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418907 wikitext text/x-wiki cp45vxiib66jdfxbw52ez0wavlspo8o 2418908 2418907 2026-05-25T19:25:39Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1043.eqiad.wmnet 2418908 wikitext text/x-wiki r2mw5k0mrjtggwdetjajqm5xmwidau2 2418911 2418908 2026-05-25T19:57:21Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1044.eqiad.wmnet 2418911 wikitext text/x-wiki jum2i1se3ukur8iu19w77tryr7q5jr0 2418912 2418911 2026-05-25T20:06:51Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418912 wikitext text/x-wiki d0nk8zaaujhfht2zoifw8ba3nkt8rt7 2418915 2418912 2026-05-25T20:15:08Z Stashbot 7414 moritzm: truncate krb5kdc.log1 (which made log rotation fail) 2418915 wikitext text/x-wiki 9fiisxk7qhv434yy7fyj0t3m2457053 2418916 2418915 2026-05-25T20:25:41Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418916 wikitext text/x-wiki ttqrx6dtp2fr7zo5jq1zvfk24411yxq 2418918 2418916 2026-05-25T20:37:34Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418918 wikitext text/x-wiki pr6nuemcagxwfple3n50n1yq1jzeizd 2418919 2418918 2026-05-25T20:37:37Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418919 wikitext text/x-wiki dz9vioeg5kmh9lv5ybj90xgd3ulyp6w 2418920 2418919 2026-05-25T20:37:39Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1044.eqiad.wmnet 2418920 wikitext text/x-wiki 84ggrqi3m6dcfwku8ctjcb2zjm6nbvk 2418921 2418920 2026-05-25T20:38:19Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1045.eqiad.wmnet 2418921 wikitext text/x-wiki gfc56r6fhn9nphx9vpvkk35ia441zlp 2418922 2418921 2026-05-25T20:49:21Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418922 wikitext text/x-wiki 0m680dak9nrgcfbckp7fjrm0guu0z72 2418924 2418922 2026-05-25T21:00:44Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418924 wikitext text/x-wiki hkht4pqvkd8jcuufvvmtbngb16aipg6 2418929 2418924 2026-05-26T02:01:04Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2418929 wikitext text/x-wiki 38b2zmashjwn6sa2z4duryi74trv85l 2418930 2418929 2026-05-26T02:07:25Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 20s) 2418930 wikitext text/x-wiki kju0kugfr22slmql4nox1q1popj0u8y 2418931 2418930 2026-05-26T03:03:22Z Stashbot 7414 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.4 refs T423913 2418931 wikitext text/x-wiki 820ry9udocomzfpycil0wlvalkf96ya 2418932 2418931 2026-05-26T03:39:46Z Stashbot 7414 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.4 refs T423913 (duration: 36m 24s) 2418932 wikitext text/x-wiki ln60ocxt6mz6ub73sqt7lgzlsa30aiq 2418933 2418932 2026-05-26T04:02:41Z Stashbot 7414 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.1 (duration: 02m 32s) 2418933 wikitext text/x-wiki 2yf3somczl4we719yihixotigcm8ipm 2418934 2418933 2026-05-26T04:34:32Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.dns.netbox 2418934 wikitext text/x-wiki 2nah0xzgv0u68rrc6sve5bw4juosbn7 2418935 2418934 2026-05-26T04:37:13Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418935 wikitext text/x-wiki so7lk8prizbax0w98ztbxxkz3ycp2l3 2418936 2418935 2026-05-26T05:12:56Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2024.codfw.wmnet,pc[1014,1024].eqiad.wmnet with reason: Maintenance on pc4 2418936 wikitext text/x-wiki nw8bdtv11f22xhsk1emlvlanz5qqgfe 2418937 2418936 2026-05-26T05:15:40Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1014.eqiad.wmnet: Maintenance on pc4 2418937 wikitext text/x-wiki lb4o5773c6lx6h845mwmwdezr4s2fcn 2418938 2418937 2026-05-26T05:15:42Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2418938 wikitext text/x-wiki 21njm5pmja61en6cvus7s4aazkhxpo9 2418939 2418938 2026-05-26T05:15:50Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2418939 wikitext text/x-wiki 7a81wdkzy9xvqjokojgw7iu3li3ehwi 2418940 2418939 2026-05-26T05:15:52Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1014.eqiad.wmnet: Maintenance on pc4 2418940 wikitext text/x-wiki f6rjk9znpospuivi1z3a3ywkb8vx9oa 2418944 2418940 2026-05-26T06:00:52Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only 2418944 wikitext text/x-wiki p8wdko0gbg18qqdfvkx54u7h27lbxw3 2418945 2418944 2026-05-26T06:01:15Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) 2418945 wikitext text/x-wiki 7pfkqm8btwzd59bj9do0k6p3yc2ueod 2418946 2418945 2026-05-26T06:02:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only 2418946 wikitext text/x-wiki qionz3o0mheavng9w0zwduzuzmke6fd 2418947 2418946 2026-05-26T06:03:00Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) 2418947 wikitext text/x-wiki b1pysc2c0ne3gjvepq9pc3csox0lfvp 2418948 2418947 2026-05-26T06:04:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 T425622 2418948 wikitext text/x-wiki 0l1cwvfrtsnrsnmxvrq7ucq8qu8ewkt 2418949 2418948 2026-05-26T06:04:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1162 with weight 0 T425622', diff saved to https://phabricator.wikimedia.org/P92907 and previous config saved to /var/cache/conftool/dbconfig/20260526-060443-fceratto.json 2418949 wikitext text/x-wiki l0lzh4ogkc08f0n6ntdo6mwcitxx9fy 2418951 2418949 2026-05-26T06:10:05Z Stashbot 7414 federico3: Starting s2 eqiad failover from db1222 to db1162 - T425622 2418951 wikitext text/x-wiki fxxn4b3y7zgtjv3n6spnaawx02ucblh 2418952 2418951 2026-05-26T06:10:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - T425622', diff saved to https://phabricator.wikimedia.org/P92908 and previous config saved to /var/cache/conftool/dbconfig/20260526-061021-fceratto.json 2418952 wikitext text/x-wiki 4iqypfxzd2m4jhxsq411h4joljnel0v 2418953 2418952 2026-05-26T06:11:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write T425622', diff saved to https://phabricator.wikimedia.org/P92909 and previous config saved to /var/cache/conftool/dbconfig/20260526-061114-fceratto.json 2418953 wikitext text/x-wiki jo70thc1z9satgyyfudb4doiwk4b7oc 2418954 2418953 2026-05-26T06:14:18Z Stashbot 7414 fceratto@dns1005: START - running authdns-update 2418954 wikitext text/x-wiki 9d83v33te6js02yb92sjw0gncahs2sh 2418955 2418954 2026-05-26T06:16:00Z Stashbot 7414 fceratto@dns1005: END - running authdns-update 2418955 wikitext text/x-wiki preugzmpgld8adf3xh3uj87eh3u6ypj 2418956 2418955 2026-05-26T06:17:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1222 T425622', diff saved to https://phabricator.wikimedia.org/P92910 and previous config saved to /var/cache/conftool/dbconfig/20260526-061656-fceratto.json 2418956 wikitext text/x-wiki cbc30s8bp5vice06j8534ioay4mvyki 2418957 2418956 2026-05-26T06:23:21Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Switchover 2418957 wikitext text/x-wiki lat86uja87g9p63l40a8qbo994e3azy 2418958 2418957 2026-05-26T06:28:55Z Stashbot 7414 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance 2418958 wikitext text/x-wiki syfu8qxud9yyev6n7ysr4erdiujzw7p 2418959 2418958 2026-05-26T06:31:50Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance 2418959 wikitext text/x-wiki fcwv9lkn6m381yornhegqjxnhq70ibq 2418960 2418959 2026-05-26T06:31:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 (T426633)', diff saved to https://phabricator.wikimedia.org/P92912 and previous config saved to /var/cache/conftool/dbconfig/20260526-063155-fceratto.json 2418960 wikitext text/x-wiki f0q1rpwhasxgbrcmfky92np982d1qyk 2418962 2418960 2026-05-26T06:35:37Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org 2418962 wikitext text/x-wiki iiy7988c3xnmdtbftymnf9apy2zoxr6 2418963 2418962 2026-05-26T06:38:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T426633)', diff saved to https://phabricator.wikimedia.org/P92914 and previous config saved to /var/cache/conftool/dbconfig/20260526-063853-fceratto.json 2418963 wikitext text/x-wiki s4kuu7jv0m1hzz66casawjl73q7y132 2418964 2418963 2026-05-26T06:41:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org 2418964 wikitext text/x-wiki 4m0wp5wjddka2pw76u4qaj5szt0h7ny 2418966 2418964 2026-05-26T06:47:54Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1222: Switchover 2418966 wikitext text/x-wiki n38oaxyhd9ienu5vbcu3zrexwdde9y5 2418967 2418966 2026-05-26T06:48:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1222.eqiad.wmnet with reason: Maintenance 2418967 wikitext text/x-wiki t43zb5la7s1riqzcna4js4d3ty6f8g1 2418968 2418967 2026-05-26T06:48:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1222 (T419635)', diff saved to https://phabricator.wikimedia.org/P92916 and previous config saved to /var/cache/conftool/dbconfig/20260526-064833-fceratto.json 2418968 wikitext text/x-wiki m1tznt36fdpp4o5b3g7kptrbx0ctr1w 2418969 2418968 2026-05-26T06:49:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92917 and previous config saved to /var/cache/conftool/dbconfig/20260526-064901-fceratto.json 2418969 wikitext text/x-wiki 9a22u2jg2qvhu44h75l7xn4bb2oegrt 2418970 2418969 2026-05-26T06:50:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org 2418970 wikitext text/x-wiki 035nt48ku59dv224xe4hb9sztdcqlcr 2418972 2418970 2026-05-26T06:53:29Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418972 wikitext text/x-wiki s3s1o3qk04fkikopy5xzsxzrkt2nl3r 2418973 2418972 2026-05-26T06:53:32Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418973 wikitext text/x-wiki s5uzwv2jzj2joh114hqs702nv50njzj 2418974 2418973 2026-05-26T06:53:35Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1045.eqiad.wmnet 2418974 wikitext text/x-wiki krjxfptx86ysyd9b5fv11tqz6n82dy5 2418975 2418974 2026-05-26T06:53:44Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1046.eqiad.wmnet 2418975 wikitext text/x-wiki fc68jrolc7r2brshrsfljlccuxh36wi 2418976 2418975 2026-05-26T06:55:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet 2418976 wikitext text/x-wiki qpdrazfzq74e082sgdlfm4uc744yqct 2418977 2418976 2026-05-26T06:58:38Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet 2418977 wikitext text/x-wiki 0n7xg51ct50o2veth67z95ynfz3gh6d 2418978 2418977 2026-05-26T06:58:44Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2418978 wikitext text/x-wiki pukywp0idgd17g85rh9drq1xzk6z32m 2418979 2418978 2026-05-26T06:59:05Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org 2418979 wikitext text/x-wiki a48kp43lb4bx7lxb4szyd955bsaeie1 2418980 2418979 2026-05-26T06:59:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92918 and previous config saved to /var/cache/conftool/dbconfig/20260526-065909-fceratto.json 2418980 wikitext text/x-wiki siui2qqm2nhlwhwmxjm2b5tsgk50qyk 2418981 2418980 2026-05-26T07:02:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org 2418981 wikitext text/x-wiki b664pj2y46apvkyovc3orl15qbqzmxl 2418982 2418981 2026-05-26T07:03:58Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet 2418982 wikitext text/x-wiki kmiku40nz36ioiudmsjsorg9k2mxxne 2418983 2418982 2026-05-26T07:04:06Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418983 wikitext text/x-wiki f6wkp2tfkueupack9rakjlkmfmz786k 2418984 2418983 2026-05-26T07:04:32Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet 2418984 wikitext text/x-wiki 6e2x6iz6c81d7qa2c1q7yetoqhhsv4g 2418985 2418984 2026-05-26T07:04:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T419635)', diff saved to https://phabricator.wikimedia.org/P92919 and previous config saved to /var/cache/conftool/dbconfig/20260526-070436-fceratto.json 2418985 wikitext text/x-wiki 4wck6iz1xg8sp4x8bdjmn6dycni5pwo 2418986 2418985 2026-05-26T07:06:42Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org 2418986 wikitext text/x-wiki k4zklciil8laoyjymi5695zd7jkr4nc 2418987 2418986 2026-05-26T07:07:01Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2418987 wikitext text/x-wiki c8oeo8ed9jqolqzxbkx6zbm7fqjl7us 2418988 2418987 2026-05-26T07:07:03Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2418988 wikitext text/x-wiki 8lvqipsollbrepcjfnjufz2qofnmzlo 2418989 2418988 2026-05-26T07:07:06Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1046.eqiad.wmnet 2418989 wikitext text/x-wiki jgubo8o358kxw71xizh3uk0ko9j9kog 2418990 2418989 2026-05-26T07:09:11Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1047.eqiad.wmnet 2418990 wikitext text/x-wiki 9skpodp6o6jnfex5xizsqylmkcdarr3 2418991 2418990 2026-05-26T07:09:14Z Stashbot 7414 moritzm: failover Ganeti master in eqiad to ganeti1048 2418991 wikitext text/x-wiki oirjsp7z7t547zms37imucbhniefzj5 2418992 2418991 2026-05-26T07:09:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T426633)', diff saved to https://phabricator.wikimedia.org/P92920 and previous config saved to /var/cache/conftool/dbconfig/20260526-070916-fceratto.json 2418992 wikitext text/x-wiki ohs5lkeggxxzkynzpxcwtjrow0v77ww 2418993 2418992 2026-05-26T07:09:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance 2418993 wikitext text/x-wiki 5wv69adr0u9mq6ww2bifg0hdpifdeds 2418994 2418993 2026-05-26T07:09:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2180 (T426633)', diff saved to https://phabricator.wikimedia.org/P92921 and previous config saved to /var/cache/conftool/dbconfig/20260526-070946-fceratto.json 2418994 wikitext text/x-wiki eecxevhe5i58erlufvdwwovfjdqv7cj 2418995 2418994 2026-05-26T07:10:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet 2418995 wikitext text/x-wiki 7qvk59dax31l141zljmdeu8c89l0a58 2418996 2418995 2026-05-26T07:11:37Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet 2418996 wikitext text/x-wiki itgjallpcnid4f0bq2ubxx4eooeeo3o 2418997 2418996 2026-05-26T07:13:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet 2418997 wikitext text/x-wiki fbyzwgxl37orwerv5mi9cqnn2upg71v 2418998 2418997 2026-05-26T07:14:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92922 and previous config saved to /var/cache/conftool/dbconfig/20260526-071444-fceratto.json 2418998 wikitext text/x-wiki 2gsjmhy2mj1l4t5iq0q63oxx1u7dvam 2418999 2418998 2026-05-26T07:15:43Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1026.eqiad.wmnet 2418999 wikitext text/x-wiki h69jz2fhvyb120445uvw1zvyont4f4p 2419000 2418999 2026-05-26T07:15:55Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet 2419000 wikitext text/x-wiki 6zxgs855m5mieqxzs0iumo54nr1hgeu 2419001 2419000 2026-05-26T07:16:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T426633)', diff saved to https://phabricator.wikimedia.org/P92923 and previous config saved to /var/cache/conftool/dbconfig/20260526-071635-fceratto.json 2419001 wikitext text/x-wiki b059mvr2ung502gbjdmavxxn4x66ed2 2419002 2419001 2026-05-26T07:18:24Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet 2419002 wikitext text/x-wiki h163tyyzl4waf1rjzaojodfwb4w9d29 2419003 2419002 2026-05-26T07:23:52Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet 2419003 wikitext text/x-wiki 85o2dulc0l2ya95omc73yuok90ax9fv 2419004 2419003 2026-05-26T07:24:19Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet 2419004 wikitext text/x-wiki gbfb6t1tu9mrza2j8ma9pdht2cjfav3 2419005 2419004 2026-05-26T07:24:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92924 and previous config saved to /var/cache/conftool/dbconfig/20260526-072452-fceratto.json 2419005 wikitext text/x-wiki h3tiog9ykuol17l1skaej7uch83k75s 2419006 2419005 2026-05-26T07:25:45Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2419006 wikitext text/x-wiki rnqb4pkizum1y3kjyg0qe5n749kbntg 2419007 2419006 2026-05-26T07:26:51Z Stashbot 7414 arthurtaylor@deploy1003: Started scap sync-world: Backport for [[gerrit:1291951|Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] 2419007 wikitext text/x-wiki p01p4wpbtidwe6n8260uh6g5m174ska 2419008 2419007 2026-05-26T07:26:54Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet 2419008 wikitext text/x-wiki dhezfg8kgl2cx8ii2f41djt0bqwtcce 2419009 2419008 2026-05-26T07:26:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260526-072643-fceratto.json 2419009 wikitext text/x-wiki jj1tk4xnyluaebw6u1cmq8d8ynbuvh8 2419010 2419009 2026-05-26T07:30:58Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet 2419010 wikitext text/x-wiki m6gglqymypcb9zve8v1revphviu0tb3 2419011 2419010 2026-05-26T07:31:06Z Stashbot 7414 arthurtaylor@deploy1003: arthurtaylor: Backport for [[gerrit:1291951|Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419011 wikitext text/x-wiki n8f4b222nii7zz31ainidv4i4x7u4qg 2419012 2419011 2026-05-26T07:32:34Z Stashbot 7414 arthurtaylor@deploy1003: arthurtaylor: Continuing with deployment 2419012 wikitext text/x-wiki lcohvxa2ya45p1t4d8r35d53koujka1 2419013 2419012 2026-05-26T07:35:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T419635)', diff saved to https://phabricator.wikimedia.org/P92926 and previous config saved to /var/cache/conftool/dbconfig/20260526-073459-fceratto.json 2419013 wikitext text/x-wiki edu33hrbp23hnoa15mef63etv6ifqfx 2419014 2419013 2026-05-26T07:35:06Z Stashbot 7414 fabfur: start rebooting magru liberica instances (T426563) 2419014 wikitext text/x-wiki gi9jqh63hg36lbi97q5x5u1v3bayrwp 2419015 2419014 2026-05-26T07:35:17Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance 2419015 wikitext text/x-wiki t1m9ywx7cculdjv4tl77kuqxri88gph 2419016 2419015 2026-05-26T07:36:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419016 wikitext text/x-wiki fyin2gxpzyapvfl5wvxkna5c4ain5ck 2419017 2419016 2026-05-26T07:36:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet 2419017 wikitext text/x-wiki g3nhi8mdfgttivgjiy9k2q43o08mszw 2419018 2419017 2026-05-26T07:36:59Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7003.magru.wmnet} and A:liberica 2419018 wikitext text/x-wiki ey4bojyfx74e8q5u86wsgitesyhtmbl 2419019 2419018 2026-05-26T07:37:04Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet 2419019 wikitext text/x-wiki dhpc1ikr31hpb0z1m2ajfxqeazogdmf 2419020 2419019 2026-05-26T07:37:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P92928 and previous config saved to /var/cache/conftool/dbconfig/20260526-073702-fceratto.json 2419020 wikitext text/x-wiki 8bd5xi3h7vdn68voyqoe3v0o68bhxsn 2419021 2419020 2026-05-26T07:38:25Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2419021 wikitext text/x-wiki mliucmayhvzrl4au428bditjsqxzt7a 2419022 2419021 2026-05-26T07:38:49Z Stashbot 7414 arthurtaylor@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291951|Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] (duration: 12m 01s) 2419022 wikitext text/x-wiki 159723zaq79tk6emzpnnys54i954gr9 2419023 2419022 2026-05-26T07:40:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet 2419023 wikitext text/x-wiki stj3gfuirzqv4l31neqr08baewrc8fy 2419024 2419023 2026-05-26T07:40:29Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet 2419024 wikitext text/x-wiki 7nuxdxf9vbvgbqrlqmgt3m0urb66tq0 2419025 2419024 2026-05-26T07:40:36Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7003.magru.wmnet} and A:liberica 2419025 wikitext text/x-wiki 4y1l8knu97pqpch5g4y898vvo9372x8 2419026 2419025 2026-05-26T07:41:40Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7001.magru.wmnet} and A:liberica 2419026 wikitext text/x-wiki rkqi8cwxkg4s4itczey2biyrf2uo5rz 2419027 2419026 2026-05-26T07:43:48Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2419027 wikitext text/x-wiki qzz0w2b4rvsgyqz0qrbiyh4y8g7nz1w 2419028 2419027 2026-05-26T07:43:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419028 wikitext text/x-wiki t4evuynguqui6xdstn2txamlrdgp0gt 2419029 2419028 2026-05-26T07:43:56Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2419029 wikitext text/x-wiki 9bi4e1xl5nmyvz2svmddgx89rfkd5mh 2419030 2419029 2026-05-26T07:44:31Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet 2419030 wikitext text/x-wiki fy1nt9a6v9hod6jpetrnqesuljb5n39 2419031 2419030 2026-05-26T07:45:27Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7001.magru.wmnet} and A:liberica 2419031 wikitext text/x-wiki lz19xej8xpxgp5p61d1sistz1qqvxih 2419032 2419031 2026-05-26T07:45:48Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419032 wikitext text/x-wiki srieax6hkyd96jb8fy17hksg5l91gyk 2419033 2419032 2026-05-26T07:45:58Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet 2419033 wikitext text/x-wiki 0wyxozij219d696jd90udc6th7910w9 2419034 2419033 2026-05-26T07:46:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet 2419034 wikitext text/x-wiki ijoq9oqqvmxaiqtagsj7vtly7ygxeyl 2419035 2419034 2026-05-26T07:47:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T426633)', diff saved to https://phabricator.wikimedia.org/P92929 and previous config saved to /var/cache/conftool/dbconfig/20260526-074710-fceratto.json 2419035 wikitext text/x-wiki d41lix0j1a9g9m31frkky5vudevh6zg 2419036 2419035 2026-05-26T07:47:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance 2419036 wikitext text/x-wiki 30mhvxih5lf5rknl5mo2ikdenzrixnq 2419037 2419036 2026-05-26T07:47:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 (T426633)', diff saved to https://phabricator.wikimedia.org/P92930 and previous config saved to /var/cache/conftool/dbconfig/20260526-074739-fceratto.json 2419037 wikitext text/x-wiki k5q91dybw7dddxy8l00zn9khzt334rd 2419038 2419037 2026-05-26T07:49:01Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1023.eqiad.wmnet 2419038 wikitext text/x-wiki 1klrs3kbjqfhyb3x8cxh68zf7ys9779 2419039 2419038 2026-05-26T07:51:22Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2419039 wikitext text/x-wiki 4y21arg3a39a1b013y40krdntvqti78 2419040 2419039 2026-05-26T07:51:24Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419040 wikitext text/x-wiki kvorfacjyiczgbwuvgk8mtfrxhhcenu 2419041 2419040 2026-05-26T07:51:26Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1047.eqiad.wmnet 2419041 wikitext text/x-wiki hzd3r3k636rykksvelivfk1uyg7y73k 2419042 2419041 2026-05-26T07:52:09Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7002.magru.wmnet} and A:liberica 2419042 wikitext text/x-wiki k90d9di74f3pd1ooojjxw62qr6xfd9d 2419043 2419042 2026-05-26T07:54:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T426633)', diff saved to https://phabricator.wikimedia.org/P92931 and previous config saved to /var/cache/conftool/dbconfig/20260526-075435-fceratto.json 2419043 wikitext text/x-wiki flgih5448d2o8ni63dcqmsbppdy9a5f 2419044 2419043 2026-05-26T07:54:57Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2419044 wikitext text/x-wiki 0lh4q9qxrwer9qhdakpxckrv1762sq3 2419045 2419044 2026-05-26T07:56:04Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7002.magru.wmnet} and A:liberica 2419045 wikitext text/x-wiki 046zhs26zudi50f30utf0yk3nqlr4bo 2419046 2419045 2026-05-26T07:56:44Z Stashbot 7414 fabfur: start rebooting drmrs liberica instances (T426563) 2419046 wikitext text/x-wiki 1tcy9q4yvoleq7c4w8ttbmy9yzudblu 2419047 2419046 2026-05-26T07:57:00Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs6003.drmrs.wmnet} and A:liberica 2419047 wikitext text/x-wiki 07gs8hgyggndl7d8rb9yw7uau29q3u7 2419048 2419047 2026-05-26T07:58:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2419048 wikitext text/x-wiki frplmojfmulq3vyupnurme0hn8cjbx1 2419049 2419048 2026-05-26T07:59:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2419049 wikitext text/x-wiki 0ro4ms1v4867q9cpd8eb27zuetvrhbt 2419050 2419049 2026-05-26T07:59:07Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2419050 wikitext text/x-wiki 6a6k5qoigds36vuxqgrt9kierhxz1y5 2419051 2419050 2026-05-26T07:59:10Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2419051 wikitext text/x-wiki r8jqm7u2358aov5b5fwwxpeukxv7q4d 2419052 2419051 2026-05-26T07:59:12Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419052 wikitext text/x-wiki 0ifg1p3k1biphgjnfvzq2fft0ktbd9f 2419053 2419052 2026-05-26T07:59:14Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1023.eqiad.wmnet 2419053 wikitext text/x-wiki kba92kblz593sstr741osxfd7vkkv0a 2419054 2419053 2026-05-26T07:59:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419054 wikitext text/x-wiki 9ow6srhhgdt7qjn6shu4bnmz9amben8 2419055 2419054 2026-05-26T07:59:56Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1024.eqiad.wmnet 2419055 wikitext text/x-wiki tcth9fq3krfjtrqegd5pasitrfnhk63 2419056 2419055 2026-05-26T07:59:58Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet 2419056 wikitext text/x-wiki bya1706ptd6ikv1jszhpfoje0koeekk 2419057 2419056 2026-05-26T08:00:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet 2419057 wikitext text/x-wiki enbl11cih8tyka9dw2lghwh4a1zvwwt 2419058 2419057 2026-05-26T08:00:24Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs6003.drmrs.wmnet} and A:liberica 2419058 wikitext text/x-wiki 7quh9k64k6ta0cc0175r4i50hmpdxhi 2419059 2419058 2026-05-26T08:01:25Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS trixie 2419059 wikitext text/x-wiki pa7r8yf19v8jkjos9l0va9igqctit9q 2419060 2419059 2026-05-26T08:04:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92932 and previous config saved to /var/cache/conftool/dbconfig/20260526-080443-fceratto.json 2419060 wikitext text/x-wiki ryoxuz08lgrjxmnwt79npjzvm720qfr 2419061 2419060 2026-05-26T08:05:00Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2419061 wikitext text/x-wiki el88qm5jmeadue98dhc79hd83gdxxmi 2419062 2419061 2026-05-26T08:09:22Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2419062 wikitext text/x-wiki 34w0l43covxzelka2p3hw4txcg3u9yn 2419063 2419062 2026-05-26T08:10:20Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs6001.drmrs.wmnet} and A:liberica 2419063 wikitext text/x-wiki kmuato5vqye9pulezsken4lo7bsns1x 2419064 2419063 2026-05-26T08:12:50Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.4 refs T423913 2419064 wikitext text/x-wiki cti1ul8pp4gmnfbqddbwpy4jtthgm9o 2419065 2419064 2026-05-26T08:13:26Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs6001.drmrs.wmnet} and A:liberica 2419065 wikitext text/x-wiki ptsksfdrshczxhhzjiioiw6b523md1w 2419066 2419065 2026-05-26T08:14:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92934 and previous config saved to /var/cache/conftool/dbconfig/20260526-081451-fceratto.json 2419066 wikitext text/x-wiki aqehqej4svmby0djrgbj9sc54kzqxkb 2419067 2419066 2026-05-26T08:18:40Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage 2419067 wikitext text/x-wiki 5rhxofqh61te0zfuzstpduhttjp9yvb 2419068 2419067 2026-05-26T08:19:54Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419068 wikitext text/x-wiki jzceod9gz96kurnlgexfjpbc1fvvj71 2419069 2419068 2026-05-26T08:20:17Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2195: Upgrading db2195.codfw.wmnet 2419069 wikitext text/x-wiki qgt3f210qlelosnhzk3t79a9i4ipb5v 2419070 2419069 2026-05-26T08:21:07Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Upgrading db2195.codfw.wmnet 2419070 wikitext text/x-wiki mwjpmvztei3i4dnqfxc7epccyzb3m4a 2419071 2419070 2026-05-26T08:23:18Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage 2419071 wikitext text/x-wiki fsgmdy0pnkomdfovm3pr81j1z1mnnnz 2419072 2419071 2026-05-26T08:23:46Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2195.codfw.wmnet with OS trixie 2419072 wikitext text/x-wiki 0ykq044o8007brwjose13q52p9r0tsp 2419073 2419072 2026-05-26T08:25:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T426633)', diff saved to https://phabricator.wikimedia.org/P92936 and previous config saved to /var/cache/conftool/dbconfig/20260526-082458-fceratto.json 2419073 wikitext text/x-wiki gg47dg6t9g1m5w39hocch4i124ym2kp 2419074 2419073 2026-05-26T08:25:25Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance 2419074 wikitext text/x-wiki jt95nk02pf6svt81jsud7hne3lao7sc 2419075 2419074 2026-05-26T08:25:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2217 (T426633)', diff saved to https://phabricator.wikimedia.org/P92937 and previous config saved to /var/cache/conftool/dbconfig/20260526-082531-fceratto.json 2419075 wikitext text/x-wiki 60tsfs259upfudscyh1z4c0ha29mcup 2419076 2419075 2026-05-26T08:30:39Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs6002.drmrs.wmnet} and A:liberica 2419076 wikitext text/x-wiki 9vxakezpofvz7qrth1o77tnf14ekc5w 2419077 2419076 2026-05-26T08:32:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T426633)', diff saved to https://phabricator.wikimedia.org/P92938 and previous config saved to /var/cache/conftool/dbconfig/20260526-083233-fceratto.json 2419077 wikitext text/x-wiki j3ykvqqkms8b2ot3t3q81hpkxowdv2s 2419078 2419077 2026-05-26T08:33:25Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292032|Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964|hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] 2419078 wikitext text/x-wiki eabafaxjkady83zxsxy790tmhziwfss 2419079 2419078 2026-05-26T08:33:44Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs6002.drmrs.wmnet} and A:liberica 2419079 wikitext text/x-wiki es4of6gvws82j4dxkw1i56m8cza9scf 2419080 2419079 2026-05-26T08:35:23Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1292032|Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964|hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419080 wikitext text/x-wiki a6u5wswtj6mgrvw7l90d1se1nx0ldtv 2419081 2419080 2026-05-26T08:35:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" 2419081 wikitext text/x-wiki e08qrnfw3omtmvgzw1kqre5ksz4dkrz 2419082 2419081 2026-05-26T08:35:27Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2419082 wikitext text/x-wiki hh7z2kocasgcj4elieokxf95sv0d572 2419083 2419082 2026-05-26T08:35:30Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1024.eqiad.wmnet 2419083 wikitext text/x-wiki h026jxo3761o9f9g9wozijxwb41fns8 2419084 2419083 2026-05-26T08:35:35Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs5006.eqsin.wmnet} and A:liberica 2419084 wikitext text/x-wiki nufxrz4hdpyirm8rp80lpg1ib1mpn3v 2419085 2419084 2026-05-26T08:39:04Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs5006.eqsin.wmnet} and A:liberica 2419085 wikitext text/x-wiki k9qp2i2c49ppid3lvp5imqk6kxc00nw 2419086 2419085 2026-05-26T08:39:11Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2419086 wikitext text/x-wiki fev1cvh2mqkc2ybymenqkgzcxplwo84 2419087 2419086 2026-05-26T08:39:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet 2419087 wikitext text/x-wiki 2gifahj56cj8ek6xwy6f5sxoas3zrjp 2419088 2419087 2026-05-26T08:39:59Z Stashbot 7414 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2419088 wikitext text/x-wiki 5jfn8a4lfljmjz6n8bnnht097y4u5g7 2419089 2419088 2026-05-26T08:40:08Z Stashbot 7414 fabfur: start rebooting eqsin liberica instances (T426563) 2419089 wikitext text/x-wiki s1yj02b1dcdz4nma42bimd1vararll2 2419090 2419089 2026-05-26T08:40:13Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs5004.eqsin.wmnet} and A:liberica 2419090 wikitext text/x-wiki tir5ufsoeplo1jgx1kz9ir3aq2igjrt 2419091 2419090 2026-05-26T08:40:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS trixie 2419091 wikitext text/x-wiki egzy423br3nrwrufirja53y81vrgu3b 2419092 2419091 2026-05-26T08:42:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92939 and previous config saved to /var/cache/conftool/dbconfig/20260526-084240-fceratto.json 2419092 wikitext text/x-wiki ou3ahy2vqrh4ifxbw0m4d57l243gbja 2419093 2419092 2026-05-26T08:43:21Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292032|Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964|hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] (duration: 09m 56s) 2419093 wikitext text/x-wiki h78q2ln10dshiy0470fbb6z9g1mvqoh 2419094 2419093 2026-05-26T08:43:26Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2195.codfw.wmnet with reason: host reimage 2419094 wikitext text/x-wiki qd7gvkc2io52qth9yof03zdotb0uopg 2419095 2419094 2026-05-26T08:43:41Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet 2419095 wikitext text/x-wiki rbe0q55hxye9ldlszml42vgflubiwiq 2419096 2419095 2026-05-26T08:43:51Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs5004.eqsin.wmnet} and A:liberica 2419096 wikitext text/x-wiki e761hsl6z23z4hgdrhaltx3951lpcln 2419097 2419096 2026-05-26T08:46:07Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293594|Allow to remove passkeys when there's only one standard 2FA method (T426872)]] 2419097 wikitext text/x-wiki s4l8awtec4zxz9euichkos20oru36rt 2419099 2419097 2026-05-26T08:48:00Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293594|Allow to remove passkeys when there's only one standard 2FA method (T426872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419099 wikitext text/x-wiki to7d3kq34j0wua8r1k0b5s545kfffy4 2419100 2419099 2026-05-26T08:48:53Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2419100 wikitext text/x-wiki e3gh8vv59rxa32czb264joxx1xnpih8 2419101 2419100 2026-05-26T08:49:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Migration of db1222.eqiad.wmnet completed 2419101 wikitext text/x-wiki pz4ryzqiki8ow9ubc1epywlcjovra0k 2419102 2419101 2026-05-26T08:49:58Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2195.codfw.wmnet with reason: host reimage 2419102 wikitext text/x-wiki nl9ovs5c250759c0ha09f9xjwwbg8pt 2419103 2419102 2026-05-26T08:50:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419103 wikitext text/x-wiki 4c8okkxyr5033ugbq18859m8u9glh2x 2419104 2419103 2026-05-26T08:50:08Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs5005.eqsin.wmnet} and A:liberica 2419104 wikitext text/x-wiki oy9bypermfwp2t71rs7drz5r1kizxr2 2419105 2419104 2026-05-26T08:50:22Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1226: Upgrading db1226.eqiad.wmnet 2419105 wikitext text/x-wiki o9hde5uw3kyewz8qaxvv966q7onomya 2419106 2419105 2026-05-26T08:51:27Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox 2419106 wikitext text/x-wiki 5dmzzz89eo9psa5pf5frpm0ct5rg3cq 2419107 2419106 2026-05-26T08:51:31Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors 2419107 wikitext text/x-wiki 6wf4oembn9og5tms2jnswyhbizyhtr8 2419108 2419107 2026-05-26T08:51:35Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors 2419108 wikitext text/x-wiki k2fz0tyi0dfexbr5la4efkqtutempol 2419109 2419108 2026-05-26T08:52:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92941 and previous config saved to /var/cache/conftool/dbconfig/20260526-085248-fceratto.json 2419109 wikitext text/x-wiki gzn3v2cwk9postyjd0kvk64dfyslbwf 2419110 2419109 2026-05-26T08:53:12Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1226: Upgrading db1226.eqiad.wmnet 2419110 wikitext text/x-wiki 6gcwy4axn9pkqw1db3pfx96uinpwfke 2419111 2419110 2026-05-26T08:53:26Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs5005.eqsin.wmnet} and A:liberica 2419111 wikitext text/x-wiki 9zfld0yjyk31z8u4uz9b1ylh2d4h9zy 2419112 2419111 2026-05-26T08:53:30Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293594|Allow to remove passkeys when there's only one standard 2FA method (T426872)]] (duration: 07m 23s) 2419112 wikitext text/x-wiki 99k655h7tqfgp21gsofpof4nyjhfnot 2419113 2419112 2026-05-26T08:53:41Z Stashbot 7414 fabfur: start rebooting ulsfo liberica instances (T426563) 2419113 wikitext text/x-wiki s2mh2r09jizgrt0d7n46wfmoyv2m2tk 2419114 2419113 2026-05-26T08:54:01Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs4008.ulsfo.wmnet} and A:liberica 2419114 wikitext text/x-wiki cxegxtvtqcw12cjeaen9jfypf4rf2yx 2419115 2419114 2026-05-26T08:55:53Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1226.eqiad.wmnet with OS trixie 2419115 wikitext text/x-wiki 0mgsrtktfh2nacz0sjpcvi2l3oxmgu5 2419116 2419115 2026-05-26T08:56:05Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors 2419116 wikitext text/x-wiki 5onto1zwpobqn5ark6zvwf0d1i4mscp 2419117 2419116 2026-05-26T08:56:08Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors 2419117 wikitext text/x-wiki hj5kp5xqur1n9xt9a7k9to4n0oul23e 2419118 2419117 2026-05-26T08:57:08Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs4008.ulsfo.wmnet} and A:liberica 2419118 wikitext text/x-wiki oxoxvqs5wz6ggd7u0sertesfek42tpo 2419119 2419118 2026-05-26T09:02:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T426633)', diff saved to https://phabricator.wikimedia.org/P92942 and previous config saved to /var/cache/conftool/dbconfig/20260526-090256-fceratto.json 2419119 wikitext text/x-wiki 9t46wh9dp9zslaztecegwmmpyulf73s 2419120 2419119 2026-05-26T09:03:02Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs4009.ulsfo.wmnet} and A:liberica 2419120 wikitext text/x-wiki 8i1rzzx2f1w8sqhiseuthllx3qnpate 2419121 2419120 2026-05-26T09:03:09Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance 2419121 wikitext text/x-wiki iyesj4g7bjwc6ivijxeg8u8fm4gy59b 2419122 2419121 2026-05-26T09:03:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2224 (T426633)', diff saved to https://phabricator.wikimedia.org/P92943 and previous config saved to /var/cache/conftool/dbconfig/20260526-090315-fceratto.json 2419122 wikitext text/x-wiki o0w6mvzpycujvpv4b1aopnbrvlnm7fv 2419124 2419122 2026-05-26T09:06:12Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs4009.ulsfo.wmnet} and A:liberica 2419124 wikitext text/x-wiki 23gzgvs0itp5y3wql9nzd4ozykm19q9 2419125 2419124 2026-05-26T09:07:18Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293658|Fix TypeError in Mandatory2FAChecker (T427251)]] 2419125 wikitext text/x-wiki brwrsls85utpmotnncdng71r1tchihx 2419126 2419125 2026-05-26T09:07:58Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2195.codfw.wmnet with OS trixie 2419126 wikitext text/x-wiki bvllgcjzlrgwy85ooumxzfnk01bvngu 2419127 2419126 2026-05-26T09:09:11Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293658|Fix TypeError in Mandatory2FAChecker (T427251)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419127 wikitext text/x-wiki d14y6zfx3ecra6kgtuqvfx19wn84g1i 2419128 2419127 2026-05-26T09:09:54Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2419128 wikitext text/x-wiki qdr57tu536p26l2f2rsj81bhuhmpi1c 2419129 2419128 2026-05-26T09:10:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T426633)', diff saved to https://phabricator.wikimedia.org/P92944 and previous config saved to /var/cache/conftool/dbconfig/20260526-091016-fceratto.json 2419129 wikitext text/x-wiki fxdxsq9w12am04vi8m7hu24j6gctoax 2419130 2419129 2026-05-26T09:10:55Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage 2419130 wikitext text/x-wiki dbwkco4oyl1k350zr36gir0ip6hekbn 2419131 2419130 2026-05-26T09:14:05Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293658|Fix TypeError in Mandatory2FAChecker (T427251)]] (duration: 06m 47s) 2419131 wikitext text/x-wiki 2p2t10f13b5ja3ohbvi84q7iudlg40i 2419132 2419131 2026-05-26T09:14:34Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{kubestage200*} and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) 2419132 wikitext text/x-wiki 9vaotf1jxk0zqzx2hu4ojewej2khqh4 2419133 2419132 2026-05-26T09:14:46Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{kubestage100*} and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) 2419133 wikitext text/x-wiki f6terki5jnw5v8tkbp8cuxze34o0wlb 2419134 2419133 2026-05-26T09:14:49Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet 2419134 wikitext text/x-wiki 99cu16m97ebqa7k8vutc5d6h54hrsbd 2419135 2419134 2026-05-26T09:14:52Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage 2419135 wikitext text/x-wiki 63i90hsyc639nxj8zkvsdk7lijqvt23 2419136 2419135 2026-05-26T09:15:00Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet 2419136 wikitext text/x-wiki i8ltbxmriaxhft7iad2fmeyjf8qsecm 2419137 2419136 2026-05-26T09:15:21Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet 2419137 wikitext text/x-wiki 8vkr9edlhh1ipj6ocf3ul7wwy8z035l 2419139 2419137 2026-05-26T09:16:44Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2195: Migration of db2195.codfw.wmnet completed 2419139 wikitext text/x-wiki eeb1gowp21gee0n70vfxn1qt965myxh 2419140 2419139 2026-05-26T09:20:04Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet 2419140 wikitext text/x-wiki gekipv2h9n00crailu2jwio10c7pwn6 2419141 2419140 2026-05-26T09:20:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92946 and previous config saved to /var/cache/conftool/dbconfig/20260526-092024-fceratto.json 2419141 wikitext text/x-wiki tl0sftaq2q8lrf91mxqljoq9rolbjk2 2419142 2419141 2026-05-26T09:21:00Z Stashbot 7414 fabfur: start rebooting esams liberica instances (T426563) 2419142 wikitext text/x-wiki b1ck6bx0579uu3s7anfa6zgdceskn8v 2419143 2419142 2026-05-26T09:21:25Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs3010.esams.wmnet} and A:liberica 2419143 wikitext text/x-wiki jdrlv8va37jyobgxhwkie5xe9fxmesj 2419144 2419143 2026-05-26T09:22:08Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet 2419144 wikitext text/x-wiki ol4mmwsbz8ajssy8a5i3gxvdr7hi97r 2419145 2419144 2026-05-26T09:22:10Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet 2419145 wikitext text/x-wiki 2l0t7fa8qtxz2v8g6eogl1dt5eho7g2 2419146 2419145 2026-05-26T09:22:21Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet 2419146 wikitext text/x-wiki p328e6ur3hmceda7uzy71eigbolnojr 2419147 2419146 2026-05-26T09:22:52Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet 2419147 wikitext text/x-wiki btgmzfnkv47ps3hkk6ywt5z00qn8b9s 2419148 2419147 2026-05-26T09:25:09Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs3010.esams.wmnet} and A:liberica 2419148 wikitext text/x-wiki 3n32ux0klz8957zp6xgnn4n8xbsi8yk 2419149 2419148 2026-05-26T09:25:18Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs3008.esams.wmnet} and A:liberica 2419149 wikitext text/x-wiki bk2mgpm74tkij7hehz77umstzarl6lc 2419150 2419149 2026-05-26T09:26:09Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293661|hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] 2419150 wikitext text/x-wiki 8432cciqdztt0l333aa9z8a02w1s2y6 2419151 2419150 2026-05-26T09:26:12Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet 2419151 wikitext text/x-wiki ml670yk6s1ayndibazoml8mbzlfiup5 2419152 2419151 2026-05-26T09:26:14Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet 2419152 wikitext text/x-wiki tu6qvmtsbz2xs2z186egy9o4rsy17sy 2419153 2419152 2026-05-26T09:26:23Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet 2419153 wikitext text/x-wiki qfmcxqb2b6uhe9pdy0qjvjm75yujpl6 2419154 2419153 2026-05-26T09:27:34Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet 2419154 wikitext text/x-wiki lkwyuzzbgvye7ajxmb8rcnvkiwv7qj4 2419155 2419154 2026-05-26T09:28:01Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293661|hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419155 wikitext text/x-wiki k8d89mpn9l4ckbgbhjk83vtmx4ffvf2 2419156 2419155 2026-05-26T09:28:50Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs3008.esams.wmnet} and A:liberica 2419156 wikitext text/x-wiki rjv6h3pqd926lvhqo3fhfqe53qpqzvz 2419157 2419156 2026-05-26T09:28:53Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2419157 wikitext text/x-wiki rz5n7o38k4uhkv59h7teff58qrdvhdi 2419158 2419157 2026-05-26T09:29:19Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet 2419158 wikitext text/x-wiki c9ko6tusczqvay1i71vxq0goozprmgl 2419159 2419158 2026-05-26T09:29:22Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet 2419159 wikitext text/x-wiki neitg94lnw2fqisfzvcp1hxtqrxxiig 2419160 2419159 2026-05-26T09:29:32Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet 2419160 wikitext text/x-wiki 7ywp86o8y80ba1vzplvb8a8hvfnqey0 2419161 2419160 2026-05-26T09:30:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92947 and previous config saved to /var/cache/conftool/dbconfig/20260526-093031-fceratto.json 2419161 wikitext text/x-wiki dvt8phm8z8t6sy1x0zd68xzpibcqc9y 2419162 2419161 2026-05-26T09:30:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw 2419162 wikitext text/x-wiki g34eaxnrquoo7mqygip468b2r7vl1re 2419163 2419162 2026-05-26T09:31:45Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1226.eqiad.wmnet with OS trixie 2419163 wikitext text/x-wiki 6gvmy5llbd7uyd6432htrmm4r35p4t4 2419164 2419163 2026-05-26T09:32:49Z Stashbot 7414 fabfur: depooling cp2043 to install haproxy-awslc (T419825) 2419164 wikitext text/x-wiki bvjpaeogcky7i4u5z334x8sbhdkz3rp 2419165 2419164 2026-05-26T09:33:01Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293661|hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] (duration: 06m 52s) 2419165 wikitext text/x-wiki 04gsy1fkyd3lpio587lrq3mwhk6c2w6 2419166 2419165 2026-05-26T09:33:03Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* 2419166 wikitext text/x-wiki 43kgzdqejqsgq6ej7fvn5g2pgk5ilw3 2419167 2419166 2026-05-26T09:33:47Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet 2419167 wikitext text/x-wiki 5c6mc6x64ts6jx22i7w1r2hihfkf3lh 2419168 2419167 2026-05-26T09:33:49Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet 2419168 wikitext text/x-wiki dodhex05y886mcn6sv9q7ujemn1ht9g 2419169 2419168 2026-05-26T09:33:58Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet 2419169 wikitext text/x-wiki 9j0kwaq55b2oi15mbnldk258ykyjhu9 2419170 2419169 2026-05-26T09:34:34Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2044.* 2419170 wikitext text/x-wiki c2is012phgg2u11ecr46andgwsybify 2419171 2419170 2026-05-26T09:34:37Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet 2419171 wikitext text/x-wiki duvb4mj308im0vmak9i8y7seiwvw3yo 2419172 2419171 2026-05-26T09:34:39Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet 2419172 wikitext text/x-wiki mt151us7l98fgua36hfqeke0pv1luum 2419173 2419172 2026-05-26T09:34:47Z Stashbot 7414 fabfur: depooling cp2044 to install haproxy-awslc (T419825) 2419173 wikitext text/x-wiki sud2qvepmuvrvv5seord9zvl8ulr9nf 2419174 2419173 2026-05-26T09:38:12Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2419174 wikitext text/x-wiki 7gautd5nzcrkkdcq4opbhyxh6pukzwz 2419175 2419174 2026-05-26T09:39:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2419175 wikitext text/x-wiki rcj7gmzvfdik97ozosw3ufxeusy10xg 2419176 2419175 2026-05-26T09:39:12Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw 2419176 wikitext text/x-wiki dl5bpwrcw6pjvurhld9msqff7k0xx65 2419177 2419176 2026-05-26T09:40:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1226: Migration of db1226.eqiad.wmnet completed 2419177 wikitext text/x-wiki 9xp6jxuyaikbbkik8tk8mkrl0yizk97 2419178 2419177 2026-05-26T09:40:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T426633)', diff saved to https://phabricator.wikimedia.org/P92950 and previous config saved to /var/cache/conftool/dbconfig/20260526-094045-fceratto.json 2419178 wikitext text/x-wiki e5gpmddk8f8ng30jck4xgs3het1jges 2419179 2419178 2026-05-26T09:41:02Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs3009.esams.wmnet} and A:liberica 2419179 wikitext text/x-wiki jk0mut73t6a1988k190oh0qhqmobfhu 2419180 2419179 2026-05-26T09:41:09Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: Maintenance 2419180 wikitext text/x-wiki q3u5knk3f2pvdd8mfhanhufhjniowd1 2419181 2419180 2026-05-26T09:41:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 (T426633)', diff saved to https://phabricator.wikimedia.org/P92951 and previous config saved to /var/cache/conftool/dbconfig/20260526-094115-fceratto.json 2419181 wikitext text/x-wiki 88ewyd0scdggqliu4r7l0nx1cqjn7pv 2419182 2419181 2026-05-26T09:41:22Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet 2419182 wikitext text/x-wiki 4wnu260fdzpojlhprvpjgyi4ta7cx77 2419183 2419182 2026-05-26T09:41:24Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet 2419183 wikitext text/x-wiki 5eveb5xbuqw9yuxloxz0dzkoq2fj9e0 2419184 2419183 2026-05-26T09:41:34Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet 2419184 wikitext text/x-wiki 0t9txsgs5yuqxf1ds3k8jftd65z6j9v 2419186 2419184 2026-05-26T09:44:04Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293665|hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] 2419186 wikitext text/x-wiki 87f8bg44z4k4jak2jdlck4kt4a22r3t 2419187 2419186 2026-05-26T09:44:57Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs3009.esams.wmnet} and A:liberica 2419187 wikitext text/x-wiki d1i8od7r08iwxbuwhb90cmiqzio6qwa 2419188 2419187 2026-05-26T09:45:56Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293665|hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419188 wikitext text/x-wiki otqq89bcypmehgja5rxat9bv4o3346z 2419189 2419188 2026-05-26T09:46:41Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet 2419189 wikitext text/x-wiki psdr8zej4vowo7nzjo4f6qpb5mus3ev 2419190 2419189 2026-05-26T09:48:00Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2419190 wikitext text/x-wiki at206mm4nio02pr03tlenmbk1lsaljt 2419191 2419190 2026-05-26T09:48:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 (T426633)', diff saved to https://phabricator.wikimedia.org/P92953 and previous config saved to /var/cache/conftool/dbconfig/20260526-094819-fceratto.json 2419191 wikitext text/x-wiki rbol4i9orc4yahaqmwdj6sgx72wt0w5 2419192 2419191 2026-05-26T09:48:44Z Stashbot 7414 fabfur: repooling cp2043 and cp2044 (haproxy-awslc) (T419825) 2419192 wikitext text/x-wiki 1p065w6pyatii9fyytkl73zpu4bo9tr 2419193 2419192 2026-05-26T09:51:37Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2044.* 2419193 wikitext text/x-wiki ohtrzs88fjyu9br89oihczcum05tk8j 2419194 2419193 2026-05-26T09:51:41Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* 2419194 wikitext text/x-wiki loq1x1rg8rslsnzl4ij6351xomo6z7l 2419195 2419194 2026-05-26T09:52:11Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293665|hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] (duration: 08m 07s) 2419195 wikitext text/x-wiki fwvfupztavkxvirzxasdhdwuhgv041v 2419196 2419195 2026-05-26T09:52:19Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-eqiad@eqiad 2419196 wikitext text/x-wiki 3its2xpjcczpw0p05wbef9zubhpyrdp 2419197 2419196 2026-05-26T09:53:37Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet 2419197 wikitext text/x-wiki o6p64aubbabkxikmhf9bbjmr79y86f3 2419199 2419197 2026-05-26T09:53:40Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet 2419199 wikitext text/x-wiki 3x3dit91eodmh3r2broakuvi4m0001g 2419200 2419199 2026-05-26T09:53:43Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{kubestage100*} and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) 2419200 wikitext text/x-wiki 31a115yv5fbjd9rczj75vksfvluu1ev 2419201 2419200 2026-05-26T09:54:22Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet 2419201 wikitext text/x-wiki dmgdkm979q295jts36i8xad1ubvcvg6 2419202 2419201 2026-05-26T09:54:24Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet 2419202 wikitext text/x-wiki tlb4t6763k8je35rwob3ifyfvslka1j 2419203 2419202 2026-05-26T09:54:34Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet 2419203 wikitext text/x-wiki 4zdehroj5u52zj9jxk02s4y2qns57r4 2419204 2419203 2026-05-26T09:55:08Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet 2419204 wikitext text/x-wiki kv4sd8tan592dfa6vg5yke9w4kotaxc 2419205 2419204 2026-05-26T09:55:37Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2419205 wikitext text/x-wiki mctlzsgnhpxhbp0eg3vvqo4pbnovyno 2419206 2419205 2026-05-26T09:55:47Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2419206 wikitext text/x-wiki 3civglhxhrrkjd99hgu22wokn47qww3 2419207 2419206 2026-05-26T09:56:00Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2419207 wikitext text/x-wiki agmxcue87rfj5qnahk8rnb9xa3jvlx5 2419208 2419207 2026-05-26T09:56:43Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2419208 wikitext text/x-wiki ldrijvml781nkn9f92627twcmoyvpmk 2419209 2419208 2026-05-26T09:56:46Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-eqiad@eqiad 2419209 wikitext text/x-wiki sz4b5s2vawfocr7jbvpz3pmr5yhbow9 2419210 2419209 2026-05-26T09:57:45Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2419210 wikitext text/x-wiki elo6cwqjzu0ehbhtbd5tjuvk931okmn 2419211 2419210 2026-05-26T09:58:07Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2419211 wikitext text/x-wiki doj6a60au1sksuaw9oxrv5j7b5v6tkv 2419212 2419211 2026-05-26T09:58:13Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2419212 wikitext text/x-wiki rc1foq403pz0omarg7lw91wvkrwnfzs 2419213 2419212 2026-05-26T09:58:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92955 and previous config saved to /var/cache/conftool/dbconfig/20260526-095827-fceratto.json 2419213 wikitext text/x-wiki gsha8sq519fma1pzzyqnx02tqw316uf 2419214 2419213 2026-05-26T09:58:34Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2419214 wikitext text/x-wiki rqqtfjejpjmx2iz394ofyftpw0oo8bt 2419215 2419214 2026-05-26T10:00:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox 2419215 wikitext text/x-wiki 4uhflo4sc105e3dknjpw9ht7xioi3vw 2419216 2419215 2026-05-26T10:01:51Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet 2419216 wikitext text/x-wiki fqfqitpha42bjjj6c29oywkzcjtqh3r 2419217 2419216 2026-05-26T10:01:54Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet 2419217 wikitext text/x-wiki aofnhz7ez9t4t1ydpyfx5laz2liyqxm 2419218 2419217 2026-05-26T10:01:57Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{kubestage200*} and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) 2419218 wikitext text/x-wiki avqeox7hlza1abuye9xoj8hpwrc6v8b 2419219 2419218 2026-05-26T10:02:13Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Migration of db2195.codfw.wmnet completed 2419219 wikitext text/x-wiki c93d27p8j0joa806bupoymwt3stjnz7 2419220 2419219 2026-05-26T10:02:16Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2419220 wikitext text/x-wiki fj1n45fskvb0qahnh2nuvqukpm3k4ws 2419221 2419220 2026-05-26T10:04:00Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293668|hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] 2419221 wikitext text/x-wiki rgxszbd5ag6tozqgk3hr0p3p81z9j3m 2419222 2419221 2026-05-26T10:05:53Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293668|hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419222 wikitext text/x-wiki ch6nyxtuemj2lsxj5s8a2gfsd51qoad 2419223 2419222 2026-05-26T10:06:36Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2419223 wikitext text/x-wiki mrqm6krm8gk2k6iclt47sjr0g515d9f 2419224 2419223 2026-05-26T10:08:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92957 and previous config saved to /var/cache/conftool/dbconfig/20260526-100834-fceratto.json 2419224 wikitext text/x-wiki 12y26hn7xp5ewl6qh30jxu0g2ydhpc0 2419225 2419224 2026-05-26T10:09:33Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw 2419225 wikitext text/x-wiki ea1bqucpdonmvpxdp58xg8296ebrqp2 2419226 2419225 2026-05-26T10:10:43Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293668|hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] (duration: 06m 42s) 2419226 wikitext text/x-wiki jv0aoeqd5py27bjt8ymo43e8tg8g498 2419227 2419226 2026-05-26T10:15:41Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2419227 wikitext text/x-wiki swrjce24hhfhd8f2jdvl75cfzgyj7we 2419228 2419227 2026-05-26T10:16:38Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2419228 wikitext text/x-wiki 2glnb2cayjj12m69to026rmtsiije21 2419229 2419228 2026-05-26T10:16:40Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw 2419229 wikitext text/x-wiki 6496vh80gvs7q2yymftxfxebaqahkwu 2419230 2419229 2026-05-26T10:18:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 (T426633)', diff saved to https://phabricator.wikimedia.org/P92959 and previous config saved to /var/cache/conftool/dbconfig/20260526-101842-fceratto.json 2419230 wikitext text/x-wiki db1hxq2mw6bgoi67hzq8a934dtaszrl 2419231 2419230 2026-05-26T10:19:31Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance 2419231 wikitext text/x-wiki gv0v6x736zfqzjw5rjhujomxzo821wg 2419232 2419231 2026-05-26T10:19:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2175 (T426633)', diff saved to https://phabricator.wikimedia.org/P92960 and previous config saved to /var/cache/conftool/dbconfig/20260526-101936-fceratto.json 2419232 wikitext text/x-wiki 1ps1lbpaje4gyvqwtb5q9uyay639rru 2419234 2419232 2026-05-26T10:24:13Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419234 wikitext text/x-wiki rsisuat0k7se66nd7d1ts0n2paufevg 2419235 2419234 2026-05-26T10:24:36Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2181: Upgrading db2181.codfw.wmnet 2419235 wikitext text/x-wiki 9mwzb9hbbgjp3sxqxyg82o2wo8c2tfl 2419236 2419235 2026-05-26T10:25:05Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2181: Upgrading db2181.codfw.wmnet 2419236 wikitext text/x-wiki fcacyk37gczjsvbilw7drnfl7677h0m 2419237 2419236 2026-05-26T10:25:58Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1226: Migration of db1226.eqiad.wmnet completed 2419237 wikitext text/x-wiki a9mcvlm8lgpp2k1tmjww5va56oz1ebv 2419238 2419237 2026-05-26T10:26:00Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2419238 wikitext text/x-wiki dxk099740w8ekxejadzat22aslm2bq0 2419239 2419238 2026-05-26T10:27:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T426633)', diff saved to https://phabricator.wikimedia.org/P92963 and previous config saved to /var/cache/conftool/dbconfig/20260526-102703-fceratto.json 2419239 wikitext text/x-wiki kdob6sxollm63tarlej4xq134gj2row 2419240 2419239 2026-05-26T10:32:46Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2419240 wikitext text/x-wiki m9fwavudmw62bkpq64u02qemvbu685z 2419241 2419240 2026-05-26T10:33:11Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2419241 wikitext text/x-wiki 4lz9w4u32h84fzt834xad4o7m75bf7f 2419242 2419241 2026-05-26T10:36:25Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2181.codfw.wmnet with OS trixie 2419242 wikitext text/x-wiki az8hrqnthfo06nb3x0dfo53pb2nxmgk 2419243 2419242 2026-05-26T10:37:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92964 and previous config saved to /var/cache/conftool/dbconfig/20260526-103711-fceratto.json 2419243 wikitext text/x-wiki 5x3aifu9nf61ywxkgvqlqj9ogd10nkp 2419244 2419243 2026-05-26T10:44:54Z Stashbot 7414 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1293095|ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] 2419244 wikitext text/x-wiki 52u3vz770eggq6s6m7ukt7ele42ppn5 2419245 2419244 2026-05-26T10:46:48Z Stashbot 7414 jiji@deploy1003: jiji: Backport for [[gerrit:1293095|ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419245 wikitext text/x-wiki d6vg4wgk41tj0m4mur3y3ycu1any59n 2419246 2419245 2026-05-26T10:47:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92966 and previous config saved to /var/cache/conftool/dbconfig/20260526-104718-fceratto.json 2419246 wikitext text/x-wiki si854podvf2g4wx50dag5ugtktw5o12 2419247 2419246 2026-05-26T10:51:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: host reimage 2419247 wikitext text/x-wiki ehbfmf3rujktvemxf58hkfa635sr2r5 2419248 2419247 2026-05-26T10:55:41Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: host reimage 2419248 wikitext text/x-wiki b4ctywbmc2s35d2t1lrraslonkx9uby 2419249 2419248 2026-05-26T10:56:32Z Stashbot 7414 jiji@deploy1003: jiji: Continuing with deployment 2419249 wikitext text/x-wiki mrms5175rwlvcvpbzcaag86g2nsht5t 2419250 2419249 2026-05-26T10:57:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T426633)', diff saved to https://phabricator.wikimedia.org/P92967 and previous config saved to /var/cache/conftool/dbconfig/20260526-105726-fceratto.json 2419250 wikitext text/x-wiki k66p6srf945a2nuh6nb9myhjwg7gphn 2419251 2419250 2026-05-26T10:57:49Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance 2419251 wikitext text/x-wiki g65m6pajuzhaaqj4k4nmp7me2vgxbfo 2419252 2419251 2026-05-26T10:57:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2189 (T426633)', diff saved to https://phabricator.wikimedia.org/P92968 and previous config saved to /var/cache/conftool/dbconfig/20260526-105755-fceratto.json 2419252 wikitext text/x-wiki exd1kaam6wsgr4nord7f8suerf7k3yq 2419253 2419252 2026-05-26T10:59:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2419253 wikitext text/x-wiki 14oym8bi7zjx8y3nsoguhkvvq6divmq 2419254 2419253 2026-05-26T10:59:28Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1214: Upgrading db1214.eqiad.wmnet 2419254 wikitext text/x-wiki dnigs2hj7gdsw2q0rxd672sbumaoh7u 2419255 2419254 2026-05-26T11:00:17Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1214: Upgrading db1214.eqiad.wmnet 2419255 wikitext text/x-wiki 6v90fdsjgfty5qm13ruyar2pflverca 2419256 2419255 2026-05-26T11:00:44Z Stashbot 7414 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293095|ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] (duration: 15m 50s) 2419256 wikitext text/x-wiki r4cu4ykv5zi6mksnsoonalwthlpwaza 2419257 2419256 2026-05-26T11:02:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1214.eqiad.wmnet with OS trixie 2419257 wikitext text/x-wiki 04n09b1ws5qqkli987m7hb6b0lriez8 2419258 2419257 2026-05-26T11:05:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T426633)', diff saved to https://phabricator.wikimedia.org/P92971 and previous config saved to /var/cache/conftool/dbconfig/20260526-110458-fceratto.json 2419258 wikitext text/x-wiki 11tbrgs7x7138xj3vkllt5tgmioqzp7 2419259 2419258 2026-05-26T11:14:17Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2181.codfw.wmnet with OS trixie 2419259 wikitext text/x-wiki jt1a5jrwphq6s51i3455eivi67cf2t9 2419260 2419259 2026-05-26T11:15:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92972 and previous config saved to /var/cache/conftool/dbconfig/20260526-111506-fceratto.json 2419260 wikitext text/x-wiki opv2pro73l669be89jtwkiq0zlqfzsh 2419261 2419260 2026-05-26T11:17:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage 2419261 wikitext text/x-wiki t1rugfr3u0pefahlee1flhdt5sojr2f 2419262 2419261 2026-05-26T11:20:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Switchover es2042 es2041 for T426199', diff saved to https://phabricator.wikimedia.org/P92974 and previous config saved to /var/cache/conftool/dbconfig/20260526-112028-fceratto.json 2419262 wikitext text/x-wiki qvxay0dhl778u9lgqpdnkt08b9hddqm 2419263 2419262 2026-05-26T11:22:19Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1024 to dbctl T418973', diff saved to https://phabricator.wikimedia.org/P92975 and previous config saved to /var/cache/conftool/dbconfig/20260526-112215-marostegui.json 2419263 wikitext text/x-wiki 8h4u57mdri9o98vl9rdienx2fgjfvfc 2419264 2419263 2026-05-26T11:22:53Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2181: Migration of db2181.codfw.wmnet completed 2419264 wikitext text/x-wiki ackjsdq1iv85laj2ewxafav3hk0c10a 2419265 2419264 2026-05-26T11:23:30Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc4 T418973', diff saved to https://phabricator.wikimedia.org/P92977 and previous config saved to /var/cache/conftool/dbconfig/20260526-112326-marostegui.json 2419265 wikitext text/x-wiki 334y9uuut0hpinjxlsata4s1zzhm7jp 2419266 2419265 2026-05-26T11:24:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage 2419266 wikitext text/x-wiki 7v1zeevalv008yc6ibzqduev9uv1di9 2419268 2419266 2026-05-26T11:25:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92978 and previous config saved to /var/cache/conftool/dbconfig/20260526-112513-fceratto.json 2419268 wikitext text/x-wiki npouhlvftp5113qtqy9v90qrdylc4ik 2419269 2419268 2026-05-26T11:29:03Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293691|Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] 2419269 wikitext text/x-wiki rr01p86oa33y7sz2ol6o1tftppfggz5 2419270 2419269 2026-05-26T11:30:33Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Migration of db1222.eqiad.wmnet completed 2419270 wikitext text/x-wiki fu2heioduxhvtu25lq67nbd2pt4tlna 2419271 2419270 2026-05-26T11:30:36Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2419271 wikitext text/x-wiki 9dj6o4pk5olpadwill76q2vt5fw4z9w 2419272 2419271 2026-05-26T11:31:01Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1293691|Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419272 wikitext text/x-wiki 40qr86ihpdbasumcot5cqp03gnq2tqt 2419273 2419272 2026-05-26T11:31:37Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment 2419273 wikitext text/x-wiki pww3hboh6mgmr5pedr3kp09p2d9zuya 2419274 2419273 2026-05-26T11:35:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T426633)', diff saved to https://phabricator.wikimedia.org/P92980 and previous config saved to /var/cache/conftool/dbconfig/20260526-113521-fceratto.json 2419274 wikitext text/x-wiki d66yuds7ud8dq0cplh8q555yt5drvpx 2419275 2419274 2026-05-26T11:35:37Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance 2419275 wikitext text/x-wiki fzejkopqe7alxeiflk2qswajsqd12q8 2419276 2419275 2026-05-26T11:35:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 (T426633)', diff saved to https://phabricator.wikimedia.org/P92981 and previous config saved to /var/cache/conftool/dbconfig/20260526-113542-fceratto.json 2419276 wikitext text/x-wiki oqcmlcn472rk00dn6p0g0wfkdscwp8j 2419277 2419276 2026-05-26T11:35:51Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293691|Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] (duration: 06m 46s) 2419277 wikitext text/x-wiki klt6t7j701hdx5on3hx9qamuw6447kl 2419278 2419277 2026-05-26T11:41:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1214.eqiad.wmnet with OS trixie 2419278 wikitext text/x-wiki sdzbfu2btxoit5rwvt4najfs4ibvl1n 2419279 2419278 2026-05-26T11:42:15Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet 2419279 wikitext text/x-wiki nkfxgk951ldafqk1mlcum1jksxqwdog 2419280 2419279 2026-05-26T11:42:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T426633)', diff saved to https://phabricator.wikimedia.org/P92983 and previous config saved to /var/cache/conftool/dbconfig/20260526-114243-fceratto.json 2419280 wikitext text/x-wiki 0cuqoezpcxvauxpfemm8zifl2xrf5ku 2419281 2419280 2026-05-26T11:44:09Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293167|hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055|hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] 2419281 wikitext text/x-wiki bp9g15p93uwof4nxnuaai6fzktg3sva 2419282 2419281 2026-05-26T11:45:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet 2419282 wikitext text/x-wiki e5vg7xroiil6nri5iah92nikeqirnmg 2419283 2419282 2026-05-26T11:46:03Z Stashbot 7414 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1293167|hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055|hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2419283 wikitext text/x-wiki cgrg5sqsgdymtmfh0gtdjamiz5b5ac6 2419284 2419283 2026-05-26T11:47:45Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet 2419284 wikitext text/x-wiki e4usaav8l4eb5767gj25cdelglt1zs1 2419285 2419284 2026-05-26T11:49:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet 2419285 wikitext text/x-wiki qkazzow3rm75o2ptas64fjihmwq31dh 2419286 2419285 2026-05-26T11:50:00Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1214: Migration of db1214.eqiad.wmnet completed 2419286 wikitext text/x-wiki gk4hkjrila1bbjhk8jspryg4pxrw5gl 2419287 2419286 2026-05-26T11:51:27Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet 2419287 wikitext text/x-wiki fravcdorxag3msnv6wkyyiyev1o8gwk 2419288 2419287 2026-05-26T11:52:43Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2419288 wikitext text/x-wiki mw2u0ht4kc975e99udmpyu0ksgxr4vf 2419289 2419288 2026-05-26T11:52:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92985 and previous config saved to /var/cache/conftool/dbconfig/20260526-115251-fceratto.json 2419289 wikitext text/x-wiki 37iarkbnom3wz5cjf2v11imkg6x4m73 2419290 2419289 2026-05-26T11:54:22Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2419290 wikitext text/x-wiki kc90jy0nh1i9z7ywyfgrnbrrg57a984 2419291 2419290 2026-05-26T11:54:54Z Stashbot 7414 jynus: stopping mediabackups@codfw for maintenance on a codfw backup media storage server T426199 2419291 wikitext text/x-wiki 951iqldiitptuftrwifyn7j01rnhql4 2419292 2419291 2026-05-26T11:55:21Z Stashbot 7414 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment 2419292 wikitext text/x-wiki ha3ag2wc22950skwwp9sg6qdvzio7cl 2419293 2419292 2026-05-26T11:55:29Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet 2419293 wikitext text/x-wiki efmtlqtbc8l4ji2o5jwwfslp5dlhugs 2419294 2419293 2026-05-26T11:56:39Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup2015.codfw.wmnet,db2197.codfw.wmnet with reason: network maintenance 2419294 wikitext text/x-wiki h2bvp5yxdk4lu1l22pp3u2ed5lvowns 0 17290 2419123 2418232 2026-05-26T09:05:27Z Stashbot 7414 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1292620 (Introduce Phan composer job - T231966) 2419123 wikitext text/x-wiki rkr2kwhgwl471ua9zn30pd8ikebnh26 0 441012 2418830 2394099 2026-05-25T14:45:30Z Filippo Giunchedi 2939 /* Community Requests */ Make Portal:Cloud_VPS/Admin/Projects_lifecycle#Requests canonical 2418830 wikitext text/x-wiki 8e6a7ki0tbsyvvdxcl89md60a9djtwk 498 443800 2418772 2418722 2026-05-25T13:24:37Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed e867fb34a0 (support proofread-page content model) 2418772 wikitext text/x-wiki p2e3yc2m3lrkvqkf6ebfv2bjyjr70op 498 443946 2418802 2418366 2026-05-25T13:51:12Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed c163567ccd (l10n updates: nb) 2418802 wikitext text/x-wiki b4ih0hrm3yqwmw55mk8k2l8bcpvnitv 498 443947 2418798 2413851 2026-05-25T13:50:00Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 2d3ce61727 (l10n updates: nl) 2418798 wikitext text/x-wiki 72vtpa5a11e231o36x04d2ujtw66h88 0 446759 2418965 2401106 2026-05-26T06:44:51Z MArostegui (WMF) 8620 dborch1001 is no longer 2418965 wikitext text/x-wiki hnuu18huq74nz7urowj9vig82k2jecu 498 447034 2418799 2418350 2026-05-25T13:50:04Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed e5d776187d (l10n updates: ko) 2418799 wikitext text/x-wiki rcu4puoz9xlv845qf13s71w8uu4tiyy 0 447295 2418941 2402148 2026-05-26T05:46:57Z MArostegui (WMF) 8620 Add new section 2418941 wikitext text/x-wiki ga2kbgtxuy2zfwmq7cbhf64bxfrs5ce 2418942 2418941 2026-05-26T05:47:20Z MArostegui (WMF) 8620 2418942 wikitext text/x-wiki sfbuj6d02ksk5dpy6orcx1f8u0zua2d 2418943 2418942 2026-05-26T05:48:02Z MArostegui (WMF) 8620 2418943 wikitext text/x-wiki e7063vl8oep9aozsqt5sqde4nfi9kel 0 449160 2418865 2418703 2026-05-25T16:37:28Z Dexbot 30554 Bot: Updating the report 2418865 wikitext text/x-wiki 3fqekvgqmf6en2y58xw57hmh9x2pmwz 2418927 2418865 2026-05-26T00:02:25Z Dexbot 30554 Bot: Updating the report 2418927 wikitext text/x-wiki mlxv5d5gpcnk7yrri57gcts9x2xrhif 2418950 2418927 2026-05-26T06:06:12Z Dexbot 30554 Bot: Updating the report 2418950 wikitext text/x-wiki i02lxiv4r17nylz69hu752iclphqos2 2418961 2418950 2026-05-26T06:33:16Z Dexbot 30554 Bot: Updating the report 2418961 wikitext text/x-wiki 24u9deicp1u8g617g9hy3k5vnz89x4g 2418971 2418961 2026-05-26T06:51:19Z Dexbot 30554 Bot: Updating the report 2418971 wikitext text/x-wiki 6vybd5parq5qvb9m5s9iiie6o21dvto 2419233 2418971 2026-05-26T10:21:52Z Dexbot 30554 Bot: Updating the report 2419233 wikitext text/x-wiki ibv0wvx99y6p7p9hdmqg2b9af1nkacl 2419267 2419233 2026-05-26T11:25:02Z Dexbot 30554 Bot: Updating the report 2419267 wikitext text/x-wiki rtoh8j1u9vi97qyput85e4vp1xcbaa3 116 453426 2418828 2418627 2026-05-25T14:40:08Z Phabbanbot 37210 Lysdexia was disabled by HakanIST 2418828 wikitext text/x-wiki o68o59uk8neecyxjniwuhcbyn7ojhyv 116 453906 2418909 2418628 2026-05-25T19:48:23Z Gitlabaccountapprovalbot 37332 lakz-99 was rejected. 2418909 wikitext text/x-wiki psovw4trubiy2lyazf5uctslvro8dfr 2418910 2418909 2026-05-25T19:51:19Z Gitlabaccountapprovalbot 37332 lakz-99 was rejected. 2418910 wikitext text/x-wiki gpn7d3an220755vh15vq96odzsvoxxz 2418925 2418910 2026-05-25T22:54:16Z Gitlabaccountapprovalbot 37332 ddshelto was rejected. 2418925 wikitext text/x-wiki 61dh428t93to5r3v332szpqz28ic3a2 12 457949 2419098 2339188 2026-05-26T08:46:17Z Redmin 23906 /* Caveats */ 2419098 wikitext text/x-wiki lyus7cye728uzy6i3emza0ivcpry1jl 0 459290 2418835 2417541 2026-05-25T15:14:35Z SNwachukwu (WMF) 39521 2418835 wikitext text/x-wiki rc7z80shocxi1us2qouwe9oqtc882t6 2419198 2418835 2026-05-26T09:53:38Z APizzata-WMF 47941 add mediawiki content private deployment 2419198 wikitext text/x-wiki 359ehuj2ago6g97w40s17gxlhukfylj 0 459699 2418928 2417491 2026-05-26T01:05:40Z Stashbot 7414 eileen: civicrm upgraded from 6b4e53a5 to 713e8508 2418928 wikitext text/x-wiki aaqmygjpz0ajsrwy3zmkgdef32opo09 0 459948 2419185 2402391 2026-05-26T09:42:53Z LToscano (WMF) 7191 /* Kubernetes API (control plane) */ 2419185 wikitext text/x-wiki 1sfm4hp7z9ekffc24ywpzibu8a5xqhd 2 460218 2418923 2026-05-25T20:52:57Z ABreault (WMF) 37749 Created page with "<imagemap> File:Pipeline tree - MediaWiki History and main child datasets.jpg|thumb|center|800px|alt=MediaWiki History and its main child datasets - clickable image|MediaWiki History and main child datasets. '''Tip:''' Most of the boxes are clickable and link to their documentation. rect 2360 444 3452 1236 [[Data_Platform/Systems/MediaWiki_replicas|Data_Platform/Systems/MediaWiki_replicas]] rect 4112 456 5244 1248 [[Help:Wiki_Replicas]] </imagemap>" 2418923 wikitext text/x-wiki at8a29twltuq1ztxnda77rmwcb76cqd