Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.9 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2432899 2432744 2026-07-02T16:45:27Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) 2432899 wikitext text/x-wiki j3y0vdnxmy27yy6nbjom5vw0pkjodvs 2432900 2432899 2026-07-02T16:45:32Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) 2432900 wikitext text/x-wiki pr3drfulijzavyvjvu4w0mminwt8wec 0 3788 2432998 2424485 2026-07-02T20:58:08Z BCornwall-WMF 31521 /* Historical TTL */ Update history link 2432998 wikitext text/x-wiki qn77vv5d5d9k6i5qdt9erz1d3h6d0fq 0 4108 2432861 2432754 2026-07-02T15:35:52Z ADancy (WMF) 17694 /* {{Deployment_day|date=2026-07-02}} */ Add 1306745 to next puppet window 2432861 wikitext text/x-wiki eqjepbnalha7ftuhqvnonsdwdyusvhx 2432896 2432861 2026-07-02T16:39:37Z ScheduleDeploymentBot 37566 Add [[gerrit:1307196]] to Thursday, July 02 UTC late backport window 2432896 wikitext text/x-wiki 20qd3cwq5qjb3d326tlyntunrxl2nyn 2432918 2432896 2026-07-02T16:56:36Z WBrown (WMF) 37696 /* {{Deployment_day|date=2026-07-07}} */ + 2432918 wikitext text/x-wiki m5213ndbptu36k80exy80r2b0ae6kx6 2432970 2432918 2026-07-02T18:51:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1307218]] to Thursday, July 02 UTC late backport window 2432970 wikitext text/x-wiki 0mpzkp6hjicx4obel0x18hv9ynuki8s 2432972 2432970 2026-07-02T19:07:05Z ScheduleDeploymentBot 37566 Add [[gerrit:1307227]] to Thursday, July 02 UTC late backport window 2432972 wikitext text/x-wiki ei77ma3o83vu6tcd55l48awfuixwfju 2432973 2432972 2026-07-02T19:07:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1307223]] to Thursday, July 02 UTC late backport window 2432973 wikitext text/x-wiki 0e7spzpoagwkoct32mdxhu63n2zqela 498 6726 2432897 2432715 2026-07-02T16:45:15Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) 2432897 wikitext text/x-wiki btsr52i4tacplzktohui2eh2kn5h52t 2432898 2432897 2026-07-02T16:45:20Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) 2432898 wikitext text/x-wiki nieoet3rfzsonk87ibzauixlor3sec0 0 7919 2432768 2432766 2026-07-02T12:07:56Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2432768 wikitext text/x-wiki 4oya91gclnkb93rop8ktnhjb6c47dbw 2432769 2432768 2026-07-02T12:10:03Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2432769 wikitext text/x-wiki jx6srnxi03xxvq78lvne3wwkkg3m89o 2432770 2432769 2026-07-02T12:19:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" 2432770 wikitext text/x-wiki 4jom68yk5qxaquils7mjpo55q0bqp5a 2432771 2432770 2026-07-02T12:19:36Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2432771 wikitext text/x-wiki gxvnz5fipw09hymvp1cvtaamhsq9naz 2432772 2432771 2026-07-02T12:19:37Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs-categories1001.eqiad.wmnet 2432772 wikitext text/x-wiki 68aoq6pd2wv2yk9xpb2m66qaecfaw9c 2432773 2432772 2026-07-02T13:02:26Z Stashbot 7414 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1305773|Phase 3 Legal contact link deployments. (T430227)]] 2432773 wikitext text/x-wiki 74uwye1jorm5bi964eo4eaufpe3qeol 2432774 2432773 2026-07-02T13:04:36Z Stashbot 7414 aude@deploy1003: jdrewniak, aude: Backport for [[gerrit:1305773|Phase 3 Legal contact link deployments. (T430227)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432774 wikitext text/x-wiki 1fvz9ziv9qxgo2jfgufmg41xl61ity0 2432775 2432774 2026-07-02T13:05:23Z Stashbot 7414 aude@deploy1003: jdrewniak, aude: Continuing with deployment 2432775 wikitext text/x-wiki soaefed34kudsw65uw0yz11kyncc2om 2432776 2432775 2026-07-02T13:09:47Z Stashbot 7414 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305773|Phase 3 Legal contact link deployments. (T430227)]] (duration: 07m 20s) 2432776 wikitext text/x-wiki 9sa8phmmqxposnzovxmhdxfp4atvon2 2432777 2432776 2026-07-02T13:11:03Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough 2432777 wikitext text/x-wiki jcim3tmdomfkcztdcxwktc4lf7geir9 2432778 2432777 2026-07-02T13:11:05Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=97) rolling restart_daemons on A:wikidough 2432778 wikitext text/x-wiki 9d199lx6lo32w5j37e3cytm9vufod9u 2432779 2432778 2026-07-02T13:11:13Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough 2432779 wikitext text/x-wiki 7uklqkc66h1d8uz0h502ox282n45o3l 2432780 2432779 2026-07-02T13:11:52Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) 2432780 wikitext text/x-wiki jgsul2t3fzpya5fza52ujdmcx97uwnv 2432781 2432780 2026-07-02T13:11:53Z Stashbot 7414 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart (exit_code=97) rolling restart_daemons on A:dnsbox and (A:dnsbox) 2432781 wikitext text/x-wiki fsjwfiinzlsw11j1a87cbbkbigfcaf0 2432782 2432781 2026-07-02T13:12:24Z Stashbot 7414 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) 2432782 wikitext text/x-wiki ayscyvanvwp6bw3as3bkrqp09h61x9n 2432783 2432782 2026-07-02T13:17:13Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns1004.wikimedia.org 2432783 wikitext text/x-wiki pc1c3pc4yg0y8ip398zy5kzjhkg09x8 2432784 2432783 2026-07-02T13:17:50Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw 2432784 wikitext text/x-wiki hj5oebj6x3ckxes6addrnsk4qddzj0l 2432785 2432784 2026-07-02T13:22:48Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw 2432785 wikitext text/x-wiki key8x117616c6z7syktrkit1w0h4xjl 2432786 2432785 2026-07-02T13:23:18Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw 2432786 wikitext text/x-wiki 3nb2aeu7swd3sl16lqq55z2xdo228il 2432787 2432786 2026-07-02T13:25:27Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough 2432787 wikitext text/x-wiki 9pvdoesrur0grgn198repm5y45zu5co 2432788 2432787 2026-07-02T13:26:18Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432788 wikitext text/x-wiki nfxj31te7becbtjop6qdhww7m02sjgc 2432790 2432788 2026-07-02T13:27:36Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432790 wikitext text/x-wiki os80dzxd6hkt402ys78g6x5nzkqui6h 2432791 2432790 2026-07-02T13:29:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432791 wikitext text/x-wiki dr3z56mkf6fpp11vt7e937vbozys2ga 2432792 2432791 2026-07-02T13:29:57Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432792 wikitext text/x-wiki eac21w5a2vfnad5rwxx2yjquoz0va85 2432793 2432792 2026-07-02T13:30:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432793 wikitext text/x-wiki hw8npx65ke690j3r8bfufwudxz449gm 2432794 2432793 2026-07-02T13:34:43Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw 2432794 wikitext text/x-wiki orfedh9fuiibt5z79qtzvkhq81ish1b 2432795 2432794 2026-07-02T13:36:06Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432795 wikitext text/x-wiki abvy0dxc6zopmoxgiybjvmej0ok3gm1 2432796 2432795 2026-07-02T13:36:33Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw 2432796 wikitext text/x-wiki mw4ddcgy0vvjudhjusage5bjqf613ye 2432797 2432796 2026-07-02T13:37:11Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw 2432797 wikitext text/x-wiki 2d6tle6zqkjgwra51lut7h2sg7oyof1 2432798 2432797 2026-07-02T13:38:58Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432798 wikitext text/x-wiki ovmv5ktkf9enlv87xq3z4fip552g35x 2432799 2432798 2026-07-02T13:40:39Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw 2432799 wikitext text/x-wiki ra70ghlbij8g5lomqg1dk6j2zuy87du 2432800 2432799 2026-07-02T13:44:08Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2432800 wikitext text/x-wiki taynchltcr6p06ywuv9rdxw91n311nv 2432801 2432800 2026-07-02T13:44:21Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2432801 wikitext text/x-wiki henfli4lk51892tws1fn4diu52v178y 2432802 2432801 2026-07-02T13:44:25Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2432802 wikitext text/x-wiki nonenjofuvrq1bjm802ut2gh71rq5yn 2432803 2432802 2026-07-02T13:47:10Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T430912 2432803 wikitext text/x-wiki p31d20xd0dre7qm5q94okcicz1bhlfp 2432804 2432803 2026-07-02T13:47:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2209 with weight 0 T430912', diff saved to https://phabricator.wikimedia.org/P94702 and previous config saved to /var/cache/conftool/dbconfig/20260702-134719-fceratto.json 2432804 wikitext text/x-wiki t7xt3oelq24fa2wik9equmw52o9db54 2432805 2432804 2026-07-02T13:48:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432805 wikitext text/x-wiki q9qgl3fv5ycxl673qw04jj508fcxgw1 2432806 2432805 2026-07-02T13:49:48Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432806 wikitext text/x-wiki a5g3yzhxspm3jtd5ay4muqfdfkmhd3y 2432808 2432806 2026-07-02T13:51:15Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2432808 wikitext text/x-wiki kw5j99kk7f3bsrr5wnydbefugleqk9x 2432809 2432808 2026-07-02T13:51:25Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2432809 wikitext text/x-wiki 5cpgq1cusv95p9txhhfcjp8es69076f 2432810 2432809 2026-07-02T13:52:02Z Stashbot 7414 federico3: Starting s3 codfw failover from db2205 to db2209 - T430912 2432810 wikitext text/x-wiki 4vcrhackr498pcwra0pqwfo575r57z5 2432811 2432810 2026-07-02T13:52:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2209 to s3 primary T430912', diff saved to https://phabricator.wikimedia.org/P94703 and previous config saved to /var/cache/conftool/dbconfig/20260702-135235-fceratto.json 2432811 wikitext text/x-wiki mfhicequxfoq4p7r1b6vmijjlhz1x37 2432812 2432811 2026-07-02T13:53:55Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432812 wikitext text/x-wiki 4y7gd86okt5uf12008a9893z8lkyvf1 2432813 2432812 2026-07-02T13:54:11Z Stashbot 7414 moritzm: installing sed security updates 2432813 wikitext text/x-wiki 3w3vatl6xe3816lyivo7rsnpbuezms1 2432814 2432813 2026-07-02T13:55:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2205 T430912', diff saved to https://phabricator.wikimedia.org/P94704 and previous config saved to /var/cache/conftool/dbconfig/20260702-135505-fceratto.json 2432814 wikitext text/x-wiki n8o2asa4cflgsnvzppxp8dk1zgyptqh 2432815 2432814 2026-07-02T13:55:14Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover 2432815 wikitext text/x-wiki rbv9wwbyykgxzmwhjty23oh1fgqv8kn 2432816 2432815 2026-07-02T13:59:33Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432816 wikitext text/x-wiki eu3tppo0gb0cl3873s865y01m13g79h 2432817 2432816 2026-07-02T13:59:47Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover 2432817 wikitext text/x-wiki eiq1f4y6wtbg9i1nhnuk14hjquqy6pj 2432818 2432817 2026-07-02T13:59:51Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover 2432818 wikitext text/x-wiki 58x42qbltcptw2ku0mmy9dd7hkubcj1 2432819 2432818 2026-07-02T14:04:46Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432819 wikitext text/x-wiki 4asiggojpdum6tvi7lyzuxakmahizm7 2432820 2432819 2026-07-02T14:06:43Z Stashbot 7414 Tran: Deployed patch for T427287 2432820 wikitext text/x-wiki 5k6xqcr30xfh270ft0ilzhg5qwedp7b 2432821 2432820 2026-07-02T14:06:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432821 wikitext text/x-wiki 64h6386on6obg5d4axh2ef8uzim0tzy 2432822 2432821 2026-07-02T14:07:06Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2432822 wikitext text/x-wiki 18u0gwppkql9osm4tobk2mgqsbr6yvy 2432824 2432822 2026-07-02T14:09:11Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover 2432824 wikitext text/x-wiki l8rwzr1yqap46r17boqijnuyasgvujv 2432825 2432824 2026-07-02T14:09:52Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2205.codfw.wmnet with reason: Maintenance 2432825 wikitext text/x-wiki 0blhigqeezlj5r4p7f6tiw68q7a6u0u 2432826 2432825 2026-07-02T14:10:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2205 (T426633)', diff saved to https://phabricator.wikimedia.org/P94706 and previous config saved to /var/cache/conftool/dbconfig/20260702-140959-fceratto.json 2432826 wikitext text/x-wiki rkvr2sxp28yyfgofmhznew047e6ggbr 2432827 2432826 2026-07-02T14:11:05Z Stashbot 7414 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) 2432827 wikitext text/x-wiki 8oxguc7s1f3133nf9j92antvwlxq6gl 2432830 2432827 2026-07-02T14:12:40Z Stashbot 7414 moritzm: installing rsync security updates 2432830 wikitext text/x-wiki 205gv1e2vcqklaokwg94wbqoasuqwuh 2432831 2432830 2026-07-02T14:16:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T426633)', diff saved to https://phabricator.wikimedia.org/P94707 and previous config saved to /var/cache/conftool/dbconfig/20260702-141621-fceratto.json 2432831 wikitext text/x-wiki ap2bbi88ec8uyqwhqhdavl1fa59pi86 2432833 2432831 2026-07-02T14:26:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94708 and previous config saved to /var/cache/conftool/dbconfig/20260702-142628-fceratto.json 2432833 wikitext text/x-wiki l7172jaru2ks5e71a2xysw9iplpbmt9 2432834 2432833 2026-07-02T14:32:00Z Stashbot 7414 moritzm: installing libdbi-perl security updates 2432834 wikitext text/x-wiki dm74pd65k893qwrb7j71cxlchngffpz 2432835 2432834 2026-07-02T14:36:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94709 and previous config saved to /var/cache/conftool/dbconfig/20260702-143636-fceratto.json 2432835 wikitext text/x-wiki ca6svm5v5jjhcwsoljw0y864ntd8yvl 2432836 2432835 2026-07-02T14:46:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T426633)', diff saved to https://phabricator.wikimedia.org/P94711 and previous config saved to /var/cache/conftool/dbconfig/20260702-144644-fceratto.json 2432836 wikitext text/x-wiki lhrtwyetbk29e7jusfln3n97wmsz7wl 2432837 2432836 2026-07-02T14:52:59Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Unblock taavi - oblivian@cumin1003" 2432837 wikitext text/x-wiki a1xcib76a1gnqcip5ez9qzve14c6i13 2432838 2432837 2026-07-02T14:53:00Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003 2432838 wikitext text/x-wiki pdznh49pchvhmtxaqy5mni3vtoz6tta 2432839 2432838 2026-07-02T14:53:05Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Security updates 2432839 wikitext text/x-wiki n43qdm1rodvpddai3a2x87g4dmfkgz0 2432840 2432839 2026-07-02T14:53:06Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2432840 wikitext text/x-wiki pjpadmvmn3tzwa04g3jhqoqs3u1yzjs 2432841 2432840 2026-07-02T14:53:13Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2432841 wikitext text/x-wiki iiuyiftf1v51omvlp4bmouzoa2f71q2 2432842 2432841 2026-07-02T14:53:14Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Security updates 2432842 wikitext text/x-wiki hzkseyh9cv6c5z4hwruo4micgt7df0y 2432843 2432842 2026-07-02T14:53:35Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2432843 wikitext text/x-wiki 8ms706h2vv0e9pjp02z9ha44d465inx 2432844 2432843 2026-07-02T14:53:50Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003 2432844 wikitext text/x-wiki 0655gv029qrhkp802l4oe2to2yjdq1d 2432845 2432844 2026-07-02T14:53:51Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Unblock taavi - oblivian@cumin1003" 2432845 wikitext text/x-wiki t7rg8b6thpans7xg230yhyzcfumjskp 2432846 2432845 2026-07-02T14:54:09Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2432846 wikitext text/x-wiki 34g7lecx10xse6tdjd9wm54htp8ue0h 2432847 2432846 2026-07-02T14:56:56Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2432847 wikitext text/x-wiki 6bvnw4au402wdiwjpmcr2p1gymnz5sq 2432848 2432847 2026-07-02T14:57:50Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2432848 wikitext text/x-wiki nb4bty0bhay0tsyh6rwmu3im2nllmxr 2432849 2432848 2026-07-02T15:08:33Z Stashbot 7414 moritzm: installing Tomcat security updates 2432849 wikitext text/x-wiki 89gaei7nh245vjfqpzvhe825zo54ay7 2432850 2432849 2026-07-02T15:13:38Z Stashbot 7414 moritzm: installing giflib security updates 2432850 wikitext text/x-wiki 7k2dwxqffzw7eag6wjena8lxahxlfm5 2432851 2432850 2026-07-02T15:15:14Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Security updates 2432851 wikitext text/x-wiki g7tcmu352ghokx1345mb5jfoxjfcasd 2432852 2432851 2026-07-02T15:15:15Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2432852 wikitext text/x-wiki 4nxti5ssvljv1hvg3t8l8kj80sslvbp 2432853 2432852 2026-07-02T15:15:26Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2432853 wikitext text/x-wiki 13zea8cytfnkt0uhbvn0xm3xrwyc30u 2432854 2432853 2026-07-02T15:15:27Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Security updates 2432854 wikitext text/x-wiki ozi9xtnbm5lzcs13shmp92etfwvy92e 2432855 2432854 2026-07-02T15:20:35Z Stashbot 7414 moritzm: installing busybox updates from trixie point release 2432855 wikitext text/x-wiki 4zhbcovhjnwd0m1rkybp00vh6ycfgv5 2432856 2432855 2026-07-02T15:24:18Z Stashbot 7414 moritzm: installing busybox updates from bookworm point release 2432856 wikitext text/x-wiki 2ai6uh6gm5yndhx3u5yj2wz1y8frz09 2432862 2432856 2026-07-02T15:42:12Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1003.eqiad.wmnet with OS bookworm 2432862 wikitext text/x-wiki 73h4sfd1uh3hipzvm7z8vjk4nxtwjcv 2432863 2432862 2026-07-02T15:45:49Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Security updates 2432863 wikitext text/x-wiki 6hyflg8k0unjgiwufz1zyouirpm6j48 2432864 2432863 2026-07-02T15:45:50Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2432864 wikitext text/x-wiki hib48eta2rt2xigcbkmthk9xb1f3vxo 2432865 2432864 2026-07-02T15:45:58Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2432865 wikitext text/x-wiki gz8oevgyhpjye773ke3jxukj04ecwu7 2432866 2432865 2026-07-02T15:45:59Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Security updates 2432866 wikitext text/x-wiki oyo96cp7fv60wilxvjkncgngbg7tgcj 2432868 2432866 2026-07-02T15:54:04Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage 2432868 wikitext text/x-wiki r2dcst0wd0o0ys815sx4jt4mgnhsf3c 2432869 2432868 2026-07-02T15:54:47Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2432869 wikitext text/x-wiki 4nj6nxhwbdg5du4204a7q457r91nj1n 2432870 2432869 2026-07-02T15:54:48Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2432870 wikitext text/x-wiki iscah3ebrnpt7936hrslm58bi88vxaf 2432871 2432870 2026-07-02T15:54:54Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2432871 wikitext text/x-wiki fcwj789ri2p874z1bo0a9hsxmv9hko5 2432872 2432871 2026-07-02T15:58:28Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage 2432872 wikitext text/x-wiki q2i8e2wzayjczybdm42hsseltxbkj0v 2432873 2432872 2026-07-02T16:08:38Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Security updates 2432873 wikitext text/x-wiki d4vpdrt7tjkelm3oi958mkrwyeaxyjl 2432874 2432873 2026-07-02T16:08:39Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.parsercache 2432874 wikitext text/x-wiki 4kk89pkh0p08jpxfi1vt0qvdoiunkta 2432875 2432874 2026-07-02T16:08:51Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2432875 wikitext text/x-wiki r11k5u4rk3otx37rnh4yc37kcc2x33o 2432876 2432875 2026-07-02T16:08:52Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Security updates 2432876 wikitext text/x-wiki 442d928x1mw54yoihmtas32eovpgrs7 2432877 2432876 2026-07-02T16:11:40Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet 2432877 wikitext text/x-wiki 6z1f99069u9k6egp64x74yw25vlv3sc 2432878 2432877 2026-07-02T16:11:56Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet 2432878 wikitext text/x-wiki 9zzm6sn9xfik9ilp9wiz9iycva9rh8v 2432879 2432878 2026-07-02T16:13:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1003.eqiad.wmnet with OS bookworm 2432879 wikitext text/x-wiki oxiybg9cbd94yt293urt3mqlnv45rpa 2432880 2432879 2026-07-02T16:16:21Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1004.eqiad.wmnet with OS bookworm 2432880 wikitext text/x-wiki mx0j8g9jdtlq9u3jnpjat59k0mvlbjv 2432881 2432880 2026-07-02T16:23:17Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: sync 2432881 wikitext text/x-wiki atmhjf7fqevcs6lbxot00o5qvh3mzt8 2432882 2432881 2026-07-02T16:23:34Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: sync 2432882 wikitext text/x-wiki ijzp7uv4smuv95023bjns09s6azk7xh 2432883 2432882 2026-07-02T16:24:27Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432883 wikitext text/x-wiki a8eslwq3qb0qafilr9icej4lh96uwic 2432884 2432883 2026-07-02T16:25:03Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet 2432884 wikitext text/x-wiki 5eh9568k0i3i3sey1wew3xkahusuqj5 2432885 2432884 2026-07-02T16:25:36Z Stashbot 7414 rzl@deploy1003: Started scap sync-world: T416623 2432885 wikitext text/x-wiki rfm237ilokcrbin8chfgxbp04j2u6oe 2432886 2432885 2026-07-02T16:25:46Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet 2432886 wikitext text/x-wiki c4g4359wrgxnmv3bf3selg61c6wdgoj 2432887 2432886 2026-07-02T16:26:35Z Stashbot 7414 rzl@deploy1003: rzl: T416623 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432887 wikitext text/x-wiki ig2rqz7in950wwt4xb3prsiabf3qa5w 2432888 2432887 2026-07-02T16:28:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage 2432888 wikitext text/x-wiki 5ofmguh117ypxen5o9goo9sngjmn4qs 2432889 2432888 2026-07-02T16:31:00Z Stashbot 7414 rzl@deploy1003: rzl: Continuing with deployment 2432889 wikitext text/x-wiki 24ofobu6z3rl9qvdv1rzz26555cp0ln 2432890 2432889 2026-07-02T16:33:37Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage 2432890 wikitext text/x-wiki 4scroyghj7ohrpgqtrij4d0eapmc637 2432891 2432890 2026-07-02T16:34:37Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432891 wikitext text/x-wiki jv9t241ehfotxb5nraqxfqv824wn38e 2432892 2432891 2026-07-02T16:35:15Z Stashbot 7414 rzl@deploy1003: Finished scap sync-world: T416623 (duration: 10m 19s) 2432892 wikitext text/x-wiki kt0hhdfxsrxkdyzmvn0gohytjyffb3n 2432893 2432892 2026-07-02T16:36:31Z Stashbot 7414 rzl@dns1004: START - running authdns-update 2432893 wikitext text/x-wiki l7oxirk1jcxpf2h7y4ev91wljkcul00 2432894 2432893 2026-07-02T16:37:21Z Stashbot 7414 rzl@dns1004: START - running authdns-update 2432894 wikitext text/x-wiki lqhvvr9mndyh4vtm5axa2ba7d9scvak 2432895 2432894 2026-07-02T16:39:29Z Stashbot 7414 rzl@dns1004: END - running authdns-update 2432895 wikitext text/x-wiki hlef2tfhmh0mkhqnoyldcusyitzkhy5 2432905 2432895 2026-07-02T16:46:11Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2432905 wikitext text/x-wiki mqrdgul5x9wioqy2vlg38a4m59cvif5 2432910 2432905 2026-07-02T16:49:35Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2432910 wikitext text/x-wiki 2ch97gox26rrvu1wemejuhidkjj1x9g 2432911 2432910 2026-07-02T16:52:52Z Stashbot 7414 ryankemper: [ml-serve-eqiad] Cleared out 1302 failed (Evicted) pods: `kubectl -n llm delete pods --field-selector=status.phase=Failed`, freeing calico-kube-controllers from OOM crashloop (evictions were caused by disk pressure) 2432911 wikitext text/x-wiki mx0nlchn2v095enl1w7w3ryhs9xiv4e 2432913 2432911 2026-07-02T16:53:44Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2432913 wikitext text/x-wiki 7vj2154bwasvx3sm8qqto8vze04b7bz 2432914 2432913 2026-07-02T16:55:21Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003" 2432914 wikitext text/x-wiki p351koe99habrwf9mmd0fljjhriqirs 2432915 2432914 2026-07-02T16:55:41Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003" 2432915 wikitext text/x-wiki 76rcbqz2wmui2f9khn6fkgjt7wjksb6 2432916 2432915 2026-07-02T16:55:43Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2432916 wikitext text/x-wiki 3ro0tpfvuwpy4pvmrf0dw8knjpl943j 2432917 2432916 2026-07-02T16:55:59Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2432917 wikitext text/x-wiki pgk4cljbvt5axuzk81hrali8p2zfzxp 2432919 2432917 2026-07-02T16:57:40Z Stashbot 7414 tappof: bump space for prometheus k8s-aux in eqiad 2432919 wikitext text/x-wiki fmflrmxssvgdxes3m3jgztjfc8byksl 2432920 2432919 2026-07-02T16:58:12Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2432920 wikitext text/x-wiki 7odarwe6dk6rbno9fc2o616nz1xaz95 2432921 2432920 2026-07-02T16:58:13Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1004.eqiad.wmnet with OS bookworm 2432921 wikitext text/x-wiki 9ni4k9fm5fubpy3gh41w24xeggzeswp 2432922 2432921 2026-07-02T16:58:46Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430879, restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -> wcqs2003.codfw.wmnet, repooling source-only afterwards 2432922 wikitext text/x-wiki 72r3yzxwemvm7s0awu6btijsozcojoo 2432923 2432922 2026-07-02T16:58:47Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Re-running - btullis@cumin1003" 2432923 wikitext text/x-wiki ghph9qsdbuw0dkmb70v587m63stvdx5 2432924 2432923 2026-07-02T16:58:49Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Re-running - btullis@cumin1003" 2432924 wikitext text/x-wiki buu0weckdbapsdkt9oqhwtxgdzhvjpm 2432925 2432924 2026-07-02T17:00:17Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1003.eqiad.wmnet with OS bookworm 2432925 wikitext text/x-wiki 7l11np0agzypud5tw99k1d8szsmtus9 2432926 2432925 2026-07-02T17:00:26Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432926 wikitext text/x-wiki 8pm6s68fj9cpavlqru3s1pv17irwsy4 2432927 2432926 2026-07-02T17:00:39Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs1003 2432927 wikitext text/x-wiki grhzphfjlnmp8kf720x3kftm18nw9nm 2432928 2432927 2026-07-02T17:00:40Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs1003 2432928 wikitext text/x-wiki 1qyrqgya1al1xh1rbwm82cv7iti7lbd 2432933 2432928 2026-07-02T17:03:34Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "running to make sure all updates are synced - cmooney@cumin1003" 2432933 wikitext text/x-wiki o1a4gbndss7e4pljgvyiq95epmcs1cx 2432934 2432933 2026-07-02T17:03:38Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "running to make sure all updates are synced - cmooney@cumin1003" 2432934 wikitext text/x-wiki dvislheg12v4m20vxhu9u7gfr1b28mt 2432935 2432934 2026-07-02T17:03:39Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432935 wikitext text/x-wiki peuvlhzjlqx044yaoulc2mlbm1aim3v 2432938 2432935 2026-07-02T17:05:08Z Stashbot 7414 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply 2432938 wikitext text/x-wiki tsozab8vltt6948uw43tz3532mesm1z 2432939 2432938 2026-07-02T17:05:20Z Stashbot 7414 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply 2432939 wikitext text/x-wiki epbdjh0blu6ql5laf8tutmrb16bjvrr 2432940 2432939 2026-07-02T17:07:43Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply 2432940 wikitext text/x-wiki k90diaaq4p32npga1m4x5kif1349kud 2432941 2432940 2026-07-02T17:08:01Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply 2432941 wikitext text/x-wiki sm6zuamcw54so3tngulr9csz6g91m7r 2432942 2432941 2026-07-02T17:08:12Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply 2432942 wikitext text/x-wiki eud0q5obyugzjy4bn7xahtxw10e9c6x 2432943 2432942 2026-07-02T17:08:37Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply 2432943 wikitext text/x-wiki skd2o5ynhsgaw266g3sppivn9tsgluu 2432945 2432943 2026-07-02T17:12:31Z Stashbot 7414 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on wcqs[2002-2003].codfw.wmnet,wcqs1002.eqiad.wmnet with reason: reimaging hosts 2432945 wikitext text/x-wiki iqcyu0wkv0sc1mn1w7h8t26aqj10ofz 2432946 2432945 2026-07-02T17:15:51Z Stashbot 7414 rscout@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2432946 wikitext text/x-wiki 6l4hsygtr61sahtwhs5ock9bm74cb1n 2432947 2432946 2026-07-02T17:16:15Z Stashbot 7414 rscout@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2432947 wikitext text/x-wiki tf6yyakax7trqx9h94yktflji6sz61z 2432948 2432947 2026-07-02T17:16:22Z Stashbot 7414 rscout@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2432948 wikitext text/x-wiki lbz6k2bpe5lfk97qalb3kvbyvtkp3g8 2432949 2432948 2026-07-02T17:16:41Z Stashbot 7414 rscout@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2432949 wikitext text/x-wiki gasixxhi3p2oekfhjc9rt1ossaiuwuf 2432950 2432949 2026-07-02T17:18:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage 2432950 wikitext text/x-wiki 3smgom6vrhl3648t0q6i2t1akvr6661 2432951 2432950 2026-07-02T17:21:43Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1292300|etcd: Ignore test-s4 from dbctl (T427059)]] 2432951 wikitext text/x-wiki 6twgob7hnkq1hpflafhub3wwu6b6l7m 2432952 2432951 2026-07-02T17:23:35Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1292300|etcd: Ignore test-s4 from dbctl (T427059)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432952 wikitext text/x-wiki t80ng8fpqege1exoo8wfyxz8se0ad3s 2432953 2432952 2026-07-02T17:24:49Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage 2432953 wikitext text/x-wiki blr9lvyi89nbam0i18t22v7swk0ld6t 2432954 2432953 2026-07-02T17:26:57Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2432954 wikitext text/x-wiki 6kv1iiqo41pxu0u0e5ukqa07zqvo4dx 2432955 2432954 2026-07-02T17:31:17Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292300|etcd: Ignore test-s4 from dbctl (T427059)]] (duration: 09m 33s) 2432955 wikitext text/x-wiki a9xx3lq3ma2dpeqgjr2fzmbf5wlg8bo 2432956 2432955 2026-07-02T17:44:19Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply 2432956 wikitext text/x-wiki amzbi38impm29geagik2813h4340tb6 2432957 2432956 2026-07-02T17:44:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply 2432957 wikitext text/x-wiki afnpf8qik0ubjjbm51wqb8367vant8u 2432959 2432957 2026-07-02T17:49:00Z Stashbot 7414 jasmine_: homer "cr*eqiad*" commit "Added new stacked control plane wikikube-ctrl1005" 2432959 wikitext text/x-wiki gdgs8b5qkjholwkpn5liqle3ehe6e71 2432960 2432959 2026-07-02T17:51:00Z Stashbot 7414 jasmine@cumin2002: conftool action : set/pooled=yes:weight=10; selector: name=wikikube-ctrl1005.eqiad.wmnet 2432960 wikitext text/x-wiki bs5xq9q3zo6owfblyo2bc228qmi7tis 2432961 2432960 2026-07-02T17:52:43Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet 2432961 wikitext text/x-wiki oesozyvfl37t7rkj1t59is4nq5186cc 2432962 2432961 2026-07-02T17:52:47Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet 2432962 wikitext text/x-wiki p50crf3zt1a9tdwumbrkxj8ziqu9lir 2432963 2432962 2026-07-02T17:58:24Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs1003.eqiad.wmnet with OS bookworm 2432963 wikitext text/x-wiki dh2zvqsyfa1n7lyhx484bsdjg0q3fyy 2432966 2432963 2026-07-02T18:13:48Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430879, restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -> wcqs2003.codfw.wmnet, repooling source-only afterwards 2432966 wikitext text/x-wiki fex2324eypake49idt0vg76troog3hx 2432967 2432966 2026-07-02T18:17:57Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2432967 wikitext text/x-wiki pik7maweioysr7bkt614jaqunrgk5i6 2432969 2432967 2026-07-02T18:39:31Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2432969 wikitext text/x-wiki tfy1i23huk81s1lfg1z7gcr7g84ufaq 2432976 2432969 2026-07-02T19:48:36Z Stashbot 7414 mutante: contint2003 - reimaging because of T430510#12067628 T418521 2432976 wikitext text/x-wiki qmh4b8qp3foi2gjuy0ksnhsmg5ke33h 2432977 2432976 2026-07-02T19:49:25Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint2003.wikimedia.org with OS trixie 2432977 wikitext text/x-wiki byydbxgoketbe912ke7cxvd8shb8dfo 2432978 2432977 2026-07-02T20:05:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T430879, restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling source-only afterwards 2432978 wikitext text/x-wiki ku3pn98xgsiuxc77e6bhnw8kjpte8np 2432979 2432978 2026-07-02T20:08:11Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint2003.wikimedia.org with reason: host reimage 2432979 wikitext text/x-wiki azj84sr5tymu572jczci73lk2w1ae8f 2432980 2432979 2026-07-02T20:08:32Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet 2432980 wikitext text/x-wiki 2eti4byqzyt3jrj55le5e7lfj2bs0dw 2432981 2432980 2026-07-02T20:08:48Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet 2432981 wikitext text/x-wiki pqqa3rhmel8slbjtbzkx8zt8ht4dvpk 2432983 2432981 2026-07-02T20:09:15Z Stashbot 7414 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1307196|mediawiki.action.edit.preview: Fix compat with `<button>`-buttons (T430956)]] 2432983 wikitext text/x-wiki eig84xuxjof35uqv6v42dhxmq3dseng 2432984 2432983 2026-07-02T20:11:11Z Stashbot 7414 sbassett@deploy1003: sbassett: Backport for [[gerrit:1307196|mediawiki.action.edit.preview: Fix compat with `<button>`-buttons (T430956)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2432984 wikitext text/x-wiki 68up2vwys0tbpc03d547349rfu3effo 2432985 2432984 2026-07-02T20:13:05Z Stashbot 7414 sbassett@deploy1003: sbassett: Continuing with deployment 2432985 wikitext text/x-wiki nka4a548wfl96crr8cekg3d2jpz9uf5 2432986 2432985 2026-07-02T20:14:00Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint2003.wikimedia.org with reason: host reimage 2432986 wikitext text/x-wiki 1w198g8h4wezetf6kmy3x8cni5w18jn 2432988 2432986 2026-07-02T20:17:28Z Stashbot 7414 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307196|mediawiki.action.edit.preview: Fix compat with `<button>`-buttons (T430956)]] (duration: 08m 13s) 2432988 wikitext text/x-wiki eima76aep7nh4fpoqw07ioh93ff3irl 2432990 2432988 2026-07-02T20:28:20Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1307218|Revert "Temporarily disable experimental ExtTagPFragment type" (T430344 T429624)]], [[gerrit:1307227|Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223|Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] 2432990 wikitext text/x-wiki 09ffy1vio5sq00uim3hb8bts0t3dees 2432991 2432990 2026-07-02T20:31:12Z Stashbot 7414 arlolra@deploy1003: cscott, arlolra: Backport for [[gerrit:1307218|Revert "Temporarily disable experimental ExtTagPFragment type" (T430344 T429624)]], [[gerrit:1307227|Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223|Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Cha 2432991 wikitext text/x-wiki lozqd7bxujr3xim0nj5rb2kemtg4bgb 2432992 2432991 2026-07-02T20:33:39Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint2003.wikimedia.org with OS trixie 2432992 wikitext text/x-wiki cmnymlwezz9iscigti51eq8qnuadyne 2432993 2432992 2026-07-02T20:35:32Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 2432993 wikitext text/x-wiki n0oxi3989jerege8eg7wyvetm7qzz4h 2432994 2432993 2026-07-02T20:35:40Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 (duration: 00m 20s) 2432994 wikitext text/x-wiki 4lxnrven3vd7kn84neul48zsakrw8en 2432995 2432994 2026-07-02T20:36:32Z Stashbot 7414 arlolra@deploy1003: cscott, arlolra: Continuing with deployment 2432995 wikitext text/x-wiki 98xd50no5jqguk39nri28g7gjd982pg 2432996 2432995 2026-07-02T20:40:55Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307218|Revert "Temporarily disable experimental ExtTagPFragment type" (T430344 T429624)]], [[gerrit:1307227|Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223|Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] (duration: 12m 35s) 2432996 wikitext text/x-wiki dla46xtqh05m05bf7ungktyf4zoh61i 2432997 2432996 2026-07-02T20:55:59Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003 2432997 wikitext text/x-wiki eqtgh99e5bjxnoyaoc9hk8qxeanp6h7 2432999 2432997 2026-07-02T20:58:26Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003 2432999 wikitext text/x-wiki 0tpgj5ow0k1gisxgmr30jmf2687e79v 2433009 2432999 2026-07-02T21:19:29Z Stashbot 7414 sbassett: Deployed security fix for T428829 2433009 wikitext text/x-wiki qzizitfdq5ycn255q2pua3e7ojz7x0b 2433010 2433009 2026-07-02T21:20:26Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430879, restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling source-only afterwards 2433010 wikitext text/x-wiki tmpl54sf64vsdnuluoooobt5khl1154 2433028 2433010 2026-07-02T21:39:17Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 2433028 wikitext text/x-wiki bdx81xeudzx2jr4hf08qhv00cna1mdc 2433029 2433028 2026-07-02T21:39:24Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 (duration: 00m 18s) 2433029 wikitext text/x-wiki 07v6ohv3npzjbrxrjmegml3hz7d58ko 2433030 2433029 2026-07-02T22:03:25Z Stashbot 7414 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on zuul.wikimedia.org with reason: reimage 2433030 wikitext text/x-wiki rynzor81agukolx9clyqlr83kdtx6ux 2433031 2433030 2026-07-02T22:03:36Z Stashbot 7414 mutante: contint1003 (zuul.wikimedia.org) - reimaging because of T430510#12067628 T418521 2433031 wikitext text/x-wiki 47vw82cnkctro5j9cf0c3qdoyfqkolj 2433032 2433031 2026-07-02T22:05:54Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint1003.wikimedia.org with OS trixie 2433032 wikitext text/x-wiki dn8uqaj80tdslck9n5u8t6000eoxx98 2433033 2433032 2026-07-02T22:23:52Z Stashbot 7414 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1003.wikimedia.org with reason: host reimage 2433033 wikitext text/x-wiki o950ufff8zfqi1x78jhakdr08ul15pg 2433034 2433033 2026-07-02T22:29:51Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1003.wikimedia.org with reason: host reimage 2433034 wikitext text/x-wiki 6ndhd66oglvqkteczpb8s6vh7nqp13x 2433036 2433034 2026-07-02T22:55:40Z Stashbot 7414 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1003.wikimedia.org with OS trixie 2433036 wikitext text/x-wiki o3dvta0qdqnm8qbnswr5f1ady944bis 2433039 2433036 2026-07-03T02:00:33Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2433039 wikitext text/x-wiki g4xgw41dpb4lfgrwk3yiau98t9xc55t 2433040 2433039 2026-07-03T02:07:51Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 18s) 2433040 wikitext text/x-wiki 2xb5215ro1yqwvtd49kjex3nbi3lt8r 2433041 2433040 2026-07-03T06:15:58Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2433041 wikitext text/x-wiki ievvxnf6lndqvnjkakseqb6vp2gv4wg 2433044 2433041 2026-07-03T08:11:54Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts mirror1001.wikimedia.org 2433044 wikitext text/x-wiki b0p61udzignj11e6tuzs0m1in7s0yk2 2433045 2433044 2026-07-03T08:18:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.netbox 2433045 wikitext text/x-wiki 9yzp36de9iaan1t79lfxta49fr9f687 2433047 2433045 2026-07-03T08:29:30Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" 2433047 wikitext text/x-wiki m4rcho3aybg363xeqgk95c3dzk3afwv 2433048 2433047 2026-07-03T08:31:53Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" 2433048 wikitext text/x-wiki 1d2143tcacth1ss1dzs75auc8mig81q 2433049 2433048 2026-07-03T08:31:54Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2433049 wikitext text/x-wiki 1cjiy5rxzw4w1bclueq8lg7e5encwvg 2433050 2433049 2026-07-03T08:31:55Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mirror1001.wikimedia.org 2433050 wikitext text/x-wiki m0rji1imze60n6ym9xic78rljsvjvo9 2433051 2433050 2026-07-03T08:49:05Z Stashbot 7414 atsukoito: depooling cirrussearch in codfw because of regression after upgrade T431091 2433051 wikitext text/x-wiki rrefxhn75jrqmna1u3e6gxy1dk8zne2 2433052 2433051 2026-07-03T08:49:51Z Stashbot 7414 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw 2433052 wikitext text/x-wiki ov0ow8spis7eb3tclbed26cee9aac2b 2433053 2433052 2026-07-03T08:50:00Z Stashbot 7414 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw 2433053 wikitext text/x-wiki peb6g67ml6bi788iucnqkxmktpzcfpz 2433054 2433053 2026-07-03T08:50:08Z Stashbot 7414 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw 2433054 wikitext text/x-wiki 3b6fe6x7g638p1vji8vsque7k4by65q 2433055 2433054 2026-07-03T08:55:44Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2433055 wikitext text/x-wiki n02k4a51gfh59xz6g9lk3lrul6i8z1b 2433056 2433055 2026-07-03T08:57:20Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2433056 wikitext text/x-wiki jiqyeeje8amqyt0z22ua2ufso24ofxl 2433057 2433056 2026-07-03T08:59:12Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2433057 wikitext text/x-wiki tdw2d6fuxwdjwosfy4p1pkza557o1pm 2433058 2433057 2026-07-03T09:00:50Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2433058 wikitext text/x-wiki cx9j6548bgczbysnr8vdyw2mqt161qd 2433059 2433058 2026-07-03T09:04:06Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2433059 wikitext text/x-wiki j4zn0timqhz8k4s3atdy56v7ixnc4mf 2433060 2433059 2026-07-03T09:05:03Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2433060 wikitext text/x-wiki a9tm4a7rflycbehpkwkw5sm7ccqzr1p 2433063 2433060 2026-07-03T09:16:54Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet 2433063 wikitext text/x-wiki lfv96t02pkz6nh1hp7rdvx6ee6l7e88 2433064 2433063 2026-07-03T09:17:28Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2433064 wikitext text/x-wiki 5hh5f77120y5foveylhb5djc8en5dbb 2433066 2433064 2026-07-03T09:25:10Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet 2433066 wikitext text/x-wiki 22t08161rukrqst1m0e9cansav6yq5f 2433067 2433066 2026-07-03T09:36:30Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2433067 wikitext text/x-wiki 42as9irogvo2y72v18xkrlnxzokqxpf 2433068 2433067 2026-07-03T09:36:48Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts backup[1004-1007].eqiad.wmnet 2433068 wikitext text/x-wiki eyemv0a8blnyj1gama8qk2g376h74w7 2433069 2433068 2026-07-03T09:39:11Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2433069 wikitext text/x-wiki eb6bwlwn55d064pipzd54690e3c1491 2433070 2433069 2026-07-03T09:52:03Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.dns.netbox 2433070 wikitext text/x-wiki 50anzo8wzfyt056b06rcaly3wfs76ze 2433072 2433070 2026-07-03T10:01:42Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003" 2433072 wikitext text/x-wiki hel1semhhkgpdndkpnya3vcy0cxfbwe 2433073 2433072 2026-07-03T10:02:01Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003" 2433073 wikitext text/x-wiki 107mn6zk8e7x4w8k6ff2sx4dwmpms9u 2433074 2433073 2026-07-03T10:02:02Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2433074 wikitext text/x-wiki r4gl2ue6xw1wv8e87d5il0sd7qed0bp 2433075 2433074 2026-07-03T10:02:03Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[1004-1007].eqiad.wmnet 2433075 wikitext text/x-wiki h3zuudddocupebz0i568ypx4si0w8lu 2433077 2433075 2026-07-03T10:19:20Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply 2433077 wikitext text/x-wiki 9599limfxhuo6nznbg48z0qt9oemqqh 2433079 2433077 2026-07-03T10:40:19Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply 2433079 wikitext text/x-wiki twcdlr2s30ik1cyf54h8g656s3zixpb 2433083 2433079 2026-07-03T11:58:56Z Stashbot 7414 jynus@cumin2003: START - Cookbook sre.hosts.decommission for hosts backup[2004-2007].codfw.wmnet 2433083 wikitext text/x-wiki fzkk6dsbp4ifqxf0s8l57gk1iomnwpk 498 16087 2432908 2315971 2026-07-02T16:46:32Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) 2432908 wikitext text/x-wiki 16zw390u9wl1tp0jthjv5j5289fo5g0 2432909 2432908 2026-07-02T16:46:39Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) 2432909 wikitext text/x-wiki t3iuuzz3sn4qjpqmeoyt28lsvqcid9k 0 17290 2432789 2432750 2026-07-02T13:27:21Z Stashbot 7414 James_F: Docker: [composer-scratch] Upgrade composer to 2.10.2 and cascade, for T428570 2432789 wikitext text/x-wiki 48bgog66t38sb2b33jpv42l5oa5ws6i 2433071 2432789 2026-07-03T09:53:17Z Stashbot 7414 James_F: Docker: [quibble-coverage] Update phpunit-patch-coverage to 0.0.18, for T423987 and T425807 2433071 wikitext text/x-wiki hhogb4dnphe2x1go4hti6ba4eqvrprr 2433076 2433071 2026-07-03T10:06:25Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches 2433076 wikitext text/x-wiki gv90lka4rtc8s9v028d3kjq8qy4j2wg 2433080 2433076 2026-07-03T11:22:07Z Stashbot 7414 James_F: Zuul: Make the in-mediawiki-tarball template real 2433080 wikitext text/x-wiki o7zzpp7qqwqyh0s905rj79gjqubyw2g 498 55577 2432906 2431412 2026-07-02T16:46:24Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) 2432906 wikitext text/x-wiki 12qvlhc8c29256jr21k961o154luz3c 2432907 2432906 2026-07-02T16:46:29Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) 2432907 wikitext text/x-wiki q8sclkoh1k0h0nap7126fmimlo38vzv 0 324433 2433000 2402150 2026-07-02T21:02:00Z BCornwall-WMF 31521 rm duplicate mariadb heading 2433000 wikitext text/x-wiki 11u5q86j4xis83a07nkwqeyl48fn8vd 2433002 2433000 2026-07-02T21:04:26Z BCornwall-WMF 31521 /* HAProxy for edge caching */ Remove extraneous info, we have that in [[CDN/History]] already 2433002 wikitext text/x-wiki 721b3ku0bgislwbm16ipq116elfeirz 2433004 2433002 2026-07-02T21:08:43Z BCornwall-WMF 31521 /* HAProxy for MariaDB */ use see template 2433004 wikitext text/x-wiki kn2qar46j73cner182ysa52rhiasf69 2433005 2433004 2026-07-02T21:09:08Z BCornwall-WMF 31521 /* HAProxy for MariaDB */ "see" -> "Main article" 2433005 wikitext text/x-wiki ovo7ielfxqbs7jsziorntncchcaywil 0 441820 2433003 2414738 2026-07-02T21:07:54Z BKing (WMF) 29017 /* Manual Steps */ 2433003 wikitext text/x-wiki 9pinv8mo2ymwjuwkyyi3in1yfigmacw 0 443291 2432968 2387397 2026-07-02T18:37:11Z SNwachukwu (WMF) 39521 Update the process for adding new wikis to indicate that it is now automated describing the process 2432968 wikitext text/x-wiki 0elt05xj0arnwmjni0mofpkpno6lca0 498 443628 2432982 2432547 2026-07-02T20:09:07Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28618176241 (https://github.com/cluebotng/component-configs/commits/c3b2ebd135508a9a0f781ed332514069fb4d8c89) 2432982 wikitext text/x-wiki 2n1vb03jveqtkgq4rbde1h7h9piej60 2433019 2432982 2026-07-02T21:26:13Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412770 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433019 wikitext text/x-wiki mw653958zy40i1p1rqwy39gky9b7gcd 2433035 2433019 2026-07-02T22:32:14Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28625556671 (https://github.com/cluebotng/component-configs/commits/e95434d48b96490f3526640acde23ce59116589b) 2433035 wikitext text/x-wiki qsxvamwuqkmb3mwzxkx30xynawco6hz 2433061 2433035 2026-07-03T09:05:27Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650009604 (https://github.com/cluebotng/component-configs/commits/a66ec11b207598051416135000b2a5815ba30080) 2433061 wikitext text/x-wiki mf3yn7hto642xb1smabkrsp0y371v8m 2433062 2433061 2026-07-03T09:14:43Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650658730 (https://github.com/cluebotng/component-configs/commits/0d3a15b63afd3afe5965be4754401712df0b4ae1) 2433062 wikitext text/x-wiki 2y7g82w0gtup51noq8prn3x9k44f72l 2433065 2433062 2026-07-03T09:25:05Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28651139847 (https://github.com/cluebotng/component-configs/commits/11ad538e5876626229416e2ba16bc9f7a7e5cace) 2433065 wikitext text/x-wiki q2512e90ie4zd57caz05pu4dyw7bwdj 498 443947 2432975 2430712 2026-07-02T19:33:58Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed bb43718fe8 (l10n updates: kw) 2432975 wikitext text/x-wiki 4texo34hx12yfi1utbkwnzhfibxqpv5 498 446211 2433015 2432378 2026-07-02T21:25:16Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412758 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433015 wikitext text/x-wiki 9d8gxi1rzapqe3skbpjf3tcoaiqea93 0 449160 2432807 2432726 2026-07-02T13:50:10Z Dexbot 30554 Bot: Updating the report 2432807 wikitext text/x-wiki 43q0l21xym5q7dd3w9xohjs5unf2ypx 2432828 2432807 2026-07-02T14:11:14Z Dexbot 30554 Bot: Updating the report 2432828 wikitext text/x-wiki e7r90hxyzotp8vckw7rh3p110xugctk 2433038 2432828 2026-07-03T00:00:18Z Dexbot 30554 Bot: Updating the report 2433038 wikitext text/x-wiki 2ja7b45pp8913haji4q8l94vhh5u7r4 0 451180 2433006 2426598 2026-07-02T21:10:05Z BCornwall-WMF 31521 ce: bold definition of cp 2433006 wikitext text/x-wiki gncj1scmj5k6z3vzrx63tvkr773n9ja 2433007 2433006 2026-07-02T21:10:37Z BCornwall-WMF 31521 varnish also rate limits 2433007 wikitext text/x-wiki ea88u8dk6o966y5irndh9xltb79ewwb 2433008 2433007 2026-07-02T21:13:56Z BCornwall-WMF 31521 /* Caching */ update link to historical caching cluster 2433008 wikitext text/x-wiki ibk8h3bky3uf7z3et7r8683m3jae1c5 0 452637 2432929 2357092 2026-07-02T17:00:43Z WBrown (WMF) 37696 + 2432929 wikitext text/x-wiki 6brk0qtvsuf2237a6v86nq3xjqm8tpn 2432930 2432929 2026-07-02T17:00:54Z WBrown (WMF) 37696 remove 2432930 wikitext text/x-wiki bc5yi0asylyudtqew0t4jyi3tq8zuda 2432931 2432930 2026-07-02T17:02:05Z WBrown (WMF) 37696 + 2432931 wikitext text/x-wiki fwe3jboypbh5gw73jb1pptveqrofb2l 2432932 2432931 2026-07-02T17:02:50Z WBrown (WMF) 37696 + 2432932 wikitext text/x-wiki 2jluk5qc01dwm5ssxkhpzo5vxjv2ehg 2432936 2432932 2026-07-02T17:03:47Z WBrown (WMF) 37696 mark as decomissioned in the infobox 2432936 wikitext text/x-wiki 3wxv0sym7u1zr07vcplpe4pcw7x6j2c 2432937 2432936 2026-07-02T17:03:59Z WBrown (WMF) 37696 fix 2432937 wikitext text/x-wiki cocp4mfog09tpwu1ogm53gxespcoizu 498 452890 2433014 2432548 2026-07-02T21:23:03Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433014 wikitext text/x-wiki tap1rjqzphu1lt9bwy5symq4saoe28w 2433027 2433014 2026-07-02T21:34:08Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433027 wikitext text/x-wiki 45vrnpyqxlupahswzrsy580rintzse2 116 453906 2432867 2432454 2026-07-02T15:48:28Z Gitlabaccountapprovalbot 37332 @tekneos was approved. 2432867 wikitext text/x-wiki 6k6vt5ij0w38ne3z9oghbf4dz71c245 0 455726 2432964 2432635 2026-07-02T18:08:11Z MMuhlenhoff (WMF) 5268 /* Bots still using the legacy setup */ Link for krdbot 2432964 wikitext text/x-wiki 4l6sw4pxmz5y56jel3fdnjalbg4a3xp 2432965 2432964 2026-07-02T18:09:01Z MMuhlenhoff (WMF) 5268 /* Bots still using the legacy setup */ Link for Huggle 2432965 wikitext text/x-wiki ktoftus6sg1dc5nasm6cix2nvtzp8ye 0 456356 2433078 2430099 2026-07-03T10:22:10Z Phuedx (WMF) 47850 Update details about GrowthBook 2433078 wikitext text/x-wiki 85gfym16d6ekz4xdvvt705vurl8linq 498 458787 2433017 2432380 2026-07-02T21:25:33Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412753 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433017 wikitext text/x-wiki 5tjmoeqx9pcaj6cgyr1yop7562b1e1f 498 458879 2432903 2337075 2026-07-02T16:45:59Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) 2432903 wikitext text/x-wiki dk30rsoddivabd5qdw6p53hple99s9m 2432904 2432903 2026-07-02T16:46:05Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) 2432904 wikitext text/x-wiki n4tnjpa0pu4qstl7s6vncwow59v9rpw 498 458884 2432901 2372223 2026-07-02T16:45:48Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) 2432901 wikitext text/x-wiki sqrrkl5x737u0prbzpg1yeb8a3u74cx 2432902 2432901 2026-07-02T16:45:54Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) 2432902 wikitext text/x-wiki 3mrbnqr2sb3amjth1tqme5pdha40x01 0 459160 2432823 2418588 2026-07-02T14:08:33Z DAdedoyin (WMF) 39150 /* Pause a Donation */ 2432823 wikitext text/x-wiki btysmevspunluwiivoe6a1fds37bynz 2432829 2432823 2026-07-02T14:11:14Z DAdedoyin (WMF) 39150 /* Paused */ 2432829 wikitext text/x-wiki aoyy34palut85iwb4xjsphfzjl4bcov 498 459243 2433012 2432445 2026-07-02T21:23:00Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433012 wikitext text/x-wiki 980q6uiyim5nnk5qnwnjg68st8ipdd8 2433020 2433012 2026-07-02T21:27:55Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433020 wikitext text/x-wiki asgm077t66nmu2yvhika159eoza6ox7 2433025 2433020 2026-07-02T21:32:30Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433025 wikitext text/x-wiki 20nqvu2rhbjyd70ed4ix2trctjppxn4 498 459244 2433018 2432381 2026-07-02T21:25:50Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412786 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433018 wikitext text/x-wiki ccmsflbybyu5z5qdi96b3j8gw2n46nn 498 459255 2433016 2432379 2026-07-02T21:25:17Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412759 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433016 wikitext text/x-wiki 74qzpbm3v91m4sqww458l3bkzo7mj9x 498 459263 2433013 2432371 2026-07-02T21:23:02Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433013 wikitext text/x-wiki otiqzuocjdlfmetjf1utp1bb84zfz02 2433022 2433013 2026-07-02T21:29:29Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) 2433022 wikitext text/x-wiki 2ydjxsj59qpcqbjatf29znjkw8l6hju 0 459290 2433042 2432206 2026-07-03T07:20:42Z KCVelaga (WMF) 38150 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2433042 wikitext text/x-wiki meia02rl90sa634psc9ndhexokcy8ci 2433043 2433042 2026-07-03T07:20:59Z KCVelaga (WMF) 38150 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2433043 wikitext text/x-wiki gph1bh8ujxgkvj1gpe8k4cp1bjb74pu 0 459699 2433037 2432599 2026-07-02T23:46:16Z Stashbot 7414 eileen: civicrm upgraded from d44caa2a to 274308a4 2433037 wikitext text/x-wiki sl7w47v9fjdjhxgtpk0g494alnq1uy6 0 460059 2433046 2420593 2026-07-03T08:27:09Z JMonton-WMF 47887 how to do a manual backfill 2433046 wikitext text/x-wiki pta86evmehxp41yi4sxg1sr9yx81kem 498 460126 2432857 2431508 2026-07-02T15:28:27Z Stashbot 7414 wmbot~gergesshamon@tools-bastion-15: [DEPLOY] Starting deployment | ref=v1.5.7 (for fix v1.5.6) 2432857 wikitext text/x-wiki fs75f4zoilw1lwgq0jdern1f2ocdtmu 2432858 2432857 2026-07-02T15:29:40Z Stashbot 7414 wmbot~gergesshamon@tools-bastion-15: [DEPLOY] Build triggered successfully | ref=v1.5.7 (for fix v1.5.6) 2432858 wikitext text/x-wiki t0nr852hb9hrp9obkzt3acfwpk9o76r 2432859 2432858 2026-07-02T15:29:40Z Stashbot 7414 wmbot~gergesshamon@tools-bastion-15: [DEPLOY] Restarting buildservice | ref=v1.5.7 (for fix v1.5.6) 2432859 wikitext text/x-wiki hjye90xqqqw8u79hyjqwgrvbuhic4b5 2432860 2432859 2026-07-02T15:29:52Z Stashbot 7414 wmbot~gergesshamon@tools-bastion-15: [DEPLOY] Deployment completed successfully | ref=v1.5.7 (for fix v1.5.6) 2432860 wikitext text/x-wiki mgq8s47li1av83z69jjnifkt9dd2ue9 0 460231 2432832 2431534 2026-07-02T14:20:46Z BKing (WMF) 29017 /* Steps */ 2432832 wikitext text/x-wiki 367ac02agdfd09l1bpakj8bqgog881q 2432912 2432832 2026-07-02T16:53:18Z BKing (WMF) 29017 /* Cluster Quorum Issues */ 2432912 wikitext text/x-wiki l12hhtszy5xa88gfk7hn9ns3jreu58u 0 460233 2432971 2429636 2026-07-02T18:57:48Z MPopov (WMF) 5891 /* Guidelines and recommendations */ Note about group-agnostic definitions 2432971 wikitext text/x-wiki 7s6v3xv4bzvo8yc0ya9f46ukqi1ri6l 0 460277 2433001 2422746 2026-07-02T21:04:00Z BCornwall-WMF 31521 /* 2023 */ mention port 80 2433001 wikitext text/x-wiki 84z4fy9tct6njbczm1ajpnyuvhjykhh 0 460318 2432767 2430103 2026-07-02T12:01:59Z Atsuko-WMF 54082 /* Verification */ from console 2432767 wikitext text/x-wiki mu5741s641a53myzpgz1qozmycu7g0n 0 460375 2432944 2431227 2026-07-02T17:11:40Z AWesterinen-WMF 54189 2432944 wikitext text/x-wiki l918g3k4w77wszyr5ajqmrkmqmtci1k 13 460406 2432958 2026-07-02T17:48:06Z Mathglot 42458 /* No error-reporting link on Cloud services error page */ new section 2432958 wikitext text/x-wiki n5boev56r3qu1rx8obxj7g90gne3zxy 1 460407 2432974 2026-07-02T19:32:47Z BCornwall-WMF 31521 /* Proposal for deletion */ new section 2432974 wikitext text/x-wiki c52i90x0is3z54pi2fcs84qiq87vd1x 116 460408 2432987 2026-07-02T20:15:27Z Ancient9983 52221 Create a basic starter page for the WikiData Authority Control Tool 2432987 wikitext text/x-wiki s7mrbxwq7jumbtsqnn2niigknt8s2m5 2432989 2432987 2026-07-02T20:20:41Z Ancient9983 52221 Add the start of a Tool template for the tool 2432989 wikitext text/x-wiki dijwjbtd0w61rj1e60ez6306l4n5oja 2433011 2432989 2026-07-02T21:22:31Z Ancient9983 52221 Update Tool template with relevant details. 2433011 wikitext text/x-wiki 2tgtm285g615n9zwm2ayzvp6r0v9776 2433021 2433011 2026-07-02T21:28:35Z Ancient9983 52221 Fill out prerequisites section 2433021 wikitext text/x-wiki snjolq4hasnm4q1yb31ujcetkvg1d0s 2433023 2433021 2026-07-02T21:30:07Z Ancient9983 52221 /* How to get help and report issues */ 2433023 wikitext text/x-wiki q6ltl9gm0x24hrnavahdta2b3k40r1k 2433024 2433023 2026-07-02T21:31:39Z Ancient9983 52221 /* Developer documentation */ 2433024 wikitext text/x-wiki djhsfwklb82z2li90lyjyhkfr09oh1j 2433026 2433024 2026-07-02T21:32:43Z Ancient9983 52221 2433026 wikitext text/x-wiki fzm3drkeu8savdfg7nl1v58uvgccrvs 1 460409 2433081 2026-07-03T11:52:51Z Peter F. Patel-Schneider 42591 /* incorrect and inefficient replacement for label service */ new section 2433081 wikitext text/x-wiki bqmcz73cuyf03xjrfapmmqzrecv7jfd 2433082 2433081 2026-07-03T11:56:18Z Peter F. Patel-Schneider 42591 /* burying the lede */ new section 2433082 wikitext text/x-wiki n3m5j1kv462399q75d5bvl02nf2bb2i