Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.8 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 420 2430945 2357389 2026-06-26T02:26:25Z Krinkle 44 [[Nova Resource:Deployment-prep/HetDeploy]] was consolidated into [[Nova Resource:Deployment-prep/How code is updated|How code is updated]]. Remove old Jenkins link, no longer used. 2430945 wikitext text/x-wiki s1da3b1mxydksmpsxm6i1hkan8t05yg 110 573 2430938 2357140 2026-06-26T02:19:49Z Krinkle 44 2430938 wikitext text/x-wiki d719rmx93vnyut77xl0m7l2qeqe65se 2430939 2430938 2026-06-26T02:20:03Z Krinkle 44 remove old link to Jenkins 2430939 wikitext text/x-wiki 7x6md6ucc5vg9phvvo4vm15jkof7e1f 2430944 2430939 2026-06-26T02:25:08Z Krinkle 44 Redirected page to [[Nova Resource:Deployment-prep]] 2430944 wikitext text/x-wiki pw0h08ttlqyw0vgolze86v16ksxi2so 498 798 2430933 2206087 2026-06-26T02:07:26Z Krinkle 44 /* Automatically updated */ update old Jenkins description with current cronjob per [[phab:T256168]] 2430933 wikitext text/x-wiki e9r8lphuow9hddv5h5s5v2om7kmnpj9 2430937 2430933 2026-06-26T02:19:33Z Krinkle 44 Redirected page to [[Nova Resource:Deployment-prep/How code is updated#MediaWiki]] 2430937 wikitext text/x-wiki dmk0w5whylqxvnjfjmzottq2ac404wx 498 1436 2430940 2371773 2026-06-26T02:21:23Z Krinkle 44 2430940 wikitext text/x-wiki tazkyiv6qf546413rgvorrx85ppc8ed 2430941 2430940 2026-06-26T02:21:48Z Krinkle 44 /* File structure */ 2430941 wikitext text/x-wiki lmghtq1a98jmaf7i7vyzaj60czzeykf 2430942 2430941 2026-06-26T02:22:32Z Krinkle 44 /* MediaWiki config */ 2430942 wikitext text/x-wiki 0q2olnf8k88tucb1yi9tcoxx6wmpgcb 2430943 2430942 2026-06-26T02:23:53Z Krinkle 44 /* New extension */ 2430943 wikitext text/x-wiki s64ujvrqm4132j5xfhl2kyxbumff24m 498 3086 2430792 2430553 2026-06-25T17:00:15Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2430792 wikitext text/x-wiki q3vw60khgfaxfuypmdi2u4lu4oy55zm 2430798 2430792 2026-06-25T17:08:16Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2430798 wikitext text/x-wiki j4k6i3wdmbprtjlc8xk3dnhnfphpbug 0 4039 2430831 2394759 2026-06-25T18:07:32Z Reedy 9 /* Deployment: Manual */ double spaces 2430831 wikitext text/x-wiki 432inkov94d9ehpyc4ondw5lgop7wox 2430835 2430831 2026-06-25T18:20:13Z Reedy 9 /* Security mitigations */ logloglog 2430835 wikitext text/x-wiki 0wyade4z6v8pdvrqx7yq6dyqsmgjkb3 0 4108 2430643 2430457 2026-06-25T12:59:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1305476]] to Thursday, June 25 UTC afternoon backport window 2430643 wikitext text/x-wiki epnvxaxuzft2jkn2w52w5vigp2fqyss 2430708 2430643 2026-06-25T14:43:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1304888]] to Thursday, June 25 UTC late backport window 2430708 wikitext text/x-wiki bjza4rki75j9zbes85brtru3sda0g0a 2430749 2430708 2026-06-25T16:21:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1305711]] to Thursday, June 25 UTC late backport window 2430749 wikitext text/x-wiki m7ys14pbv6g0ak4kujvlmcbdf3691f9 2430753 2430749 2026-06-25T16:24:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1305720]] to Thursday, June 25 UTC late backport window 2430753 wikitext text/x-wiki fghbn1075033vruslnz2l3s10nam7b5 2430755 2430753 2026-06-25T16:26:37Z ScheduleDeploymentBot 37566 Add [[gerrit:1305719]] to Thursday, June 25 UTC late backport window 2430755 wikitext text/x-wiki 80cbh8p73jrz8169jzpii8j2u5rpo51 2430793 2430755 2026-06-25T17:02:02Z ScheduleDeploymentBot 37566 Add [[gerrit:1305724]] to Thursday, June 25 UTC late backport window 2430793 wikitext text/x-wiki o5asjyfzth0ed6c4nyra7wugrknip8t 2430825 2430793 2026-06-25T17:50:53Z ScheduleDeploymentBot 37566 Add [[gerrit:1304706]] to Thursday, June 25 UTC late backport window 2430825 wikitext text/x-wiki ei1i43cy465w0sxbxf6v0pwcijlih0s 2430843 2430825 2026-06-25T18:33:04Z Cscott 41391 /* {{Deployment_day|date=2026-06-25}} */ Deployed the Parsoid backport already 2430843 wikitext text/x-wiki 3j6f541k5fm45smcnq8yyg0e9aevdx3 2430858 2430843 2026-06-25T19:00:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1305733]] to Thursday, June 25 UTC late backport window 2430858 wikitext text/x-wiki dplwr7re7ewn2t8g1hc4py0aj3enluz 2430886 2430858 2026-06-25T19:18:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1305639]] to Thursday, June 25 UTC late backport window 2430886 wikitext text/x-wiki 9f7mbb6np2u84x9jlwrcgg3swg77sfi 2430897 2430886 2026-06-25T20:02:35Z Cscott 41391 /* {{Deployment_day|date=2026-06-25}} */ Remove two already-deployed patches 2430897 wikitext text/x-wiki 384ru716nupdye3fn9xdiy65lns2iiu 2430956 2430897 2026-06-26T05:39:54Z ScheduleDeploymentBot 37566 Add [[gerrit:1305811]] to Monday, June 29 UTC afternoon backport window 2430956 wikitext text/x-wiki pgwcxsq7mnzjddeeu5v9qn6nqqhhcrq 0 4118 2430683 2429666 2026-06-25T13:57:13Z WBrown (WMF) 37696 /* Clean up after deletion */ + 2430683 wikitext text/x-wiki 8j3szzsgizijqllzsz0qp7zaesrjo63 498 5034 2430935 2334384 2026-06-26T02:15:40Z Krinkle 44 Update old description of Jenkins with current cronjob as of May 2026, per [[phab:T256168]] 2430935 wikitext text/x-wiki 3jt7l9gj7ib46p52cjx169n763829ss 2430936 2430935 2026-06-26T02:19:17Z Krinkle 44 Moved overview here from [[Nova Resource:Deployment-prep/HetDeploy]] 2430936 wikitext text/x-wiki 8tath4rwuhddhlzpofsmmxn0gp52lic 2430947 2430936 2026-06-26T02:28:28Z Krinkle 44 2430947 wikitext text/x-wiki 1szzbeo26qf6c4a8pyc64v73cmi31q3 2430948 2430947 2026-06-26T02:29:20Z Krinkle 44 2430948 wikitext text/x-wiki 1wkg175686v7ypy5zufqoqmpg2ioz63 2430949 2430948 2026-06-26T02:29:50Z Krinkle 44 2430949 wikitext text/x-wiki 64pif4ahljflqq3x9gltr9p4j5t2yy8 2430950 2430949 2026-06-26T02:35:34Z Krinkle 44 2430950 wikitext text/x-wiki b61r71njd024tjuomq1vxz2dfow3yhd 498 6726 2430756 2430356 2026-06-25T16:28:22Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2430756 wikitext text/x-wiki ef2wz9bgyqdnigucoq37abfcmqhlph1 2430757 2430756 2026-06-25T16:28:29Z Stashbot 7414 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli 2430757 wikitext text/x-wiki iepc8lmzc2zx9w42docb9zhystik3ob 2430761 2430757 2026-06-25T16:29:01Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2430761 wikitext text/x-wiki 8o838i5dicu52arywjfctbsx56scl0n 2430767 2430761 2026-06-25T16:37:17Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2430767 wikitext text/x-wiki td5cegpn5m16xuz1u6e10qq6pdgp7ln 0 7919 2430635 2430634 2026-06-25T12:04:36Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A7 2430635 wikitext text/x-wiki aligo52jwviscuyxxtzhknd681wdokb 2430636 2430635 2026-06-25T12:06:26Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 31 hosts with reason: Rack A7 depool 2430636 wikitext text/x-wiki k3n4w47531uiru55dov8reywfemyq6t 2430637 2430636 2026-06-25T12:33:08Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1243.eqiad.wmnet with reason: host reimage 2430637 wikitext text/x-wiki lwuj41qmbd0mw4l28w1hj13i93aip2l 2430638 2430637 2026-06-25T12:35:02Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2237.codfw.wmnet with reason: host reimage 2430638 wikitext text/x-wiki fib4htb66za65bbhr9jv0iikze2zxed 2430639 2430638 2026-06-25T12:38:58Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1243.eqiad.wmnet with reason: host reimage 2430639 wikitext text/x-wiki 5l4c0hpdsnhhyfdjqqwkea9hxj4j1g2 2430640 2430639 2026-06-25T12:43:01Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2237.codfw.wmnet with reason: host reimage 2430640 wikitext text/x-wiki 1tp92cmajirbl5x7207lpxcocp56aui 2430641 2430640 2026-06-25T12:44:34Z Stashbot 7414 XioNoX: lsw1-a7-codfw> request system reboot - T429817 2430641 wikitext text/x-wiki ohu94cq6s6x4omu9j67v1niqyuo1oz1 2430642 2430641 2026-06-25T12:56:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1243.eqiad.wmnet with OS trixie 2430642 wikitext text/x-wiki n05w8hp92lgkudd6drwh6cdie3ebh8i 2430644 2430642 2026-06-25T12:59:40Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2237.codfw.wmnet with OS trixie 2430644 wikitext text/x-wiki 490ivv1fe6f7ct172v4wge38zjcprv6 2430645 2430644 2026-06-25T13:00:17Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1077.eqiad.wmnet with OS trixie 2430645 wikitext text/x-wiki pu8bg8h3wougjd0md3qiai68t0aw77r 2430646 2430645 2026-06-25T13:02:35Z Stashbot 7414 moritzm: installing glib2.0 security updates 2430646 wikitext text/x-wiki c1kvii8wrwz4tkgt4m2wippf2lcnlng 2430647 2430646 2026-06-25T13:05:28Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.ganeti.addnode for new host ganeti2028.codfw.wmnet to cluster codfw and group A 2430647 wikitext text/x-wiki h73rqzdycwfs45f33vxhdiw8do66e4n 2430648 2430647 2026-06-25T13:06:20Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1305476|isvwiki: set timezone, sitename and logos (T429935)]], [[gerrit:1248909|Use Hadoop for Mostcategories on commonswiki (T413362)]] 2430648 wikitext text/x-wiki nyycjs3k7r5m273x9qbnutvymnlnogi 2430649 2430648 2026-06-25T13:06:26Z Stashbot 7414 moritzm: re-added ganeti2028 to codfw/A Ganeti cluster T429817 2430649 wikitext text/x-wiki mvn6sktxlzufh36iid2d0p5ef4qq6kk 2430650 2430649 2026-06-25T13:07:59Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2258-2259].codfw.wmnet 2430650 wikitext text/x-wiki deaosaw889igr6031puiu6kgar81vil 2430651 2430650 2026-06-25T13:08:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2258-2259].codfw.wmnet 2430651 wikitext text/x-wiki 1owrmjpaqudjeszm7kbogappfprixfn 2430652 2430651 2026-06-25T13:08:24Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet,dse-k8s-wdqs-test2001.codfw.wmnet 2430652 wikitext text/x-wiki fi2jybmlsyol1602re2m4rburylnzym 2430653 2430652 2026-06-25T13:08:29Z Stashbot 7414 zabe@deploy1003: zabe, anzx: Backport for [[gerrit:1305476|isvwiki: set timezone, sitename and logos (T429935)]], [[gerrit:1248909|Use Hadoop for Mostcategories on commonswiki (T413362)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430653 wikitext text/x-wiki 68ijgx59cpzyvpmfmowzh5u05bxj6nu 2430654 2430653 2026-06-25T13:08:31Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet,dse-k8s-wdqs-test2001.codfw.wmnet 2430654 wikitext text/x-wiki r3f0j43xs6dxpc7zd685a2t76fvyla9 2430655 2430654 2026-06-25T13:09:17Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2225: rack depool 2430655 wikitext text/x-wiki clkfq98qvnr39qmioi51ok5v62frn44 2430656 2430655 2026-06-25T13:09:26Z Stashbot 7414 zabe@deploy1003: zabe, anzx: Continuing with deployment 2430656 wikitext text/x-wiki 8a87y8eer5d81q7qfq8n6gux3cogngb 2430657 2430656 2026-06-25T13:09:46Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2224: rack depool 2430657 wikitext text/x-wiki gr7wtexz1ur9x6vrtdmnmqb21z0fdac 2430658 2430657 2026-06-25T13:11:34Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1243: Migration of db1243.eqiad.wmnet completed 2430658 wikitext text/x-wiki oxy27bvqpvrbpqrp1hdlb79v608dslg 2430659 2430658 2026-06-25T13:12:06Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1077.eqiad.wmnet with reason: host reimage 2430659 wikitext text/x-wiki 7uu0hqdfadt6y40nlcb3lgcits5gy71 2430660 2430659 2026-06-25T13:13:38Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2430660 wikitext text/x-wiki osto3o5xbkbobxc3kuj3cql7enrqwhb 2430661 2430660 2026-06-25T13:13:51Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305476|isvwiki: set timezone, sitename and logos (T429935)]], [[gerrit:1248909|Use Hadoop for Mostcategories on commonswiki (T413362)]] (duration: 07m 30s) 2430661 wikitext text/x-wiki q3xueeukneb531mfid8wskkc956aob3 2430662 2430661 2026-06-25T13:14:05Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2430662 wikitext text/x-wiki paw28zbibu9m7dka3cfzfsyjajv6d3k 2430663 2430662 2026-06-25T13:14:08Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2430663 wikitext text/x-wiki 8po8tvse3v0pv88sit6mu6k1dmp22vh 2430664 2430663 2026-06-25T13:14:35Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2430664 wikitext text/x-wiki 7lsf1ldcznfb5xr7wsya366ea89t0lw 2430665 2430664 2026-06-25T13:15:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2237: Migration of db2237.codfw.wmnet completed 2430665 wikitext text/x-wiki 0kg7d06k1v1zk6nmpa30isgohsmchha 2430666 2430665 2026-06-25T13:16:25Z Stashbot 7414 zabe: zabe@deploy1003:~$ mwscript namespaceDupes.php isvwiki --fix # T429935 2430666 wikitext text/x-wiki e0j21rpbjsfgh7u2u110r0ljjxyxzp3 2430667 2430666 2026-06-25T13:16:53Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2430667 wikitext text/x-wiki aibsch6mncj1aw3jvwrf9tx4fu2dch7 2430668 2430667 2026-06-25T13:16:58Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2430668 wikitext text/x-wiki mlnfwm7m1anw9q5wswp66200xs5k0dt 2430669 2430668 2026-06-25T13:19:01Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1077.eqiad.wmnet with reason: host reimage 2430669 wikitext text/x-wiki mqw2m1mkqsnzndyywyhzshlrin5ffvp 2430670 2430669 2026-06-25T13:21:02Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2028.codfw.wmnet to cluster codfw and group A 2430670 wikitext text/x-wiki 89pk0hd8bzgqrvx78vtmwcytp49beqn 2430671 2430670 2026-06-25T13:28:35Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2430671 wikitext text/x-wiki 3rwp18z9tu5a12fapp5qjxdxhwyc5cg 2430672 2430671 2026-06-25T13:28:37Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2430672 wikitext text/x-wiki niw7omp934dlmndsfiutx2aoa5hqz6g 2430673 2430672 2026-06-25T13:28:40Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2430673 wikitext text/x-wiki 3vtqzp8236b2242yow6os7i7u4fk345 2430674 2430673 2026-06-25T13:28:42Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2430674 wikitext text/x-wiki mprryxtpjd73959nbm6sbdc1o1nxl17 2430676 2430674 2026-06-25T13:37:37Z Stashbot 7414 moritzm: installing imagemagick security updates 2430676 wikitext text/x-wiki 76tidu33kuy1kur1a7lp8jn95qhzlrw 2430677 2430676 2026-06-25T13:51:13Z Stashbot 7414 jmm@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host build2004.codfw.wmnet with OS trixie 2430677 wikitext text/x-wiki ng3payjgrcbq9go89m4djij6iersi3p 2430678 2430677 2026-06-25T13:51:15Z Stashbot 7414 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host build2004.codfw.wmnet 2430678 wikitext text/x-wiki c11cxsls87lu3i3hgungb8c1ut4ik9e 2430679 2430678 2026-06-25T13:54:45Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2225: rack depool 2430679 wikitext text/x-wiki ansyg4ky4331b0gn5jtgkd736pgy4ty 2430680 2430679 2026-06-25T13:55:14Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2224: rack depool 2430680 wikitext text/x-wiki ihp502a4o05mm69hrul9ezj8op6cm8u 2430681 2430680 2026-06-25T13:57:04Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1243: Migration of db1243.eqiad.wmnet completed 2430681 wikitext text/x-wiki rm9z1kidy5r2c5578rlxx6q6whskrms 2430682 2430681 2026-06-25T13:57:06Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430682 wikitext text/x-wiki 9hjo9hbj28qm0edojy0l4mu431nvh5s 2430684 2430682 2026-06-25T14:00:42Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2237: Migration of db2237.codfw.wmnet completed 2430684 wikitext text/x-wiki i4pi4zr5f1quzesyk26i7bc5c2y9t2f 2430685 2430684 2026-06-25T14:00:44Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430685 wikitext text/x-wiki 90lft4fbosalfvx95ft1of33psdbtxn 2430686 2430685 2026-06-25T14:04:03Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1077.eqiad.wmnet with OS trixie 2430686 wikitext text/x-wiki pxsuhw8ljnn7fvz0wnk7jb0b0lxg5ro 2430687 2430686 2026-06-25T14:05:17Z Stashbot 7414 Dreamy_Jazz: Ran `delete from cuci_user where ciu_ciwm_id = 4;` for T430156 2430687 wikitext text/x-wiki f90xy6z7tjk9e1reb8ij3b4wgbyk1e6 2430688 2430687 2026-06-25T14:08:36Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430688 wikitext text/x-wiki qgedc4fcvmbwrotxccxg8legnc3yj2d 2430689 2430688 2026-06-25T14:08:40Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@eqiad T429893 2430689 wikitext text/x-wiki al9w4u4v87utva79vnlfogslf45y0vt 2430690 2430689 2026-06-25T14:08:56Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1247: Upgrading db1247.eqiad.wmnet 2430690 wikitext text/x-wiki 8hlvhoxsyr1kn5ooz2frqyt7er251y8 2430691 2430690 2026-06-25T14:09:27Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1247: Upgrading db1247.eqiad.wmnet 2430691 wikitext text/x-wiki 0f60ny4iurdoi8mu1iz95olso1679w6 2430692 2430691 2026-06-25T14:11:33Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1247.eqiad.wmnet with OS trixie 2430692 wikitext text/x-wiki szof0vnjn338ry1o7olb8crdh6xz5ze 2430693 2430692 2026-06-25T14:11:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430693 wikitext text/x-wiki d7g5kaaeq7t2lkmv6adxv79cnn75ebm 2430694 2430693 2026-06-25T14:11:57Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@codfw T429893 2430694 wikitext text/x-wiki 2eo85in766mhwp07rknx5zrk258kp77 2430695 2430694 2026-06-25T14:12:12Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2245: Upgrading db2245.codfw.wmnet 2430695 wikitext text/x-wiki 8mbjar68wi0oxwaklvqh22godfistkk 2430696 2430695 2026-06-25T14:12:42Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: Upgrading db2245.codfw.wmnet 2430696 wikitext text/x-wiki 8gg74hfsxr1a2blvcxlhuheh6abvfep 2430697 2430696 2026-06-25T14:14:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2245.codfw.wmnet with OS trixie 2430697 wikitext text/x-wiki 0piv8ojt74mjhappka4m7mkkpp75wdp 2430698 2430697 2026-06-25T14:21:03Z Stashbot 7414 hashar: Restarting CI Jenkins on contint1002 2430698 wikitext text/x-wiki poii288310bgbs43dne6b0o03xqh050 2430701 2430698 2026-06-25T14:28:04Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host build2004.codfw.wmnet with OS trixie 2430701 wikitext text/x-wiki saru2yoxbz5yhfbe1owa8y7zaxzgvsi 2430702 2430701 2026-06-25T14:28:32Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1247.eqiad.wmnet with reason: host reimage 2430702 wikitext text/x-wiki olwry99k31d7mzz0dtnp755tvozedkg 2430703 2430702 2026-06-25T14:30:33Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2245.codfw.wmnet with reason: host reimage 2430703 wikitext text/x-wiki eriykfdzyn43m7rhh0btcdy0d03n9ji 2430704 2430703 2026-06-25T14:34:21Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1247.eqiad.wmnet with reason: host reimage 2430704 wikitext text/x-wiki gsjw9u540c6h8uz1mdmk7gxznf9gndj 2430705 2430704 2026-06-25T14:36:25Z Stashbot 7414 marostegui: Drop database apiportalwiki on sanitarium and wikireplicas T430102 2430705 wikitext text/x-wiki ghjvkgbyv1v34ocircit83dn3u4sbkt 2430706 2430705 2026-06-25T14:38:08Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2245.codfw.wmnet with reason: host reimage 2430706 wikitext text/x-wiki ogdc0y1f4vwdfhwf91qd52khk24ijy7 2430707 2430706 2026-06-25T14:42:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2234.codfw.wmnet with OS trixie 2430707 wikitext text/x-wiki 7vzfm9coktvcftuner6pvd2v5nlj5i2 2430709 2430707 2026-06-25T14:48:07Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on build2004.codfw.wmnet with reason: host reimage 2430709 wikitext text/x-wiki hp3cjgspp6xkxsild8cknz4u4lzaoai 2430710 2430709 2026-06-25T14:51:47Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1247.eqiad.wmnet with OS trixie 2430710 wikitext text/x-wiki qoegyvmx0kr107v6jr7wj8l5fqp36k1 2430711 2430710 2026-06-25T14:52:05Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on build2004.codfw.wmnet with reason: host reimage 2430711 wikitext text/x-wiki ra4ptlv2mbofvj5s961mfj4e9pjvksx 2430713 2430711 2026-06-25T14:56:10Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2245.codfw.wmnet with OS trixie 2430713 wikitext text/x-wiki ophlmwpdkhifv7o84942m1s340fejjz 2430714 2430713 2026-06-25T14:59:40Z Stashbot 7414 papaul: ongoing maintenance on cr2-eqdfw 2430714 wikitext text/x-wiki relx5iz6yunc34ntgodvcanfeej78fu 2430715 2430714 2026-06-25T15:00:39Z Stashbot 7414 pt1979@cumin2003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-eqdfw cr2-eqdfw IPv6 with reason: junos upgrade 2430715 wikitext text/x-wiki sjl75r9968a1k5rdl0vksa52c4x1rzg 2430716 2430715 2026-06-25T15:02:43Z Stashbot 7414 pt1979@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: junos upgrade 2430716 wikitext text/x-wiki 14e28pkgn6e7r8lawdgn5foetu06vws 2430717 2430716 2026-06-25T15:06:15Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1247: Migration of db1247.eqiad.wmnet completed 2430717 wikitext text/x-wiki 0fk0jpkhti94vnidnxx73lo1nq95l0h 2430718 2430717 2026-06-25T15:08:44Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host build2004.codfw.wmnet with OS trixie 2430718 wikitext text/x-wiki 6efhfxsokgkph2u8l1zyfcn9to97jwa 2430719 2430718 2026-06-25T15:11:17Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2245: Migration of db2245.codfw.wmnet completed 2430719 wikitext text/x-wiki 81xwb1uvabi121pkywgcqwk9czjoqrd 2430720 2430719 2026-06-25T15:12:31Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2430720 wikitext text/x-wiki hpujxi636uoigg6kuqhsj3kl7n27iab 2430721 2430720 2026-06-25T15:25:14Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2430721 wikitext text/x-wiki 3myymtuypxz6pxl0kc4z8s06znjba5a 2430722 2430721 2026-06-25T15:26:03Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2430722 wikitext text/x-wiki j5fodq2auyskf1xev9v950ifnugyiio 2430723 2430722 2026-06-25T15:26:15Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2430723 wikitext text/x-wiki 6h2ilu3cq2tn8yih0je60gy656qlqlk 2430724 2430723 2026-06-25T15:26:46Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2430724 wikitext text/x-wiki 0b8yrpc4mmnxkee6bsxppjdr5xxt06t 2430725 2430724 2026-06-25T15:26:51Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2430725 wikitext text/x-wiki aydxd8wmvwb5azdrtpr42t3gmqzpgzn 2430726 2430725 2026-06-25T15:27:19Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2430726 wikitext text/x-wiki 02c6fcs0cx8h4w9vj1c82lkxpfr56ms 2430727 2430726 2026-06-25T15:36:05Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2430727 wikitext text/x-wiki dzz48ybtixf785zbqq3qyceaa2h19yq 2430728 2430727 2026-06-25T15:40:22Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2430728 wikitext text/x-wiki q903zjjpb0tgsfd381vn7smjg3ssdd8 2430729 2430728 2026-06-25T15:47:11Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2430729 wikitext text/x-wiki 6ol82zczizjhi3nxeglro3a7li99hmq 2430730 2430729 2026-06-25T15:48:34Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2430730 wikitext text/x-wiki c5g9h854qu98vc9ian3rku6ru0yrxmj 2430731 2430730 2026-06-25T15:51:46Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1247: Migration of db1247.eqiad.wmnet completed 2430731 wikitext text/x-wiki m3cicq6fbas0uyjtgo9e7eek0eud9ww 2430732 2430731 2026-06-25T15:51:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430732 wikitext text/x-wiki e76xvt8konqmf8gcusf6co2tyawctc2 2430734 2430732 2026-06-25T15:56:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Migration of db2245.codfw.wmnet completed 2430734 wikitext text/x-wiki kpy50ktj2w2rgcq4tiejxcnwvbe2m20 2430735 2430734 2026-06-25T15:56:51Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430735 wikitext text/x-wiki 6uobsihgzr005exqjlnexn96zv58b95 2430737 2430735 2026-06-25T16:02:21Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2234.codfw.wmnet with OS trixie 2430737 wikitext text/x-wiki h2gemmyhv9u0iln7ubvtswpbdpfram4 2430741 2430737 2026-06-25T16:17:04Z Stashbot 7414 pt1979@cumin2003: START - Cookbook sre.hosts.remove-downtime for cr2-eqdfw,cr2-eqdfw IPv6 2430741 wikitext text/x-wiki blnxeaywnxc9kpmv71vqkqteliqsv7g 2430742 2430741 2026-06-25T16:17:07Z Stashbot 7414 pt1979@cumin2003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqdfw,cr2-eqdfw IPv6 2430742 wikitext text/x-wiki 1as1cajv5w6zxnnv6xpewcmnk100okp 2430743 2430742 2026-06-25T16:20:23Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430743 wikitext text/x-wiki 4fc1oae37e9p4epq94q08uffvf5eois 2430744 2430743 2026-06-25T16:20:27Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@eqiad T429893 2430744 wikitext text/x-wiki ke4sp1okqcqvflf85fjb2ll4x3jv1o0 2430745 2430744 2026-06-25T16:20:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1248: Upgrading db1248.eqiad.wmnet 2430745 wikitext text/x-wiki a6kiihgn6hvv4zzv2ixselh6v6pfqm6 2430746 2430745 2026-06-25T16:20:47Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2430746 wikitext text/x-wiki n98iow7gedbp97zu6dq86jhqorgx4w6 2430747 2430746 2026-06-25T16:20:51Z Stashbot 7414 cwilliams@cumin1003: dbmaint on s4@codfw T429893 2430747 wikitext text/x-wiki d6376vo1twsfyzffgfz5vp921n7pw2h 2430748 2430747 2026-06-25T16:21:08Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2246: Upgrading db2246.codfw.wmnet 2430748 wikitext text/x-wiki 9cfmxejz0z380mk8xbht1v0uk8peowd 2430750 2430748 2026-06-25T16:21:42Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2246: Upgrading db2246.codfw.wmnet 2430750 wikitext text/x-wiki sgumqhaqh2712p8gu0h014lmcdemq82 2430751 2430750 2026-06-25T16:21:44Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1248: Upgrading db1248.eqiad.wmnet 2430751 wikitext text/x-wiki t0nq9syoozmbsmix7c4rz3w6olpkpvx 2430752 2430751 2026-06-25T16:23:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1248.eqiad.wmnet with OS trixie 2430752 wikitext text/x-wiki ebpl21qam1nbjb7kyqa34hvpc929p6a 2430754 2430752 2026-06-25T16:25:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2246.codfw.wmnet with OS trixie 2430754 wikitext text/x-wiki tqg9hupimbcd5vetx464b02jbkaux2m 2430766 2430754 2026-06-25T16:37:08Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1305719|Bump wikimedia/parsoid to 0.24.0-a12 (T353697 T384490 T387374 T387520 T387521 T391624 T393295 T420336 T429624 T429688 T429822)]], [[gerrit:1305720|Bump wikimedia/parsoid to 0.24.0-a12 (T429822)]] 2430766 wikitext text/x-wiki fik52ccde9zss5c442iqu2rz1waxmrc 2430768 2430766 2026-06-25T16:39:01Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1305719|Bump wikimedia/parsoid to 0.24.0-a12 (T353697 T384490 T387374 T387520 T387521 T391624 T393295 T420336 T429624 T429688 T429822)]], [[gerrit:1305720|Bump wikimedia/parsoid to 0.24.0-a12 (T429822)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430768 wikitext text/x-wiki kiwtug71g1u7b4ow1xwbp7zwxugag66 2430770 2430768 2026-06-25T16:40:31Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1248.eqiad.wmnet with reason: host reimage 2430770 wikitext text/x-wiki 3g46ueaxtnoqh7u6j4ys21ae2ezcmzk 2430771 2430770 2026-06-25T16:41:23Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker1160.eqiad.wmnet with OS trixie 2430771 wikitext text/x-wiki 5cqnm5eu6pzgv2b6bsruuqtj6tnfc21 2430772 2430771 2026-06-25T16:41:39Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2246.codfw.wmnet with reason: host reimage 2430772 wikitext text/x-wiki iyefqc2k0gios08dymjlqn7ev1a5fdw 2430773 2430772 2026-06-25T16:41:56Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1160 2430773 wikitext text/x-wiki 699hvn57egumbu1ivp860g0xv7oa359 2430774 2430773 2026-06-25T16:42:14Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2430774 wikitext text/x-wiki 6r3lzuhi9vd6otm1t70q1ut2bv7qbfr 2430775 2430774 2026-06-25T16:42:59Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.netbox 2430775 wikitext text/x-wiki jzh5rm29liikv9wjr420lioa9ttblki 2430776 2430775 2026-06-25T16:45:08Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1248.eqiad.wmnet with reason: host reimage 2430776 wikitext text/x-wiki pjhovlbg3nr1dy099sx1p3z93ebltwu 2430777 2430776 2026-06-25T16:46:54Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305719|Bump wikimedia/parsoid to 0.24.0-a12 (T353697 T384490 T387374 T387520 T387521 T391624 T393295 T420336 T429624 T429688 T429822)]], [[gerrit:1305720|Bump wikimedia/parsoid to 0.24.0-a12 (T429822)]] (duration: 09m 54s) 2430777 wikitext text/x-wiki a380t7md56d809lwegawc1i89231iux 2430778 2430777 2026-06-25T16:47:18Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1160 - jasmine@cumin2002" 2430778 wikitext text/x-wiki pa5mea7bqvw2l5yemmom7yu4b2lvt2b 2430779 2430778 2026-06-25T16:47:23Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1160 - jasmine@cumin2002" 2430779 wikitext text/x-wiki 1sls0bmu2dee7lmiupiclsxvvjja0st 2430780 2430779 2026-06-25T16:47:25Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430780 wikitext text/x-wiki o5u17si37qhn31w1ypnoe5k3cixhq3u 2430781 2430780 2026-06-25T16:47:27Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker1160.eqiad.wmnet 116.48.64.10.in-addr.arpa 6.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2430781 wikitext text/x-wiki a6abqvpp5exci5qxlvhegsqwb5mj7qj 2430782 2430781 2026-06-25T16:47:30Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1160.eqiad.wmnet 116.48.64.10.in-addr.arpa 6.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2430782 wikitext text/x-wiki abusd9bnaq2pwzi9n1y0887iwh2erzr 2430783 2430782 2026-06-25T16:47:32Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1160 2430783 wikitext text/x-wiki gk9ehhhi312sm0yab30a956xfdkvwwn 2430784 2430783 2026-06-25T16:48:56Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1160 2430784 wikitext text/x-wiki 1jkk3sbiwagglfjy1kzscxw7l62aqwt 2430785 2430784 2026-06-25T16:48:58Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1160 2430785 wikitext text/x-wiki t2u6dkvsmggvc4sg7upfil0cto1eah0 2430786 2430785 2026-06-25T16:50:04Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2246.codfw.wmnet with reason: host reimage 2430786 wikitext text/x-wiki 5jmyktg75pauwd9tgo6yltjiqccoswm 2430789 2430786 2026-06-25T16:50:52Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1305711|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (T429624 T429822 T391624)]] 2430789 wikitext text/x-wiki 6m7j70l8kaqg4sy6w3kaya9trd9ncy2 2430790 2430789 2026-06-25T16:52:58Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1305711|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (T429624 T429822 T391624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430790 wikitext text/x-wiki 0kzaecluqu2yce84e9cy02p2exw66oc 2430791 2430790 2026-06-25T16:59:15Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2430791 wikitext text/x-wiki qwdu2e2ja29f4z4n9esjsozo9gq5njt 2430794 2430791 2026-06-25T17:02:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1248.eqiad.wmnet with OS trixie 2430794 wikitext text/x-wiki bm4zvq60yxkt70wlenkt1t5fopmxa7l 2430796 2430794 2026-06-25T17:04:06Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1160.eqiad.wmnet with reason: host reimage 2430796 wikitext text/x-wiki 5y36ti515ou2pf1apjy5rf74cvetd44 2430797 2430796 2026-06-25T17:07:47Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2246.codfw.wmnet with OS trixie 2430797 wikitext text/x-wiki iai057odv03qdhv0zegywc5fvbwb8qm 2430799 2430797 2026-06-25T17:09:23Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1160.eqiad.wmnet with reason: host reimage 2430799 wikitext text/x-wiki iottehfn4c9lml94bs4wh6uuhwuyjap 2430803 2430799 2026-06-25T17:15:02Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1248: Migration of db1248.eqiad.wmnet completed 2430803 wikitext text/x-wiki h0crma2u5x8s0ti0vn4bp204w1angbz 2430804 2430803 2026-06-25T17:22:17Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2246: Migration of db2246.codfw.wmnet completed 2430804 wikitext text/x-wiki q1ognaaoaw7hyxgi8mexcvuiehh5ht8 2430805 2430804 2026-06-25T17:30:02Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1160.eqiad.wmnet with OS trixie 2430805 wikitext text/x-wiki m6ko80p96i4ebdomvpipgrpeqpcwjta 2430806 2430805 2026-06-25T17:32:55Z Stashbot 7414 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply 2430806 wikitext text/x-wiki 7road3tknxlcidrp4hi5j6bc8a2y2kg 2430807 2430806 2026-06-25T17:33:09Z Stashbot 7414 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply 2430807 wikitext text/x-wiki 3glg3ctp4ev8o25qky0m95qykp22b5n 2430808 2430807 2026-06-25T17:33:24Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply 2430808 wikitext text/x-wiki 217t538zigpjuhyyysty3nr9ku7cl8p 2430809 2430808 2026-06-25T17:34:10Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply 2430809 wikitext text/x-wiki qfub30yo66wzlwtw3ic3pbwx5iqwv4j 2430810 2430809 2026-06-25T17:34:22Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply 2430810 wikitext text/x-wiki 1mad48lfkdkd7vp6adx107p5nkonkhq 2430811 2430810 2026-06-25T17:34:37Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply 2430811 wikitext text/x-wiki 98r4abd19ailejy7r9jazyhok8rgn6g 2430812 2430811 2026-06-25T17:37:34Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker1161.eqiad.wmnet with OS trixie 2430812 wikitext text/x-wiki ieo4ow99p0brenqyouzmjrmd6rn4ix0 2430813 2430812 2026-06-25T17:38:06Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1161 2430813 wikitext text/x-wiki f0npb6qi5zcwff7vq19mrmzezivwt4j 2430814 2430813 2026-06-25T17:38:50Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.netbox 2430814 wikitext text/x-wiki cg5b9zqhkyzudob6y8t497x2ovaisgr 2430815 2430814 2026-06-25T17:41:16Z Stashbot 7414 cscott@deploy1003: Rolling back deployment 2430815 wikitext text/x-wiki 9qvcg2tlvfz4abek1nvgxdmjug5hiju 2430817 2430815 2026-06-25T17:42:51Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305711|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (T429624 T429822 T391624)]] (duration: 51m 59s) 2430817 wikitext text/x-wiki ck1d0b6s8oaxk5isw07fdy43cbzp50x 2430818 2430817 2026-06-25T17:47:37Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1161 - jasmine@cumin2002" 2430818 wikitext text/x-wiki 09df3u5u23d879dlkzdx5pglnl4zqlj 2430819 2430818 2026-06-25T17:47:42Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1161 - jasmine@cumin2002" 2430819 wikitext text/x-wiki qbeb8w29vaa3n039m4ge5l1gdugdg7e 2430820 2430819 2026-06-25T17:47:44Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430820 wikitext text/x-wiki lqb0buwt4pbwudu473oqb04bk5tphoe 2430821 2430820 2026-06-25T17:47:47Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker1161.eqiad.wmnet 118.48.64.10.in-addr.arpa 8.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2430821 wikitext text/x-wiki bgzu202xnpmicczfd3e11zeunbs4vwv 2430822 2430821 2026-06-25T17:47:50Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1161.eqiad.wmnet 118.48.64.10.in-addr.arpa 8.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2430822 wikitext text/x-wiki h8p3a8v8mg611ke1uwtozh8dyrlwg8g 2430823 2430822 2026-06-25T17:47:57Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1161 2430823 wikitext text/x-wiki b7csmu1rfioc167fqkrqsik6v86aome 2430824 2430823 2026-06-25T17:48:38Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2430824 wikitext text/x-wiki pfwd8v2qgzg58y5alunttoa4y6znpsa 2430826 2430824 2026-06-25T17:58:33Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1161 2430826 wikitext text/x-wiki tw0a8yv0tq92r3zq30rv4kjz5l1h5ox 2430827 2430826 2026-06-25T17:58:36Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1161 2430827 wikitext text/x-wiki dwgu5op1g53fzynhgdjbyq23v12lz2y 2430828 2430827 2026-06-25T18:00:29Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1248: Migration of db1248.eqiad.wmnet completed 2430828 wikitext text/x-wiki tkrjqtzpqje4b22jhgxlbkssum4fvkk 2430829 2430828 2026-06-25T18:00:32Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430829 wikitext text/x-wiki eojspnvb4skirv7kxne8t8i7nr6fbwy 2430830 2430829 2026-06-25T18:03:09Z Stashbot 7414 reedy@deploy1003: Synchronized private/: (no justification provided) (duration: 06m 14s) 2430830 wikitext text/x-wiki 0u1cn3u6x86fwlw9xjzm64rbpq8rykv 2430832 2430830 2026-06-25T18:07:47Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2246: Migration of db2246.codfw.wmnet completed 2430832 wikitext text/x-wiki b2ta5ze3l4rpc7g139ylarbl54kadbu 2430833 2430832 2026-06-25T18:07:50Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2430833 wikitext text/x-wiki b7k0hcp5cpuwgu8x2uqzzi7jcul2czt 2430834 2430833 2026-06-25T18:15:31Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1161.eqiad.wmnet with reason: host reimage 2430834 wikitext text/x-wiki g7wcrz9rsg1f1ts0eln3ip887vfx00f 2430836 2430834 2026-06-25T18:20:16Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1161.eqiad.wmnet with reason: host reimage 2430836 wikitext text/x-wiki a9c8hl8t3iujgd2esay5oy0xulkpctv 2430837 2430836 2026-06-25T18:20:54Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2430837 wikitext text/x-wiki jrbapz0dpui0z2ewk8npoe1c8b9htm8 2430838 2430837 2026-06-25T18:23:50Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430838 wikitext text/x-wiki 6pt5pc2gxbm60t6v44800g5eldwrekc 2430839 2430838 2026-06-25T18:25:19Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1004 2430839 wikitext text/x-wiki 08ge2h0xy5b3ax36w9kq68z2rvhz15f 2430840 2430839 2026-06-25T18:26:11Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1004 2430840 wikitext text/x-wiki 6s754zcp1qs7c5trrlpi54l1tjhe707 2430841 2430840 2026-06-25T18:28:29Z Stashbot 7414 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.8 refs T423917 2430841 wikitext text/x-wiki gc1m8k5m9taia3xd740iniug9tx7b16 2430844 2430841 2026-06-25T18:38:17Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1305501|InitialiseSettings: Require 2FA for all on arbcom_*wiki and conductwiki (T428103)]] 2430844 wikitext text/x-wiki ody33tmtd9yp435ha285w6p0phj0zrg 2430845 2430844 2026-06-25T18:40:21Z Stashbot 7414 reedy@deploy1003: reedy: Backport for [[gerrit:1305501|InitialiseSettings: Require 2FA for all on arbcom_*wiki and conductwiki (T428103)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430845 wikitext text/x-wiki ogx3amqfj7yy2tkmyqy11j6t9mrs58z 2430846 2430845 2026-06-25T18:41:12Z Stashbot 7414 reedy@deploy1003: reedy: Continuing with deployment 2430846 wikitext text/x-wiki 64ezlmrjn82ffxfbcwgtp3uh0wym3sm 2430847 2430846 2026-06-25T18:41:22Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1004 2430847 wikitext text/x-wiki rl04o4qy3cftcszbd8q35v62ggl9xum 2430848 2430847 2026-06-25T18:41:41Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1161.eqiad.wmnet with OS trixie 2430848 wikitext text/x-wiki 33t5f0gnig87awp22r53lb2rti1hd6d 2430849 2430848 2026-06-25T18:43:08Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1004 2430849 wikitext text/x-wiki 0ueqeu30c3704bydy0gwjhzc1cofgnu 2430850 2430849 2026-06-25T18:45:29Z Stashbot 7414 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305501|InitialiseSettings: Require 2FA for all on arbcom_*wiki and conductwiki (T428103)]] (duration: 07m 12s) 2430850 wikitext text/x-wiki lr8yezs7ganiy18bugbszeelx1rlt0x 2430851 2430850 2026-06-25T18:50:02Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2430851 wikitext text/x-wiki gpth6ll5o3r50ig1xlgjpckvmdx3wn7 2430852 2430851 2026-06-25T18:57:33Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker1162.eqiad.wmnet with OS trixie 2430852 wikitext text/x-wiki 435m8cf9nwvjqy9nfh80zg7shc7yg6x 2430853 2430852 2026-06-25T18:57:48Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1004] - vriley@cumin1003" 2430853 wikitext text/x-wiki rm993at5254tynq5118e1zi61nc2n7c 2430854 2430853 2026-06-25T18:57:53Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1004] - vriley@cumin1003" 2430854 wikitext text/x-wiki lofqas8q02dgo483wudn60y4v3h9v9k 2430855 2430854 2026-06-25T18:57:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430855 wikitext text/x-wiki 4jxmx1w7w2rbybffe5fhpz194mtrp40 2430856 2430855 2026-06-25T18:57:57Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1162 2430856 wikitext text/x-wiki j9ebivuk3dkq1eto3gjmp43t5uzlv7h 2430857 2430856 2026-06-25T18:58:47Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2430857 wikitext text/x-wiki hgijtu0vihadc084e4b1ncgmyoow2ie 2430859 2430857 2026-06-25T19:03:03Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1004] - vriley@cumin1003" 2430859 wikitext text/x-wiki oueml1lt3v6pgplnulx0qwhlkw5bm3e 2430860 2430859 2026-06-25T19:03:07Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1004] - vriley@cumin1003" 2430860 wikitext text/x-wiki aos6iqtq32fay1tc1drxii7xy1curtv 2430861 2430860 2026-06-25T19:03:09Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430861 wikitext text/x-wiki 99lpklzej6irp0l5r4ak2z49ctmf17p 2430862 2430861 2026-06-25T19:03:28Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.netbox 2430862 wikitext text/x-wiki pvmjfpbap248qnd2m5dnrjn7hf45imn 2430863 2430862 2026-06-25T19:05:41Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1305733|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (T429624 T429822 T391624)]] 2430863 wikitext text/x-wiki dnutuq917mafm7awfjojwq675nqekzs 2430864 2430863 2026-06-25T19:06:30Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430864 wikitext text/x-wiki 260ua22nf38bp5vknus3blg7flsk7e5 2430865 2430864 2026-06-25T19:06:33Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker1162.eqiad.wmnet 108.48.64.10.in-addr.arpa 8.0.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2430865 wikitext text/x-wiki e5z1jhq3m7mfliqrueh1jahqqm8x6g1 2430866 2430865 2026-06-25T19:06:35Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1162.eqiad.wmnet 108.48.64.10.in-addr.arpa 8.0.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2430866 wikitext text/x-wiki rh6pio6r25fpbgqcdzw0uvhieh76fks 2430867 2430866 2026-06-25T19:06:37Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1162 2430867 wikitext text/x-wiki ntf9i5cgw1rhlf91ek0dy28s7m8hryl 2430868 2430867 2026-06-25T19:07:04Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2430868 wikitext text/x-wiki sdfbnmiub02t8f6defxfa5z4yn5j7zo 2430869 2430868 2026-06-25T19:07:08Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2002 2430869 wikitext text/x-wiki 0f4137mu4gkco6uslxhdrvtnqvzy6tv 2430870 2430869 2026-06-25T19:07:13Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2002 2430870 wikitext text/x-wiki 2g1b4onlg8s2oevp3hha893g20gua32 2430871 2430870 2026-06-25T19:07:35Z Stashbot 7414 cscott@deploy1003: cscott, arlolra: Backport for [[gerrit:1305733|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (T429624 T429822 T391624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430871 wikitext text/x-wiki cofqdgso6x6vj3fl57dccefliv2q5jp 2430872 2430871 2026-06-25T19:07:41Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2002 2430872 wikitext text/x-wiki pci42r6w44lre5f8nrfwj0gbdgyimgv 2430873 2430872 2026-06-25T19:07:45Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2002 2430873 wikitext text/x-wiki r1vqqnzq8ucmu0wnztemzd9m7hslvcl 2430874 2430873 2026-06-25T19:08:14Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1162 2430874 wikitext text/x-wiki inyayrwps4n2x58it20zvx0as3u15kb 2430875 2430874 2026-06-25T19:08:16Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1162 2430875 wikitext text/x-wiki 0tnlv6ev15wkttxd19zuhtroka96y0g 2430876 2430875 2026-06-25T19:08:20Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2002 2430876 wikitext text/x-wiki ktekriuhq7iusi6qw96dn622d3juph3 2430877 2430876 2026-06-25T19:08:23Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2002 2430877 wikitext text/x-wiki coahcip6515vex9p445opephxonpf00 2430878 2430877 2026-06-25T19:08:48Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2003 2430878 wikitext text/x-wiki 47drogr2u27ssu587ux5svej6678o49 2430879 2430878 2026-06-25T19:08:51Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2003 2430879 wikitext text/x-wiki gyu1jmtyzhbvqfxf1y6dyv1pyx87azf 2430880 2430879 2026-06-25T19:10:02Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430880 wikitext text/x-wiki diagrcaigv764aljb200woje4et4j41 2430881 2430880 2026-06-25T19:10:13Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1004 2430881 wikitext text/x-wiki la5nnhvt3ctisa7e84ktruz47ly85nx 2430882 2430881 2026-06-25T19:11:25Z Stashbot 7414 cscott@deploy1003: cscott, arlolra: Continuing with deployment 2430882 wikitext text/x-wiki 3tbcewf86b6qzgjl10ffbvwjoie1uvm 2430883 2430882 2026-06-25T19:11:32Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1004 2430883 wikitext text/x-wiki 9ibt998digfqsbdpuc5jj7q6km4t9dd 2430884 2430883 2026-06-25T19:12:31Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2430884 wikitext text/x-wiki idfmmzbup6ebixqb4j79b5chadp4oi8 2430885 2430884 2026-06-25T19:18:26Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2430885 wikitext text/x-wiki gb1urahoenh4q27332jbs8xrkgpm0ho 2430887 2430885 2026-06-25T19:24:41Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1162.eqiad.wmnet with reason: host reimage 2430887 wikitext text/x-wiki nnsqplh8ob4h2b9z0pflnla4wkfacm9 2430888 2430887 2026-06-25T19:26:31Z Stashbot 7414 cscott@deploy1003: Rolling back deployment 2430888 wikitext text/x-wiki g1tnit301g0utucgfshsgvjxp786gfb 2430889 2430888 2026-06-25T19:27:29Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool es2045: rack depool 2430889 wikitext text/x-wiki 3gcymy2c75n15xvn14vnbx8eax8tzv5 2430890 2430889 2026-06-25T19:27:49Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305733|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (T429624 T429822 T391624)]] (duration: 22m 07s) 2430890 wikitext text/x-wiki 3sikmrpquxtmiwxrqv9d1958guw2jws 2430891 2430890 2026-06-25T19:29:42Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1162.eqiad.wmnet with reason: host reimage 2430891 wikitext text/x-wiki kyrarmlz6kuw5ojvhrld0vnvpj119dh 2430892 2430891 2026-06-25T19:49:10Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1162.eqiad.wmnet with OS trixie 2430892 wikitext text/x-wiki hzg7r027l736anr3j1ilnru89e9ukpl 2430893 2430892 2026-06-25T19:52:06Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1305746|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (take 2) (T429624 T429822 T391624)]] 2430893 wikitext text/x-wiki fi02ehuawr6ygbguvkkbl8vdwelcgvt 2430894 2430893 2026-06-25T19:53:36Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1305746|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (take 2) (T429624 T429822 T391624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430894 wikitext text/x-wiki 3zeglpfmbjkt1t2i2c2gof3hf8mj2y8 2430895 2430894 2026-06-25T19:54:43Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2430895 wikitext text/x-wiki b5b4os1id0sxodp1g10jc6sk7y4bkbe 2430896 2430895 2026-06-25T19:59:05Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305746|Turn on Parsoid's 'ReturnExperimentalPFragmentTypes' (take 2) (T429624 T429822 T391624)]] (duration: 06m 58s) 2430896 wikitext text/x-wiki qbujiuz3d3dkir4t1kiauhzx8wozajg 2430898 2430896 2026-06-25T20:08:59Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1304888|Deploy PRV to 5 wikis (T429830)]], [[gerrit:1305290|Enable ULS v2 by default across all wikis]], [[gerrit:1305639|Remove wgCiteRemoveSyntheticRefsUnsafe feature flag from production and beta cluster config (T428232)]] 2430898 wikitext text/x-wiki 5xd8sqy65m247ix5akidlepbyedxm3q 2430899 2430898 2026-06-25T20:10:53Z Stashbot 7414 arlolra@deploy1003: arlolra, mareikeheuer, abi: Backport for [[gerrit:1304888|Deploy PRV to 5 wikis (T429830)]], [[gerrit:1305290|Enable ULS v2 by default across all wikis]], [[gerrit:1305639|Remove wgCiteRemoveSyntheticRefsUnsafe feature flag from production and beta cluster config (T428232)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430899 wikitext text/x-wiki hmn830lkhdzs402zx5x3h5zjcxe6bej 2430900 2430899 2026-06-25T20:12:42Z Stashbot 7414 arlolra@deploy1003: arlolra, mareikeheuer, abi: Continuing with deployment 2430900 wikitext text/x-wiki h4g7pgj2jnhy8i1v4l8kxud88af459a 2430901 2430900 2026-06-25T20:12:56Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2045: rack depool 2430901 wikitext text/x-wiki cv6028j209jxtne0fmsor6799x211jy 2430902 2430901 2026-06-25T20:17:02Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1304888|Deploy PRV to 5 wikis (T429830)]], [[gerrit:1305290|Enable ULS v2 by default across all wikis]], [[gerrit:1305639|Remove wgCiteRemoveSyntheticRefsUnsafe feature flag from production and beta cluster config (T428232)]] (duration: 08m 04s) 2430902 wikitext text/x-wiki 7a7n40djbgmh655itette31du2lrs3b 2430904 2430902 2026-06-25T20:51:51Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2430904 wikitext text/x-wiki 3vqlvrpi8mnmq7n2e8m81bff9m44dop 2430905 2430904 2026-06-25T20:55:02Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2430905 wikitext text/x-wiki 3xyjx4k550zeanteuqdgqase10ykr5f 2430906 2430905 2026-06-25T20:55:43Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2430906 wikitext text/x-wiki gzy2tugq4xhca1u7lef12xkk56n776t 2430908 2430906 2026-06-25T20:58:33Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430908 wikitext text/x-wiki 57wopg22c6xx9sjxb31uh5264rpddrs 2430909 2430908 2026-06-25T20:59:30Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2430909 wikitext text/x-wiki fllxzwj7s6j5jdp0tbjdpnrtxtb3adc 2430910 2430909 2026-06-25T21:04:09Z Stashbot 7414 dani@deploy1003: Started scap sync-world: Backport for [[gerrit:1304706|Undeploy English Wikipedia Mobile App Survey (T428876)]] 2430910 wikitext text/x-wiki oclivjtprjts0res652o37z00d93uf6 2430911 2430910 2026-06-25T21:06:06Z Stashbot 7414 dani@deploy1003: dani: Backport for [[gerrit:1304706|Undeploy English Wikipedia Mobile App Survey (T428876)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430911 wikitext text/x-wiki pl6nzdeub8x88uzjzahvomq8bsgemzb 2430912 2430911 2026-06-25T21:06:45Z Stashbot 7414 dani@deploy1003: dani: Continuing with deployment 2430912 wikitext text/x-wiki 090q5w3hvntijy4ij7cccx8vpcwenwh 2430913 2430912 2026-06-25T21:11:03Z Stashbot 7414 dani@deploy1003: Finished scap sync-world: Backport for [[gerrit:1304706|Undeploy English Wikipedia Mobile App Survey (T428876)]] (duration: 06m 54s) 2430913 wikitext text/x-wiki 0ujqp4glr69tkjsqc4mixjn613hwhr7 2430914 2430913 2026-06-25T21:15:15Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1305749|Restore carousel beta opt-in, suppressed when enabled sitewide (T429414)]], [[gerrit:1305741|Roll back mobile image carousel from sitewide to beta opt-in (T429414)]] 2430914 wikitext text/x-wiki gy14ov6q39zdswpf5blylbcdyy3k9ka 2430915 2430914 2026-06-25T21:15:50Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2430915 wikitext text/x-wiki 877spta4ad7gftljvd0qmpas4x177jz 2430916 2430915 2026-06-25T21:32:08Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2003 2430916 wikitext text/x-wiki 0l3rinqjvtlpcqj7pdijxka4hk2cyh6 2430917 2430916 2026-06-25T21:33:11Z Stashbot 7414 jdlrobson@deploy1003: egardner, jdlrobson: Backport for [[gerrit:1305749|Restore carousel beta opt-in, suppressed when enabled sitewide (T429414)]], [[gerrit:1305741|Roll back mobile image carousel from sitewide to beta opt-in (T429414)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2430917 wikitext text/x-wiki thme1u6a82srkeej0vc0jfb4wjd2vyt 2430918 2430917 2026-06-25T21:35:34Z Stashbot 7414 jdlrobson@deploy1003: egardner, jdlrobson: Continuing with deployment 2430918 wikitext text/x-wiki im0bou8iy4s4z8fcdfac9q03f8lvfp0 2430920 2430918 2026-06-25T21:47:43Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305749|Restore carousel beta opt-in, suppressed when enabled sitewide (T429414)]], [[gerrit:1305741|Roll back mobile image carousel from sitewide to beta opt-in (T429414)]] (duration: 32m 28s) 2430920 wikitext text/x-wiki nnhreeglg3zdwf0mv8wp2epanubfgw7 2430921 2430920 2026-06-25T21:56:54Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T426862 - bking@cumin2003 2430921 wikitext text/x-wiki 5fdbfxzmtz7rsk39fnb9e008uvleokb 2430922 2430921 2026-06-25T23:31:25Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart 2430922 wikitext text/x-wiki 56slf8m4aiq4gbd3r3mcm73b4mv9lbk 2430923 2430922 2026-06-25T23:31:46Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart 2430923 wikitext text/x-wiki sy2tzsqdusy3ys48bmcwqdkyvgyr7fn 2430925 2430923 2026-06-25T23:41:53Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) 2430925 wikitext text/x-wiki 6e9g1mntfnjz0r554f10bjnen0gh8pf 2430926 2430925 2026-06-25T23:56:19Z Stashbot 7414 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) 2430926 wikitext text/x-wiki 2n648x0nn46uu0a563iu7oi01l60djk 2430928 2430926 2026-06-26T00:17:18Z Stashbot 7414 ryankemper: T429919 Followed the steps in https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater#Clean_up_object_storage to clear out stale checkpoints 2430928 wikitext text/x-wiki 8uwxf2nr3f9hnl87hjdjgqlehpa6h1h 2430931 2430928 2026-06-26T02:00:44Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2430931 wikitext text/x-wiki 97upqvey7th19lyg0fbqum6gbtmriam 2430934 2430931 2026-06-26T02:07:53Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 09s) 2430934 wikitext text/x-wiki 1kgghwkcem4mvqyf235wz6h35az9co8 2430951 2430934 2026-06-26T04:08:31Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2430951 wikitext text/x-wiki 75qirk93l2jrxagu9mp1wjbwcfa47my 2430952 2430951 2026-06-26T04:08:59Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2430952 wikitext text/x-wiki qc3xe28ehuv2m1t9qemo9jifrdlhvei 2430953 2430952 2026-06-26T05:08:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2234.codfw.wmnet with OS trixie 2430953 wikitext text/x-wiki 901y38c40f9cvllighsumibxje22x2q 2430954 2430953 2026-06-26T05:23:52Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2234.codfw.wmnet with reason: host reimage 2430954 wikitext text/x-wiki 7uasybnf2d4dwdz0fkp35x3xtorkvl5 2430955 2430954 2026-06-26T05:28:56Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2234.codfw.wmnet with reason: host reimage 2430955 wikitext text/x-wiki s2tuimvyzjzbcmov02snr2gchiy9et4 2430959 2430955 2026-06-26T05:51:55Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2234.codfw.wmnet with OS trixie 2430959 wikitext text/x-wiki k0dprvbpvh0bnacwxdhybsi40lwvox8 2430964 2430959 2026-06-26T07:38:49Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.hosts.reboot-unattended Sequential unattended reboot of 6 host(s) [team=collaboration-services, os=bookworm] 2430964 wikitext text/x-wiki 99x2i9owg013m86rfeu59o8m3yoxb7e 2430965 2430964 2026-06-26T07:39:04Z Stashbot 7414 jelto@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-unattended (exit_code=99) Sequential unattended reboot of 6 host(s) [team=collaboration-services, os=bookworm] 2430965 wikitext text/x-wiki o1h7bso1p82lzkwrhma6lf85hohpycf 2430969 2430965 2026-06-26T09:05:23Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database isvwiki (T429938) 2430969 wikitext text/x-wiki 9nc88i5st6fyza89pj4tlanio101kfo 2430970 2430969 2026-06-26T09:05:34Z Stashbot 7414 fnegri@cumin1003: END (FAIL) - Cookbook sre.wikireplicas.add-wiki (exit_code=99) for database isvwiki (T429938) 2430970 wikitext text/x-wiki 86tbwxyxdh425cttvon8s2eg48s5lx2 2430971 2430970 2026-06-26T09:09:24Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database isvwiki (T429938) 2430971 wikitext text/x-wiki trumbftntsqnb82dngl2w9vb5kiee8a 2430972 2430971 2026-06-26T09:09:35Z Stashbot 7414 fnegri@cumin1003: END (FAIL) - Cookbook sre.wikireplicas.add-wiki (exit_code=99) for database isvwiki (T429938) 2430972 wikitext text/x-wiki 84ge8qev3sayzbxj13jrzvdapfxmquk 2430973 2430972 2026-06-26T09:15:34Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database isvwiki (T429938) 2430973 wikitext text/x-wiki 6zp4exud0n718nyxyuh9quaay4211it 2430975 2430973 2026-06-26T09:42:43Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database magwiki (T428282) 2430975 wikitext text/x-wiki jgs3unigf55v1x6ess3cs7hu25p776l 2430976 2430975 2026-06-26T09:42:55Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database magwiki (T428282) 2430976 wikitext text/x-wiki jjmqdsuepxyxaww60iie1l0q4hsmk1x 2430977 2430976 2026-06-26T10:12:56Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database isvwiki (T429938) 2430977 wikitext text/x-wiki hfumq3bisvrwbumr1z1bu7ecmt8fb7u 2430978 2430977 2026-06-26T10:40:08Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.dns.netbox 2430978 wikitext text/x-wiki bblehsftg7nsghlvvrjpqda2j223ddm 2430979 2430978 2026-06-26T10:44:24Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new IPs for cloudvirt1078-80 - filippo@cumin1003" 2430979 wikitext text/x-wiki ttmoimvgemau1enbiw1soq26rc2rtal 2430980 2430979 2026-06-26T10:44:32Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new IPs for cloudvirt1078-80 - filippo@cumin1003" 2430980 wikitext text/x-wiki ste8s44q7w5blu693uw6jc135jlvluw 2430981 2430980 2026-06-26T10:44:35Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2430981 wikitext text/x-wiki hxn18oj1au92n0ab0hxsdk3a90lsf8s 2430982 2430981 2026-06-26T10:45:06Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1078.eqiad.wmnet with OS trixie 2430982 wikitext text/x-wiki 886ahb2y0mrx7fi1kdow8llh5ptgbyw 2430983 2430982 2026-06-26T10:45:27Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1079.eqiad.wmnet with OS trixie 2430983 wikitext text/x-wiki okpgnvhngm5rit2btmt3q1q748m9wpb 2430984 2430983 2026-06-26T10:45:48Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1080.eqiad.wmnet with OS trixie 2430984 wikitext text/x-wiki 24je7ux9g36lgor70ba4mvwdacj5k5x 2430985 2430984 2026-06-26T10:53:00Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2430985 wikitext text/x-wiki md6qrkp4de402wo5mtstx91a1l8nb55 2430986 2430985 2026-06-26T10:55:02Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2430986 wikitext text/x-wiki 36w1vg6quppv94cwvcdyoqvat83c80a 2430987 2430986 2026-06-26T10:56:47Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1078.eqiad.wmnet with reason: host reimage 2430987 wikitext text/x-wiki 37q1h6l4v9b5zklhrqhpeqg4ry9uuq8 2430988 2430987 2026-06-26T10:57:03Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1079.eqiad.wmnet with reason: host reimage 2430988 wikitext text/x-wiki 5l824byx3bpihhhc6kn4kwyhkwh97ew 2430989 2430988 2026-06-26T10:57:14Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1080.eqiad.wmnet with reason: host reimage 2430989 wikitext text/x-wiki iyzjqoh6zqy60bqnkv56yex8dyudw4x 2430990 2430989 2026-06-26T11:04:01Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1078.eqiad.wmnet with reason: host reimage 2430990 wikitext text/x-wiki 415iz2hz5x5joc8rdzekpm07nkv8byh 2430991 2430990 2026-06-26T11:09:11Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1079.eqiad.wmnet with reason: host reimage 2430991 wikitext text/x-wiki p7t6ak67r3zsvekx918crryap249gwy 2430992 2430991 2026-06-26T11:13:53Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1080.eqiad.wmnet with reason: host reimage 2430992 wikitext text/x-wiki o4to1hs1uhlnwhtu8h3jweftr4p4d4v 2430993 2430992 2026-06-26T11:18:42Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1078.eqiad.wmnet with OS trixie 2430993 wikitext text/x-wiki lnazch068aa9sw8riaf9jj0oseji8qj 2430994 2430993 2026-06-26T11:23:56Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2430994 wikitext text/x-wiki ihq815vafs09eeclb7m6f35mgmx9fwt 2430995 2430994 2026-06-26T11:24:24Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1079.eqiad.wmnet with OS trixie 2430995 wikitext text/x-wiki iygrubp2fzn7cxumnibcfhkg0yy3que 2430996 2430995 2026-06-26T11:25:58Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2430996 wikitext text/x-wiki r8r28twxp0pzy3sdeqhcvvt4snogm61 2430997 2430996 2026-06-26T11:26:03Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2430997 wikitext text/x-wiki 62mv8d7ysmdnpajbk8tpv6rbgyyvr04 2430998 2430997 2026-06-26T11:30:53Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1080.eqiad.wmnet with OS trixie 2430998 wikitext text/x-wiki rnxqidehdvfpq2xlrftnp6uje82172d 2430999 2430998 2026-06-26T11:32:05Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: sync 2430999 wikitext text/x-wiki j8otn2vccaqngkudpgah0kvow0j3vid 2431000 2430999 2026-06-26T11:32:22Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: sync 2431000 wikitext text/x-wiki pdo284pkwv1f65yjtv5vgo2wi7wxqtc 2431001 2431000 2026-06-26T11:32:46Z Stashbot 7414 _joe_: rolling restart of zotero in eqiad T430279 2431001 wikitext text/x-wiki m8gcv2riybc6hj2qhyrlbb6i9a00may 2431002 2431001 2026-06-26T11:42:34Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: sync 2431002 wikitext text/x-wiki 07qqg6c9uov94zojltjw9ho0bibugjl 2431003 2431002 2026-06-26T11:42:48Z Stashbot 7414 _joe_: rolling restart of citoid in eqiad T430279 2431003 wikitext text/x-wiki 1qbskmvxm8ihr1sh4n0ngihyigcfy6b 2431004 2431003 2026-06-26T11:42:52Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2431004 wikitext text/x-wiki ch5rszqphqdyz8giwwt6siynjtost8p 2431005 2431004 2026-06-26T11:43:01Z Stashbot 7414 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: sync 2431005 wikitext text/x-wiki r1liimv1wh5yz1v579vrc1d9wloi419 2431006 2431005 2026-06-26T11:44:52Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2431006 wikitext text/x-wiki rsqcv9cq9earh1eclz17eif1gomrlwq 498 9925 2430700 2428736 2026-06-25T14:25:32Z Stashbot 7414 inflatador: delete unused servers deployment-cirrussearch1[2-4] T425585 2430700 wikitext text/x-wiki 913dadmbj01avtmm182uoiy016z8xts 2430788 2430700 2026-06-25T16:50:24Z Stashbot 7414 inflatador: add 60GB cinder vol to deployment-cirrussearch15 T425585 2430788 wikitext text/x-wiki ory79ai1ilc98ms9o22yoxcelrohq6c 0 17290 2430699 2430599 2026-06-25T14:25:30Z Stashbot 7414 inflatador: delete unused servers deployment-cirrussearch1[2-4] T425585 2430699 wikitext text/x-wiki 02q4v6wlwnmmi1aq34b0pasidvydx6x 2430787 2430699 2026-06-25T16:50:21Z Stashbot 7414 inflatador: add 60GB cinder vol to deployment-cirrussearch15 T425585 2430787 wikitext text/x-wiki 7tjo8qrazff607fo54u5wz7g9h6bic4 2430903 2430787 2026-06-25T20:49:58Z Stashbot 7414 dancy: Investigating castor-save-workspace-cache clog 2430903 wikitext text/x-wiki 7hawj57m3i2njfjkia1epkrf997ax0r 2430907 2430903 2026-06-25T20:57:50Z Stashbot 7414 dancy: Restarting Jenkins to unstick builds 2430907 wikitext text/x-wiki oraqqpsh8ndb3wz9crn7b6ooxtin7p7 0 20832 2430932 2250071 2026-06-26T02:06:37Z Star walker 40724 Tidied up article 2430932 wikitext text/x-wiki i5cokl6nwg47c9tasg8lhha3pz7nyxh 10 442718 2430929 2377462 2026-06-26T00:47:26Z Star walker 40724 Tidied up template 2430929 wikitext text/x-wiki cjou4qaqrxhn0bklxw2jqpg1ti93fjf 498 442831 2430758 2404325 2026-06-25T16:28:31Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2430758 wikitext text/x-wiki pcohsf0xcx7o4uk4grmlbaye7j9ii35 2430759 2430758 2026-06-25T16:28:46Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) 2430759 wikitext text/x-wiki 8a5ybmcjyfm9irl9fpo26c586l9nzbw 2430760 2430759 2026-06-25T16:28:49Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2430760 wikitext text/x-wiki te4esfk10n4jx2616mvnvjejyq23jov 2430762 2430760 2026-06-25T16:29:06Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) 2430762 wikitext text/x-wiki ci3syk5i76ha13mydi51yo2dprth1jn 2430763 2430762 2026-06-25T16:29:23Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2430763 wikitext text/x-wiki iczu3adg3wkgh6eocsg8oaqn451mnqy 2430764 2430763 2026-06-25T16:29:41Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) 2430764 wikitext text/x-wiki iinig80mkougoyg40l6m88vq9fdqlap 2430765 2430764 2026-06-25T16:30:49Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.nfs.add_server 2430765 wikitext text/x-wiki q5m7flvnvonars66zopj8z594dvpmup 2430769 2430765 2026-06-25T16:39:03Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) 2430769 wikitext text/x-wiki mtf02q71oy1th9bbribkfaah4ivn0cq 498 443947 2430712 2424267 2026-06-25T14:53:36Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed b3412fb959 (l10n updates: cs) 2430712 wikitext text/x-wiki n3vbfjlbbdt0rl6ehv2xpugjdzifkf3 0 445037 2430930 2380589 2026-06-26T01:47:22Z Star walker 40724 interlink 2430930 wikitext text/x-wiki l66sdy6yszi6vn8o20etyzdsfuvn3tk 0 446030 2430919 2408269 2026-06-25T21:45:51Z RKemper (WMF) 17058 /* Clean up object storage */ Alert was changed a year ago 2430919 wikitext text/x-wiki 5jczjhawx41qc2ew7jpx56hbb1mlenb 12 449113 2430795 2428559 2026-06-25T17:02:34Z FNegri-WMF 32595 Reverted edit by [[Special:Contributions/Wbm1058|Wbm1058]] ([[User talk:Wbm1058|talk]]) to last revision by [[User:SSapaty (WMF)|SSapaty (WMF)]] 2414347 wikitext text/x-wiki atq95y9iu3vjoyhuq2rk0obo0mmbp4m 0 449160 2430927 2430495 2026-06-26T00:02:17Z Dexbot 30554 Bot: Updating the report 2430927 wikitext text/x-wiki qxr7kz2o39s88urnwb8wbg353gl8w7u 0 452153 2430733 2428609 2026-06-25T15:53:45Z DCaro (WMF) 18271 Updated some of the new services with who I think should own them, and updated some of the support levels to things we rely on platform-wise to have three stars. 2430733 wikitext text/x-wiki 6k03a5tfhfrmpp9bxv72bcod5b58mf1 2430736 2430733 2026-06-25T15:59:59Z DCaro (WMF) 18271 Added audience, as in who is the user of that service, with an initial estimation of the values 2430736 wikitext text/x-wiki 8hxzragxbereuidge9v7ergtzn6zxeb 116 453906 2430740 2430513 2026-06-25T16:09:23Z Gitlabaccountapprovalbot 37332 @sakuraemad was approved. 2430740 wikitext text/x-wiki qcjixt2zuebfxl0y4lzqwcpcjbzrqz0 2430957 2430740 2026-06-26T05:51:16Z Gitlabaccountapprovalbot 37332 @govindlaltl was approved. 2430957 wikitext text/x-wiki afjg5kh6xj3azj47wigvyxjo0di2afm 2430958 2430957 2026-06-26T05:51:21Z Gitlabaccountapprovalbot 37332 @arpithhhaaa was approved. 2430958 wikitext text/x-wiki n3bnv1hjt4h2q5h9llwa5vpym1qtj7m 2430961 2430958 2026-06-26T07:30:31Z Gitlabaccountapprovalbot 37332 samsaesque was rejected. 2430961 wikitext text/x-wiki 74x3pui90r3u2d1mz292jdnmqr25pnl 2430962 2430961 2026-06-26T07:30:43Z Gitlabaccountapprovalbot 37332 @anushka10patel was approved. 2430962 wikitext text/x-wiki j7csv2wdqe85ovca90y1iu5ugc17da9 2430963 2430962 2026-06-26T07:30:44Z Gitlabaccountapprovalbot 37332 @reeti was approved. 2430963 wikitext text/x-wiki 5rlc4o4nl70f37js027t2idmsae060h 2430974 2430963 2026-06-26T09:39:23Z Gitlabaccountapprovalbot 37332 @lakshita28 was approved. 2430974 wikitext text/x-wiki k3ott54s2eskxuxia9vvshv7a8yamml 0 455989 2430802 2429667 2026-06-25T17:13:25Z JVanderhoop-WMF 41069 Added measurement plan link 2430802 wikitext text/x-wiki 3k0mn194rsbn0auk4f56c4twb961x63 498 459165 2430967 2396304 2026-06-26T08:42:28Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 18 volumes 2430967 wikitext text/x-wiki 8k615soi3gg83os6j6rqty9ddamo2p2 2430968 2430967 2026-06-26T08:42:34Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 18 volumes 2430968 wikitext text/x-wiki 9auaf3pzw50kgv10ijekxj6ffa52j52 498 459359 2430946 2357141 2026-06-26T02:27:28Z Krinkle 44 Changed redirect target from [[Obsolete:Deployment-prep/Documentation]] to [[Nova Resource:Deployment-prep]] 2430946 wikitext text/x-wiki pw0h08ttlqyw0vgolze86v16ksxi2so 0 459699 2430675 2430368 2026-06-25T13:35:23Z Stashbot 7414 jgleeson: payments-wiki upgraded from ab4fd16b to c2cdbf24 2430675 wikitext text/x-wiki 5f3fqik4mys4vqhgtqgpyq46s0jw9mv 2430966 2430675 2026-06-26T08:17:40Z Stashbot 7414 eileen: civicrm upgraded from a90a5451 to ebab84c8 2430966 wikitext text/x-wiki q4jox14veit88bwioay5srm9fhek0ub 0 460232 2430800 2429913 2026-06-25T17:09:29Z MPopov (WMF) 5891 /* View and understand experiment results */ Section on interpreting paradoxical results 2430800 wikitext text/x-wiki jptudvrjgdzdhod58cyzflsv5ayafud 2430801 2430800 2026-06-25T17:11:25Z MPopov (WMF) 5891 /* Interpret paradoxical results */ 2430801 wikitext text/x-wiki id4ehl3pknla0hddumibo0brmx67vbc 0 460234 2430842 2429915 2026-06-25T18:32:07Z MPopov (WMF) 5891 Remove draft notice 2430842 wikitext text/x-wiki bf42ykhko8hjexxl05upljkfqa7ejze 0 460336 2430924 2426754 2026-06-25T23:34:00Z Nemoralis 21869 add copy buttons 2430924 wikitext text/x-wiki 5hkca8eyn37g08mzqwz184jb6eht7q5 2430960 2430924 2026-06-26T06:33:24Z ABran-WMF 37196 2430960 wikitext text/x-wiki bmxfbuyxramwsj07gs2q4f5bax6871e 0 460351 2430816 2428343 2026-06-25T17:41:32Z BKing (WMF) 29017 2430816 wikitext text/x-wiki k97e7do4pec9b3y51vtdf88weq3kchh 3 460381 2430738 2026-06-25T16:07:49Z StrikerBot 8475 Welcome to Toolforge! 2430738 wikitext text/x-wiki jjo45vaqscucrjqixrptf0ppsjp26w7 3 460382 2430739 2026-06-25T16:08:51Z StrikerBot 8475 Welcome to Toolforge! 2430739 wikitext text/x-wiki ppfx8qafare2t6a6xytc3g5kfnhsy85