Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.22 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 845 2461956 2456819 2026-09-29T21:59:34Z Southparkfan 1843 2461956 wikitext text/x-wiki jlfg5rebk66esz2602r3s9rdglhfwbk 498 3086 2462030 2461626 2026-09-30T08:51:20Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2462030 wikitext text/x-wiki pzku2k30cxksaasziwj7t6sslxmbh35 2462032 2462030 2026-09-30T08:55:05Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2462032 wikitext text/x-wiki 7d48g04ys9203ekxpg15nvg0vdp24kq 2462033 2462032 2026-09-30T09:00:14Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2462033 wikitext text/x-wiki 53uzhnvkfbufn9hhe7t9m5tm79tclkp 2462035 2462033 2026-09-30T09:02:49Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2462035 wikitext text/x-wiki 37ucwtigu0u4s2h9aorbg9cm6szgxrv 2462043 2462035 2026-09-30T09:31:18Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api (T432865) 2462043 wikitext text/x-wiki 5jesu15i2epflaxxqgxdyhxxjlup8sn 2462044 2462043 2026-09-30T09:32:14Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api (T432865) 2462044 wikitext text/x-wiki 4s8h74p2hg3pu6al5ytlwl21apb1amx 2462049 2462044 2026-09-30T09:39:17Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2462049 wikitext text/x-wiki i4mvpmenxuv9krteloybglhzezx65c4 2462051 2462049 2026-09-30T09:42:02Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2462051 wikitext text/x-wiki 9nrwug2f86exsqcjykursmy4306o485 2462052 2462051 2026-09-30T09:46:31Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2462052 wikitext text/x-wiki jn8gxu9gdfo6bdad4t1wzjiklrsdlhd 2462053 2462052 2026-09-30T10:01:20Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2462053 wikitext text/x-wiki rr36vi8wej9olcmeued62fdiqudh3y1 0 4108 2461687 2461662 2026-09-29T12:59:24Z ScheduleDeploymentBot 37566 Add [[gerrit:1345999]] to Tuesday, September 29 UTC afternoon backport window 2461687 wikitext text/x-wiki cuohmggpwu6y85x3qszyf2fduo13j45 2461689 2461687 2026-09-29T13:02:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1345216]] to Tuesday, September 29 UTC afternoon backport window 2461689 wikitext text/x-wiki 0bioi8lspu0femnav79eo8an3v639tl 2461758 2461689 2026-09-29T16:14:11Z ScheduleDeploymentBot 37566 Add [[gerrit:1346060]] to Wednesday, September 30 UTC afternoon backport window 2461758 wikitext text/x-wiki 44ay1wey1v39d0kyf3xp9zl8betqzvu 2461770 2461758 2026-09-29T16:29:23Z ScheduleDeploymentBot 37566 Add [[gerrit:1346012]] to Tuesday, September 29 UTC late backport window 2461770 wikitext text/x-wiki thk1x77annw3d5nm42omeahdwcl36k0 2461771 2461770 2026-09-29T16:31:03Z DCausse (WMF) 5604 /* {{Deployment_day|date=2026-09-29}} */ 2461771 wikitext text/x-wiki 44ay1wey1v39d0kyf3xp9zl8betqzvu 2461772 2461771 2026-09-29T16:31:31Z ScheduleDeploymentBot 37566 Add [[gerrit:1346012]] to Wednesday, September 30 UTC morning backport window 2461772 wikitext text/x-wiki i7mcge7zel432cupgqn1agsx16vp03q 2461792 2461772 2026-09-29T17:46:31Z ScheduleDeploymentBot 37566 Add [[gerrit:1346087]] to Tuesday, September 29 UTC late backport window 2461792 wikitext text/x-wiki glsxqipq9zxjxr57p7qk5o51ojr64l5 2462037 2461792 2026-09-30T09:05:51Z ScheduleDeploymentBot 37566 Add [[gerrit:1345584]] to Wednesday, September 30 UTC afternoon backport window 2462037 wikitext text/x-wiki sockw5yi7masjfcxy3gxub8i856cz4c 2462058 2462037 2026-09-30T10:19:53Z SLyngshede-WMF 31501 /* Week of October 05 */ 2462058 wikitext text/x-wiki mz9gdyj5e9rckmgu10ue04cy4v0w8dh 498 6726 2461756 2461623 2026-09-29T16:03:50Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2461756 wikitext text/x-wiki q9fd8zjhr9apqu8yqbzawn0o2ugeo38 2461757 2461756 2026-09-29T16:13:07Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2461757 wikitext text/x-wiki tiw47387rk7sabeba61lp7hzsmy9gnl 2461780 2461757 2026-09-29T16:54:46Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2461780 wikitext text/x-wiki rsw2zbzp7o2xaawa8hksfdrijnyrbjh 2461787 2461780 2026-09-29T17:10:42Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2461787 wikitext text/x-wiki 3dbiokjf9macuvk6sec9thtvtkmjlqt 2462016 2461787 2026-09-30T07:58:04Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2462016 wikitext text/x-wiki r1icslhk6h769usud20dybf0f9tprja 2462018 2462016 2026-09-30T08:07:19Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2462018 wikitext text/x-wiki 9l13skdi1tgfohdrah2yuqaqxnhlym0 2462023 2462018 2026-09-30T08:27:24Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2462023 wikitext text/x-wiki 080e17sg9f6cpgjdemt72pl8cwcvcwe 2462025 2462023 2026-09-30T08:27:47Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2462025 wikitext text/x-wiki 3k4lg5zlz6gl5dljz4n98ssrdypfzu8 2462026 2462025 2026-09-30T08:29:11Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2462026 wikitext text/x-wiki 8hdqwcdyy02pvpmg52x8hnc7uctekab 2462027 2462026 2026-09-30T08:29:31Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2462027 wikitext text/x-wiki 2iv11ts0x3td1mpnqd2a5f3vbj5rosv 2462028 2462027 2026-09-30T08:30:55Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2462028 wikitext text/x-wiki szd8wyjccf7k385sssc5b0yb1errhg2 2462029 2462028 2026-09-30T08:44:18Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2462029 wikitext text/x-wiki kfbdtfthwdzgpxblh47v5lgtx7wo2q3 2462040 2462029 2026-09-30T09:11:04Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api (T432865) 2462040 wikitext text/x-wiki 6se24dcn88sehctztgnd9ezmcpeldmi 2462041 2462040 2026-09-30T09:12:07Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api (T432865) 2462041 wikitext text/x-wiki 1fvdc24z5rwz1jwok9abjolx2fwk6z8 0 7919 2461665 2461660 2026-09-29T12:02:48Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2461665 wikitext text/x-wiki oph1xeql0r7nowiy1hh9pan1wwqkpbb 2461666 2461665 2026-09-29T12:02:59Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461666 wikitext text/x-wiki 5z84sftboxufisnpwvlp5n292zejec6 2461667 2461666 2026-09-29T12:03:15Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2461667 wikitext text/x-wiki 706nmef5uwkv5jr42vct0q2u1ymldad 2461668 2461667 2026-09-29T12:03:16Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2461668 wikitext text/x-wiki 16c0lpx189vqmnbfz89iqflbehudlbl 2461669 2461668 2026-09-29T12:03:48Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2461669 wikitext text/x-wiki 3feh26a9d4fnoy1ubqz7xq9kc97f6s4 2461670 2461669 2026-09-29T12:06:47Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2461670 wikitext text/x-wiki idr9ajr1pyhv2ouq02vl9frlbrkre12 2461671 2461670 2026-09-29T12:06:50Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2461671 wikitext text/x-wiki k9yhpv1jnxmjqbze4vanfvwv8vlhoii 2461672 2461671 2026-09-29T12:06:50Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2461672 wikitext text/x-wiki 40e1aa8yvykanvqsrquft1zlxurdjar 2461673 2461672 2026-09-29T12:06:54Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2461673 wikitext text/x-wiki rp9fq6lfwostlmxdkum5vw9p9wwr4so 2461674 2461673 2026-09-29T12:27:25Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2461674 wikitext text/x-wiki 2c8akwzo1vlca1tanvny2fz65jrw4cr 2461675 2461674 2026-09-29T12:28:23Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2461675 wikitext text/x-wiki ax2elvyjsg9rr8a45zvhu868klwunv7 2461676 2461675 2026-09-29T12:30:46Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2461676 wikitext text/x-wiki nl27yvxm8l3ze4bc92dfd89yhanv3t6 2461677 2461676 2026-09-29T12:31:30Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2461677 wikitext text/x-wiki pdaigu1g64ioje8f70dcktite9xsft4 2461678 2461677 2026-09-29T12:39:41Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2461678 wikitext text/x-wiki 09rn2ci38xh28w71nutatqsr7re6sd9 2461679 2461678 2026-09-29T12:40:09Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1345986|ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] 2461679 wikitext text/x-wiki ewpdeyr5uqyo22ysx067mya6ty2yp01 2461681 2461679 2026-09-29T12:44:18Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1345986|ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461681 wikitext text/x-wiki lpe9fp21i5fb4oth71h8fih83ahakkk 2461682 2461681 2026-09-29T12:47:11Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Continuing with deployment 2461682 wikitext text/x-wiki njtbc7b9eavkh49q74e3a4adkc8y5s0 2461683 2461682 2026-09-29T12:52:35Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345986|ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] (duration: 12m 26s) 2461683 wikitext text/x-wiki 9z717jovo4dveda1029hrlhmm2ho2ho 2461684 2461683 2026-09-29T12:53:38Z Stashbot 7414 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging 2461684 wikitext text/x-wiki 9arhr8nfv058kjt2dvby1bnla0abd2p 2461685 2461684 2026-09-29T12:54:05Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2461685 wikitext text/x-wiki eh56pcgd1zh2huy8tbzjdp5vmgyrvg9 2461686 2461685 2026-09-29T12:54:52Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2461686 wikitext text/x-wiki 5qnu1wrj0l6n3ji2vkulxlno5l8q78v 2461688 2461686 2026-09-29T13:00:27Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343957|InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] 2461688 wikitext text/x-wiki i4kuksv1ry9rvobe19ti2ymn4jdb5h0 2461690 2461688 2026-09-29T13:04:58Z Stashbot 7414 samtar@deploy1003: samtar: Backport for [[gerrit:1343957|InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461690 wikitext text/x-wiki 9vhln9inqhd5kim7ef259j4m5szdm0j 2461691 2461690 2026-09-29T13:05:30Z Stashbot 7414 samtar@deploy1003: samtar: Continuing with deployment 2461691 wikitext text/x-wiki i8rsojshkmcu37gn7feoedirevw7l9g 2461692 2461691 2026-09-29T13:10:47Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343957|InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] (duration: 10m 21s) 2461692 wikitext text/x-wiki fbobc4tqjgfke9xz7x74ebjcwc8hfbf 2461693 2461692 2026-09-29T13:14:32Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343958|CommonSettings: Load ModeratorToolkit extension (T438334)]] 2461693 wikitext text/x-wiki 258ki3h6eia0cp2datou1n5wlwrhlfs 2461694 2461693 2026-09-29T13:15:38Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2461694 wikitext text/x-wiki cwlo4228r05bant5bl2sxescdnpx0pj 2461695 2461694 2026-09-29T13:15:47Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2461695 wikitext text/x-wiki i6ii8cbvrfhhfxcde2w6ao7m1xn1sjq 2461696 2461695 2026-09-29T13:15:52Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2461696 wikitext text/x-wiki k0upk1szrq6pxeu18rubkbeossnk3gk 2461697 2461696 2026-09-29T13:15:57Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2461697 wikitext text/x-wiki 0atw4zihhg63a05854wm19mabobzv3c 2461698 2461697 2026-09-29T13:16:23Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2461698 wikitext text/x-wiki ogbyls02c0yi31nxj6hlki7ap3pwe77 2461699 2461698 2026-09-29T13:16:45Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2461699 wikitext text/x-wiki mfpi2robdt9fwwlkhz2a6qf9st9gil1 2461700 2461699 2026-09-29T13:19:12Z Stashbot 7414 samtar@deploy1003: samtar: Backport for [[gerrit:1343958|CommonSettings: Load ModeratorToolkit extension (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461700 wikitext text/x-wiki 16epo5yetu9ketyte88wbuc7tzc6c3v 2461701 2461700 2026-09-29T13:20:33Z Stashbot 7414 samtar@deploy1003: samtar: Continuing with deployment 2461701 wikitext text/x-wiki 7qp2blnq812jgggqd0x5vbqs96khw1w 2461702 2461701 2026-09-29T13:21:10Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461702 wikitext text/x-wiki 4el9d2tx11385sjmntph65sh468flae 2461703 2461702 2026-09-29T13:25:50Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343958|CommonSettings: Load ModeratorToolkit extension (T438334)]] (duration: 11m 17s) 2461703 wikitext text/x-wiki pkv0kx6n2zkdrttoyeysjb2lod3yyiy 2461704 2461703 2026-09-29T13:30:17Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2461704 wikitext text/x-wiki 8kaqjgywogjxiwvonn5xv4318e0h44h 2461705 2461704 2026-09-29T13:30:40Z Stashbot 7414 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2461705 wikitext text/x-wiki kacxz704u3h182dxjnfwj32su46wj1j 2461706 2461705 2026-09-29T13:34:28Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345952|fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953|fix: prevent AccountSetup from being scrollable (T436872)]] 2461706 wikitext text/x-wiki e591253hdgv93eex4pdr778klq3cruf 2461707 2461706 2026-09-29T13:38:25Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1345952|fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953|fix: prevent AccountSetup from being scrollable (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461707 wikitext text/x-wiki fvtir1gggorbwzdmem2m6pd0mnpbp1c 2461708 2461707 2026-09-29T13:40:49Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment 2461708 wikitext text/x-wiki crwdh3l6e4g8bbrkrc1c6qlil86tt7f 2461710 2461708 2026-09-29T13:46:07Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345952|fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953|fix: prevent AccountSetup from being scrollable (T436872)]] (duration: 11m 39s) 2461710 wikitext text/x-wiki m9gpx42y6epi9ffuk9f1i0mg3mqdk60 2461711 2461710 2026-09-29T13:47:18Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461711 wikitext text/x-wiki bdlm04ni5qc5diyyedf960zvl749hkv 2461712 2461711 2026-09-29T13:47:42Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie 2461712 wikitext text/x-wiki o8dacoen60a0h2om5unm06iqh5cwxrt 2461713 2461712 2026-09-29T13:47:43Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet 2461713 wikitext text/x-wiki acfgv9azl1eanyc2sb8ofg78s5xkde7 2461714 2461713 2026-09-29T13:51:08Z Stashbot 7414 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: T331941 2461714 wikitext text/x-wiki ldoax4v4tu049pjezvwh73orqxi3ia2 2461715 2461714 2026-09-29T13:51:16Z Stashbot 7414 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: T331941 (duration: 00m 09s) 2461715 wikitext text/x-wiki 0sz3dypmevkuqe1a4tboogetivrr4bf 2461716 2461715 2026-09-29T13:54:56Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461716 wikitext text/x-wiki 5aue1do87a6m310bdkrmmhhim4kqh3k 2461717 2461716 2026-09-29T13:56:46Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1345216|File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298|ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] 2461717 wikitext text/x-wiki gho41xmgaxkvl72byb707qktzyqqavc 2461718 2461717 2026-09-29T13:58:46Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2461718 wikitext text/x-wiki 6kd9cwe81q4m4xxnlzdnw7db8qicqe8 2461720 2461718 2026-09-29T13:59:08Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary 2461720 wikitext text/x-wiki 0obuxgsr6n3arrtvz1y49a3rtc1pug9 2461721 2461720 2026-09-29T14:00:56Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1345216|File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298|ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461721 wikitext text/x-wiki 1ekgl5ufx64kx4fm446qr6qdcsjvbav 2461722 2461721 2026-09-29T14:02:11Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.dns.netbox 2461722 wikitext text/x-wiki isisfuianafxvvs56ormc3yk18hfztl 2461723 2461722 2026-09-29T14:05:49Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2461723 wikitext text/x-wiki de0dyg4cdm8ma1uv9jjxw659o1607i7 2461724 2461723 2026-09-29T14:06:05Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" 2461724 wikitext text/x-wiki 51i1mpdg52kgtuoanv3irhc2nzq369j 2461727 2461724 2026-09-29T14:07:09Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" 2461727 wikitext text/x-wiki rs7b27qewi8sdlz3adbbaqpkw56j10x 2461728 2461727 2026-09-29T14:07:10Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2461728 wikitext text/x-wiki e8k6jvcsetnlhk4xcxvwx1f8tfh3vhg 2461729 2461728 2026-09-29T14:07:56Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461729 wikitext text/x-wiki s0vf1ehdzjj01s7ynrfwa8z0lxd5njg 2461730 2461729 2026-09-29T14:11:11Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345216|File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298|ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] (duration: 14m 26s) 2461730 wikitext text/x-wiki ry57338r8l8yzhld7e6brsyu9jn68iq 2461731 2461730 2026-09-29T14:12:17Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461731 wikitext text/x-wiki 74d8ac1ysfdt7s4uezr7zz80glgkkrc 2461732 2461731 2026-09-29T14:12:31Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw 2461732 wikitext text/x-wiki 3o5qyt2nbse76064lzvz5bxr8bwhilw 2461733 2461732 2026-09-29T14:16:11Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet 2461733 wikitext text/x-wiki actqucm4ra4iodu3b85hqx3xzi42de0 2461734 2461733 2026-09-29T14:16:12Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet 2461734 wikitext text/x-wiki goth2t31pijkxa1wflwb9dht70bjkny 2461735 2461734 2026-09-29T14:19:34Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie 2461735 wikitext text/x-wiki 3tcn2xuymw53vjzw1l20yicsrncy6c6 2461736 2461735 2026-09-29T14:21:57Z Stashbot 7414 moritzm: installing rsyslog security updates 2461736 wikitext text/x-wiki is93a3gyqyme7mmau23kah1hf3yaaok 2461737 2461736 2026-09-29T14:21:59Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox 2461737 wikitext text/x-wiki 6pgg2cqvjb6pn1v6kttu7wbavufc8pn 2461738 2461737 2026-09-29T14:23:06Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox 2461738 wikitext text/x-wiki ju3oo0y0dbz141v255fes6kxn1k0nbo 2461739 2461738 2026-09-29T14:32:46Z Stashbot 7414 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues 2461739 wikitext text/x-wiki akhqzltfcnumputl2b5bz5xneec3z7c 2461743 2461739 2026-09-29T15:08:24Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2461743 wikitext text/x-wiki ajbinfiauxyb6oyldubr4pocpidr06t 2461744 2461743 2026-09-29T15:08:58Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2461744 wikitext text/x-wiki q7jxgcxlwfpmorfegjukrh7743t2lgs 2461745 2461744 2026-09-29T15:09:04Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage 2461745 wikitext text/x-wiki nh4yov0ndy9o25qalsi1k7uj9mm4mg5 2461747 2461745 2026-09-29T15:14:27Z Stashbot 7414 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage 2461747 wikitext text/x-wiki cwrvy04w90vbmnywmu5q5lhe1tmagyi 2461749 2461747 2026-09-29T15:23:25Z Stashbot 7414 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie 2461749 wikitext text/x-wiki 1m5yj8sd0oxymle9xzobls7h38m6j7m 2461750 2461749 2026-09-29T15:39:31Z Stashbot 7414 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () 2461750 wikitext text/x-wiki bl2f82r7b79jbp6ukc5adb4c9uhvd38 2461751 2461750 2026-09-29T15:43:13Z Stashbot 7414 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () 2461751 wikitext text/x-wiki c2ke8o5odn6w95vsd4ahb7zidjzi5sp 2461755 2461751 2026-09-29T15:59:09Z Stashbot 7414 jhathaway: updating puppet stdlib, should be a noop 2461755 wikitext text/x-wiki lafpdy7wy0hxi2erja9eil3mo74g6i1 2461760 2461755 2026-09-29T16:15:16Z Stashbot 7414 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply 2461760 wikitext text/x-wiki bk5g9ukfge6xeq19ibinptyiufmgaaf 2461761 2461760 2026-09-29T16:16:46Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2461761 wikitext text/x-wiki areb8nc0yydc71jdbm4pj6r1naf8hjs 2461762 2461761 2026-09-29T16:18:24Z Stashbot 7414 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply 2461762 wikitext text/x-wiki 99xuxs849ma451pw4a7rqlsqs0hzicg 2461763 2461762 2026-09-29T16:19:30Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2461763 wikitext text/x-wiki jxnx8x99c2gaeisu4iho1bhm6v74rma 2461764 2461763 2026-09-29T16:19:44Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2461764 wikitext text/x-wiki k8fw4v6z7nta0u5n3dtf4ngcv2ecpwg 2461765 2461764 2026-09-29T16:19:50Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2461765 wikitext text/x-wiki 6b6h3k5hifbvmvo51wy8qkz4743ttz0 2461766 2461765 2026-09-29T16:20:00Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # T418194 2461766 wikitext text/x-wiki o4s1d4ivp3rfnerg8018r7qlrcx2mm1 2461767 2461766 2026-09-29T16:20:20Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2461767 wikitext text/x-wiki rp3xb7i3tcj15q75ddjpjsmkrn25xpi 2461768 2461767 2026-09-29T16:21:06Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2461768 wikitext text/x-wiki gjevo7yyrt82flq8979kbd5ag3un0mp 2461769 2461768 2026-09-29T16:22:20Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2461769 wikitext text/x-wiki 5qtqeedlphpm9yez6w9vq55olz887ww 2461773 2461769 2026-09-29T16:35:23Z Stashbot 7414 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` T436186 2461773 wikitext text/x-wiki gdx7sbi0cp5yv4wjnjl9fynt71jk56q 2461774 2461773 2026-09-29T16:36:21Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028 2461774 wikitext text/x-wiki 9h7wc1v2lm9cs169b984jsfyiaxwfye 2461775 2461774 2026-09-29T16:36:39Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet 2461775 wikitext text/x-wiki gt51ze7bhosc4t2taeazwzb1ykgboa8 2461776 2461775 2026-09-29T16:54:29Z Stashbot 7414 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host 2461776 wikitext text/x-wiki lab6900n0v5e1ijcomplg4rq246ahiy 2461777 2461776 2026-09-29T16:54:33Z Stashbot 7414 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s) 2461777 wikitext text/x-wiki 57ukee5t55u34unacgrxdsclyc1qn4h 2461778 2461777 2026-09-29T16:54:37Z Stashbot 7414 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host 2461778 wikitext text/x-wiki toxsmaz4sf2sainwu7bt183ujj8w57v 2461779 2461778 2026-09-29T16:54:42Z Stashbot 7414 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s) 2461779 wikitext text/x-wiki cznzu3w0x4p52v1f0x6aiq4ht9bwhm3 2461781 2461779 2026-09-29T16:54:49Z Stashbot 7414 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host 2461781 wikitext text/x-wiki gb64c3akk7g19odnd2kx4quanimdqab 2461782 2461781 2026-09-29T16:54:53Z Stashbot 7414 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s) 2461782 wikitext text/x-wiki hzfncdgxeeczbj5c25ocgh03gha6n3j 2461783 2461782 2026-09-29T16:58:59Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.* 2461783 wikitext text/x-wiki tm7myckts77jkj5b3zvd57l1kqii742 2461784 2461783 2026-09-29T17:00:57Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards 2461784 wikitext text/x-wiki 7o0j2jddflnpa29ozip0fftmjqw0ek5 2461785 2461784 2026-09-29T17:01:15Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards 2461785 wikitext text/x-wiki m1ubc2d7arepc7thuxixov0pjk52kig 2461786 2461785 2026-09-29T17:01:44Z Stashbot 7414 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards 2461786 wikitext text/x-wiki b6bq3lisbwq1773x9yp6nowqzmg7br1 2461788 2461786 2026-09-29T17:39:12Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2461788 wikitext text/x-wiki p2km2z0i36yl4go7rqsk3olqz36qkm7 2461789 2461788 2026-09-29T17:44:02Z Stashbot 7414 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train 2461789 wikitext text/x-wiki fe1zrtawnxleedg1knsytohok1fxqmm 2461790 2461789 2026-09-29T17:45:56Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2461790 wikitext text/x-wiki 4ffpja7cytzqbs4a5wsiv5kk8wa84dy 2461791 2461790 2026-09-29T17:46:04Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2461791 wikitext text/x-wiki 6gnk2lqbz6gf9rdjze2k1xcc9sqycd3 2461793 2461791 2026-09-29T17:50:17Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2461793 wikitext text/x-wiki 9p1z4jmsmvi57l0bj4o5o7m6h9kjcs5 2461794 2461793 2026-09-29T17:51:14Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie 2461794 wikitext text/x-wiki 50ngbw0k4bvt9brtuosd4tei93axhsr 2461795 2461794 2026-09-29T17:51:54Z Stashbot 7414 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] 2461795 wikitext text/x-wiki mlle8h0naxrjw4yswc6ggy1wuk4732d 2461796 2461795 2026-09-29T17:52:40Z Stashbot 7414 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s) 2461796 wikitext text/x-wiki rwzp0mp6959ctd6sj3fqmet3oi6uwme 2461797 2461796 2026-09-29T17:53:09Z Stashbot 7414 sandraebele: Deploying Refinery as part of weekly deployment train 2461797 wikitext text/x-wiki hewk230rsfkcsqld7t1hbhbi9fvkk80 2461798 2461797 2026-09-29T17:53:34Z Stashbot 7414 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] 2461798 wikitext text/x-wiki rspkgkg0glnhlc5825jqzek8uymmh1m 2461799 2461798 2026-09-29T17:54:26Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461799 wikitext text/x-wiki ns5r387pxaf3bmxggqysjqnc5mlmtcv 2461800 2461799 2026-09-29T17:54:27Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461800 wikitext text/x-wiki q4wp33vzwshqlsb9hsjgmfe4nwcz744 2461801 2461800 2026-09-29T17:55:11Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461801 wikitext text/x-wiki j2glghzgs2tdcfac0ogk2rw6yorrjhs 2461802 2461801 2026-09-29T17:57:17Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461802 wikitext text/x-wiki 7cl1n57w990a7a8ydnwosqbn2vmmgq0 2461803 2461802 2026-09-29T17:58:29Z Stashbot 7414 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s) 2461803 wikitext text/x-wiki 9xph6jjmbc8x5lrx01cykj22iwbq013 2461804 2461803 2026-09-29T17:59:20Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards 2461804 wikitext text/x-wiki fk6vbtljcok08xmbi5gr4az8qw44b5v 2461805 2461804 2026-09-29T18:01:34Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards 2461805 wikitext text/x-wiki 6a1bosy0sci580dogf1r1afshpj55ia 2461806 2461805 2026-09-29T18:05:27Z Stashbot 7414 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () 2461806 wikitext text/x-wiki hvu49f0dvegziu2xuzupevjzz0b70o4 2461807 2461806 2026-09-29T18:05:29Z Stashbot 7414 sandraebele: Deployed Refinery as part of weekly deployment train 2461807 wikitext text/x-wiki ikw68k5yfyxd4w7y32obgcjx79b00r9 2461808 2461807 2026-09-29T18:06:19Z Stashbot 7414 brennen: 1.47.0-wmf.22 train status (T438218) - no current blockers, rolling to group0 2461808 wikitext text/x-wiki 2vd6z1q5aikj0bz1gh4p3hnaikrct7h 2461809 2461808 2026-09-29T18:08:49Z Stashbot 7414 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards 2461809 wikitext text/x-wiki 49y0t7lb2yjuzikj2zm4hgxgb7qtwgg 2461810 2461809 2026-09-29T18:15:10Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461810 wikitext text/x-wiki 0sio034xsj5sexm3m1s0myauulmhl23 2461811 2461810 2026-09-29T18:15:12Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461811 wikitext text/x-wiki 1jgvccve9yox4b5zye5yzoi3a3x5x8e 2461812 2461811 2026-09-29T18:15:37Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461812 wikitext text/x-wiki 8qomrpnjnrzer9zl8d9vi5tphnu42we 2461813 2461812 2026-09-29T18:18:09Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2461813 wikitext text/x-wiki n9g0znpigb8ob0wc3bs0vh0hum9opru 2461814 2461813 2026-09-29T18:18:18Z Stashbot 7414 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs T438218 2461814 wikitext text/x-wiki a8ho9dzuz76vtnykjk8t4rws9r3pd5i 2461815 2461814 2026-09-29T18:19:31Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461815 wikitext text/x-wiki 2e1qpxv1f6efid2kbch8ew5lj3uxvb8 2461816 2461815 2026-09-29T18:19:37Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461816 wikitext text/x-wiki m5jb90gtubx23m2h56osxeo2rrp4gtc 2461817 2461816 2026-09-29T18:19:54Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461817 wikitext text/x-wiki sa2nqwbft9s5ntkwl8npy3qryckbj3y 2461818 2461817 2026-09-29T18:24:20Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461818 wikitext text/x-wiki jolgnz6lydfzc8jl86a3smbutrca82m 2461819 2461818 2026-09-29T18:25:17Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461819 wikitext text/x-wiki 1h86lntkss1buncjg8p2wf3y1ur814l 2461820 2461819 2026-09-29T18:26:00Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet 2461820 wikitext text/x-wiki 2acxik680svr35z8il0kqzsqbn32o5e 2461821 2461820 2026-09-29T18:26:06Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet 2461821 wikitext text/x-wiki dhwswe6qshg8440zb2c0gfuluhhgknx 2461822 2461821 2026-09-29T18:26:11Z Stashbot 7414 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet 2461822 wikitext text/x-wiki avh5vti6gkxbbnxjgg5a264p025pvpt 2461824 2461822 2026-09-29T18:27:28Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.dns.netbox 2461824 wikitext text/x-wiki jmfw9exfqpuw4gzn4851u5fudkzalr0 2461826 2461824 2026-09-29T18:30:14Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2461826 wikitext text/x-wiki et9hcqbtuo0fcln9pznqivqhmxvvuje 2461827 2461826 2026-09-29T18:30:46Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2461827 wikitext text/x-wiki 5dzc3vdb50m3zfxvc6dgm90fxqutv8v 2461828 2461827 2026-09-29T18:31:31Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" 2461828 wikitext text/x-wiki q41oes45v9vxjz7tvzc11ih1c4cqn1m 2461829 2461828 2026-09-29T18:34:56Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" 2461829 wikitext text/x-wiki aax40dclonsep48nr4yecesxppg1q9n 2461830 2461829 2026-09-29T18:34:57Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2461830 wikitext text/x-wiki odb4dzul6soayhz4opsm7gg88mjbpv1 2461831 2461830 2026-09-29T18:35:07Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.dns.netbox 2461831 wikitext text/x-wiki ku71n9lt8q0bogl5fmxmxtqzyetjeot 2461833 2461831 2026-09-29T18:39:46Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" 2461833 wikitext text/x-wiki bnpk03rwory3wn1hk1ba6fjaj6ql0ok 2461835 2461833 2026-09-29T18:41:32Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" 2461835 wikitext text/x-wiki dcmo7buedolsi9r7lt6wl3l5fb2c4ei 2461836 2461835 2026-09-29T18:41:33Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2461836 wikitext text/x-wiki 1wcq3sn4zsyxwkvttul4nxwtruxy6bk 2461837 2461836 2026-09-29T18:45:24Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.dns.netbox 2461837 wikitext text/x-wiki 73s27ib50b2hn41184c3p3z6wqut7yl 2461838 2461837 2026-09-29T18:49:20Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" 2461838 wikitext text/x-wiki qk8rrk9306wasg2dk2vg67a6br715fn 2461839 2461838 2026-09-29T18:49:48Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" 2461839 wikitext text/x-wiki riggolgjlcyw2x1g0srrssw4rufsvtv 2461840 2461839 2026-09-29T18:49:49Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2461840 wikitext text/x-wiki 9agck45bsh4rvsam0fbmla9pzh1f25z 2461841 2461840 2026-09-29T18:50:01Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.dns.netbox 2461841 wikitext text/x-wiki r4p71kntmycrqfqjybzvgytjr0c8rzm 2461845 2461841 2026-09-29T18:56:50Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm 2461845 wikitext text/x-wiki rwud3eo2h3yn3cf36pmlidyzjtdzyqx 2461846 2461845 2026-09-29T18:56:52Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" 2461846 wikitext text/x-wiki hsczrxvu958vncuklyumuyr7c00l2nx 2461847 2461846 2026-09-29T18:56:55Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" 2461847 wikitext text/x-wiki 14eymao2l7cg9jsmsvaqdma38y9nrgg 2461848 2461847 2026-09-29T18:56:56Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2461848 wikitext text/x-wiki r3e3hktccbzomsx1qw0xl1jey6oeev3 2461849 2461848 2026-09-29T18:57:43Z Stashbot 7414 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.* 2461849 wikitext text/x-wiki swhti4d26krk6088mhjp1k0rthfqu7b 2461850 2461849 2026-09-29T19:02:09Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors 2461850 wikitext text/x-wiki lq4tdttuwftwr8zsy5c7vwyn2ylz9mh 2461851 2461850 2026-09-29T19:02:12Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors 2461851 wikitext text/x-wiki 743mzob3dyfvr1orjpn9mgp9xezx714 2461852 2461851 2026-09-29T19:03:25Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm 2461852 wikitext text/x-wiki d82fdwuuv3qdtcx83dpuaqtzwhatsem 2461853 2461852 2026-09-29T19:03:41Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm 2461853 wikitext text/x-wiki s7nbshezza6ivgv30s7uinc5asu8m44 2461854 2461853 2026-09-29T19:04:04Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm 2461854 wikitext text/x-wiki tvz3emikgzmxm5jgoy44etk54r5ubai 2461855 2461854 2026-09-29T19:05:17Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461855 wikitext text/x-wiki abxmfty086qlezri6qx9zryvyugx5y5 2461856 2461855 2026-09-29T19:05:47Z Stashbot 7414 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm 2461856 wikitext text/x-wiki botwd093a313yyy7l5az2p4qzy1fjiu 2461857 2461856 2026-09-29T19:06:19Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461857 wikitext text/x-wiki n9b4bs8t1vddoynl0jsfogmwuxm0ywu 2461858 2461857 2026-09-29T19:06:41Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461858 wikitext text/x-wiki 9a8hbt4j1zvl3mxz3037g67yso5yggz 2461859 2461858 2026-09-29T19:07:19Z Stashbot 7414 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm 2461859 wikitext text/x-wiki 2nwagr4gy7243hen0l24og6xenvb2hw 2461860 2461859 2026-09-29T19:08:04Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461860 wikitext text/x-wiki 9x6m2hangvv7r1lqi7ery2ctglhimro 2461861 2461860 2026-09-29T19:08:18Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461861 wikitext text/x-wiki 0hfo3bhny90bjthxdqpby9vfvb1qozs 2461862 2461861 2026-09-29T19:08:22Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461862 wikitext text/x-wiki a23wvvbf06ng16piisj07ok7uq7wi84 2461863 2461862 2026-09-29T19:08:25Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461863 wikitext text/x-wiki ol42713qucyte98ba6hfe1k0u7eyxsx 2461864 2461863 2026-09-29T19:09:01Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461864 wikitext text/x-wiki gw4x4m4nt14p403z7ee2fxdewznu84f 2461865 2461864 2026-09-29T19:09:08Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461865 wikitext text/x-wiki mlatsvnha9q4sd0f2xacb4olf9uoler 2461866 2461865 2026-09-29T19:09:14Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461866 wikitext text/x-wiki ctwwwf1txbjqh4fftrevcg3zng7usli 2461867 2461866 2026-09-29T19:10:19Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm 2461867 wikitext text/x-wiki 65qohe0si9h0adr3hs7yrdfj4hta15i 2461868 2461867 2026-09-29T19:10:32Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm 2461868 wikitext text/x-wiki 4keoo04kt8724fiqgug6rvnp3ou0uhc 2461869 2461868 2026-09-29T19:15:10Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461869 wikitext text/x-wiki ocysxbfsi8gbn6kr5q4uyn06vihunum 2461870 2461869 2026-09-29T19:15:58Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461870 wikitext text/x-wiki 46d7agc1u4wv4ghfhpqra7k3mbr50xv 2461871 2461870 2026-09-29T19:17:46Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461871 wikitext text/x-wiki gnqxasdity6tcnlf1ix22jbipd1vdmq 2461872 2461871 2026-09-29T19:19:11Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm 2461872 wikitext text/x-wiki dnsffw2xq8di4n77024sdmrayue82el 2461873 2461872 2026-09-29T19:22:40Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm 2461873 wikitext text/x-wiki h5ae7sakxum15szhtquzwaz4dlfs8bt 2461874 2461873 2026-09-29T19:23:41Z Stashbot 7414 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply 2461874 wikitext text/x-wiki rs6cn1otskabq0bes2yrzlmtvekx3tb 2461875 2461874 2026-09-29T19:24:03Z Stashbot 7414 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply 2461875 wikitext text/x-wiki sfmvcoc7vm062k5x6p0tvizezghbbnb 2461876 2461875 2026-09-29T19:27:54Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2461876 wikitext text/x-wiki 7mvq384i3l6egafyd6e0y3odiox6f2s 2461877 2461876 2026-09-29T19:29:15Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461877 wikitext text/x-wiki 5i6ub8czgmlo2egb41ptech8m3w9c8t 2461878 2461877 2026-09-29T19:30:18Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461878 wikitext text/x-wiki 97rzn8badfhqcwuke78hhjbxa544i89 2461879 2461878 2026-09-29T19:33:32Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage 2461879 wikitext text/x-wiki 0be6lx5inf1me2difhmbby4yzw8cjax 2461880 2461879 2026-09-29T19:33:53Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm 2461880 wikitext text/x-wiki 8jchxyeeyxhgb4m0yjb4pewtmoz4a68 2461881 2461880 2026-09-29T19:37:05Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage 2461881 wikitext text/x-wiki 05pb1flan2pp4footecb7lgrbbcjgkk 2461883 2461881 2026-09-29T19:38:01Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage 2461883 wikitext text/x-wiki hcoz8xcwjoo94pyzwf2iswhrmvpsx3g 2461884 2461883 2026-09-29T19:39:59Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461884 wikitext text/x-wiki 82m0g3cmdkw536ptzujtccpchqgxqrd 2461885 2461884 2026-09-29T19:40:16Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461885 wikitext text/x-wiki pw6qblcjo449obqkfcyt6tazk5y7mre 2461886 2461885 2026-09-29T19:40:36Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage 2461886 wikitext text/x-wiki bloycr9xsihskz2iydar2ps5uexgpwa 2461889 2461886 2026-09-29T19:48:39Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage 2461889 wikitext text/x-wiki dtxd0ql7mg1kb6zcwk4xinacg16ecw2 2461892 2461889 2026-09-29T19:52:42Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" 2461892 wikitext text/x-wiki 5c76dmoe1pbsolyoq9utwzjknpq6p9d 2461893 2461892 2026-09-29T19:53:21Z Stashbot 7414 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () 2461893 wikitext text/x-wiki l62gaqgoh27oaadlfxznev6f5zn3agl 2461894 2461893 2026-09-29T19:53:35Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage 2461894 wikitext text/x-wiki dp1x8gtfh24ij8yp8mtghkjcmk121g6 2461895 2461894 2026-09-29T19:55:13Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" 2461895 wikitext text/x-wiki 04j0fdqoqnxy2pwpr0k3cqpdnz9m7r2 2461896 2461895 2026-09-29T19:55:15Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm 2461896 wikitext text/x-wiki nun6cmxc7c9596tx3sit2si4e1wl8s2 2461897 2461896 2026-09-29T19:55:58Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" 2461897 wikitext text/x-wiki fzyrdqxtjow1t90e16dhf8maki8wqws 2461898 2461897 2026-09-29T19:56:07Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply 2461898 wikitext text/x-wiki ts3put3hvx6le4jnd9dz6r72561ommf 2461899 2461898 2026-09-29T19:56:08Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2461899 wikitext text/x-wiki 72l3txskgp0hwvvbjneuih3fp6iykj3 2461900 2461899 2026-09-29T19:56:17Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply 2461900 wikitext text/x-wiki 97qowexhfd5o60u2yg118hq8iduv2fc 2461901 2461900 2026-09-29T19:57:06Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2461901 wikitext text/x-wiki tbws500c1p75xke55usvh9ib2m5s9ql 2461903 2461901 2026-09-29T20:04:17Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" 2461903 wikitext text/x-wiki 7eehnmfzxlke4d8rmoptjzswbwk91zt 2461904 2461903 2026-09-29T20:04:19Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm 2461904 wikitext text/x-wiki 7hki4061ee62a4rlzh96f18zzwdxfno 2461905 2461904 2026-09-29T20:08:33Z Stashbot 7414 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1344774|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649|Remove foundationwiki's CSP report-only header (T423691)]] 2461905 wikitext text/x-wiki qt9kg27ufr7a286rnu9hthzlfkn5seo 2461906 2461905 2026-09-29T20:09:27Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" 2461906 wikitext text/x-wiki 7v4i9qr6i551zou9409uhq1kohj0h6j 2461907 2461906 2026-09-29T20:11:59Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" 2461907 wikitext text/x-wiki qm2dx5aw37t3tihx31vtr8iarw4flyd 2461908 2461907 2026-09-29T20:12:00Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm 2461908 wikitext text/x-wiki 4yk09zi1eipkvyel2lryhxa92igcrn2 2461909 2461908 2026-09-29T20:12:39Z Stashbot 7414 sbassett@deploy1003: sbassett: Backport for [[gerrit:1344774|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649|Remove foundationwiki's CSP report-only header (T423691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461909 wikitext text/x-wiki j8be3cyfsrqhuqyvw0oy0e00gt50gf3 2461910 2461909 2026-09-29T20:19:16Z Stashbot 7414 sbassett@deploy1003: sbassett: Rolling back deployment 2461910 wikitext text/x-wiki qxhpmgfv565uwos2xhjcln0eqlt5g5a 2461912 2461910 2026-09-29T20:22:38Z Stashbot 7414 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344774|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649|Remove foundationwiki's CSP report-only header (T423691)]] (duration: 14m 07s) 2461912 wikitext text/x-wiki jznh4bio3q7nih5laor6zqfhpdndvnp 2461914 2461912 2026-09-29T20:32:48Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042 2461914 wikitext text/x-wiki jc0hpqelfmp1dt47rp9yi9fins56mky 2461915 2461914 2026-09-29T20:33:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2461915 wikitext text/x-wiki td94hdbycjn9ap43peqlgvfehhz6bmg 2461916 2461915 2026-09-29T20:33:52Z Stashbot 7414 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1346087|InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] 2461916 wikitext text/x-wiki 8kn5j16v76vx9jcatyiuc6a7dmsah3i 2461917 2461916 2026-09-29T20:38:04Z Stashbot 7414 cklimas@deploy1003: cklimas: Backport for [[gerrit:1346087|InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461917 wikitext text/x-wiki 7568gd6p5lsbb6b9re0a074p99uciry 2461918 2461917 2026-09-29T20:42:54Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" 2461918 wikitext text/x-wiki pd63rw777rm9mbe6kzztt5xs4hbbq1f 2461919 2461918 2026-09-29T20:43:02Z Stashbot 7414 cklimas@deploy1003: cklimas: Continuing with deployment 2461919 wikitext text/x-wiki 217maxo7411jt9b1y4jksqp3k0nbvtf 2461920 2461919 2026-09-29T20:45:35Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" 2461920 wikitext text/x-wiki 9k485y2ugd8meyrkfsutb9806qyew23 2461921 2461920 2026-09-29T20:45:36Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2461921 wikitext text/x-wiki d0n1073vl6a6gxi01z2n91yvlhofluf 2461922 2461921 2026-09-29T20:45:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors 2461922 wikitext text/x-wiki f8fnxtqvvrlizw2lwb8q10dt52ct7xl 2461923 2461922 2026-09-29T20:45:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors 2461923 wikitext text/x-wiki hhwy8zel595vh4yck0yezf05kzajead 2461924 2461923 2026-09-29T20:45:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042 2461924 wikitext text/x-wiki q7yitpxja8hnp4rpfafjydqv6q1wmtz 2461925 2461924 2026-09-29T20:46:42Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042 2461925 wikitext text/x-wiki 39xhay9oa1fmhd6fgaylun63plr66uw 2461926 2461925 2026-09-29T20:47:19Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042 2461926 wikitext text/x-wiki 1pqmthf5uao7gtuwogrb1vui8cabmwy 2461927 2461926 2026-09-29T20:48:23Z Stashbot 7414 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346087|InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] (duration: 14m 32s) 2461927 wikitext text/x-wiki pxk79u8b584gj4g3k3bei7moej9p943 2461928 2461927 2026-09-29T20:51:27Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm 2461928 wikitext text/x-wiki 2kdqngp6db24t8vsbou5hc4v584itn3 2461929 2461928 2026-09-29T20:55:19Z Stashbot 7414 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1344405|Beta Cluster: Use different Redis instance for LockManager (T436480)]] 2461929 wikitext text/x-wiki gm892dhqmrg27gamvcd8e5662gkvbr8 2461930 2461929 2026-09-29T20:59:38Z Stashbot 7414 cklimas@deploy1003: cklimas, southparkfan: Backport for [[gerrit:1344405|Beta Cluster: Use different Redis instance for LockManager (T436480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2461930 wikitext text/x-wiki k580qp8tfv8aaa97b1yyrdlbca6hyzd 2461931 2461930 2026-09-29T21:02:39Z Stashbot 7414 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment 2461931 wikitext text/x-wiki 8sjvrs5so9rrw8ihfwfuh4e4ylciitc 2461932 2461931 2026-09-29T21:03:38Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] 2461932 wikitext text/x-wiki sch6es3waj76ybz77a7py2o7xceh5pw 2461933 2461932 2026-09-29T21:03:41Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] 2461933 wikitext text/x-wiki 4fanzm5ttg1fcd206sqtq58b93dzjzb 2461934 2461933 2026-09-29T21:03:57Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] 2461934 wikitext text/x-wiki t8k3pzmplnplzxek6sfr2kon3wrvpox 2461935 2461934 2026-09-29T21:04:01Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] 2461935 wikitext text/x-wiki a1lkatuitd45zqstf9oac9c5q4bs12d 2461936 2461935 2026-09-29T21:06:43Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage 2461936 wikitext text/x-wiki tvyciw1ep82tg2rh4gjju58dhnwurbw 2461937 2461936 2026-09-29T21:06:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] 2461937 wikitext text/x-wiki 9ub8tu60yq0zhz3yss5wodbrw3j291l 2461938 2461937 2026-09-29T21:06:59Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] 2461938 wikitext text/x-wiki iczy4pf9bjo1um0nq9cw3c9il40vz9j 2461939 2461938 2026-09-29T21:08:04Z Stashbot 7414 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344405|Beta Cluster: Use different Redis instance for LockManager (T436480)]] (duration: 12m 45s) 2461939 wikitext text/x-wiki 0vc79viegom6vw1zjhj01u9jyxshpyh 2461940 2461939 2026-09-29T21:11:30Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage 2461940 wikitext text/x-wiki 53rwgugm6cje3u4dyzm428f8ec3tft5 2461941 2461940 2026-09-29T21:16:04Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2461941 wikitext text/x-wiki l34hrzlok4dsbxpry7uvijnogu1rbgd 2461942 2461941 2026-09-29T21:16:20Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2461942 wikitext text/x-wiki 2udebbrdjc335ko22d4xtwuimgw2wi8 2461943 2461942 2026-09-29T21:20:23Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2461943 wikitext text/x-wiki am8c5g80yr6cqa8gy6buxsvnsxjqicn 2461944 2461943 2026-09-29T21:20:35Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2461944 wikitext text/x-wiki jgq688gyneq0lnmcb89uiuesbr1brak 2461945 2461944 2026-09-29T21:27:20Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm 2461945 wikitext text/x-wiki me4msbijo2czexh4hi3ohq9v1rxe8gt 2461946 2461945 2026-09-29T21:29:08Z Stashbot 7414 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - T427353 T395938 2461946 wikitext text/x-wiki 9kuy0y073z6lmd7oua5b2ex7it1xvc9 2461949 2461946 2026-09-29T21:55:05Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2461949 wikitext text/x-wiki amzwn4xq8srdxb9ksr9bct3ouf0zwll 2461950 2461949 2026-09-29T21:55:30Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2461950 wikitext text/x-wiki nff92wq9x7muw0qgzypj41kcfr21rmo 2461951 2461950 2026-09-29T21:55:42Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2461951 wikitext text/x-wiki 6jjeb5yallb0ub6g7v8n0dhlzchkmf1 2461952 2461951 2026-09-29T21:56:08Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2461952 wikitext text/x-wiki i3191k44hyhy7rab1f6ou6cgt1aoynn 2461953 2461952 2026-09-29T21:56:23Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043 2461953 wikitext text/x-wiki 9c8w43yfrjxdmz5ejmi0ii71msv91rh 2461954 2461953 2026-09-29T21:56:49Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2461954 wikitext text/x-wiki 6gh6lji3qy0rm24zkmbeyfuoeboxnw3 2461955 2461954 2026-09-29T21:56:54Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044 2461955 wikitext text/x-wiki 8n3k8miac0s0lvwkw1j3xw2e64uiag7 2461957 2461955 2026-09-29T22:01:08Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" 2461957 wikitext text/x-wiki 29bbjq402g4abqjp5wo0k1ynle1j3q3 2461958 2461957 2026-09-29T22:01:23Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" 2461958 wikitext text/x-wiki s57efw1jn1rbpww1ih3bnpmlc14eqgs 2461959 2461958 2026-09-29T22:01:24Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2461959 wikitext text/x-wiki i81cgcas6d7s5zkt9dzkr35utapzza9 2461960 2461959 2026-09-29T22:01:25Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors 2461960 wikitext text/x-wiki rs9sib6i2d9od7r7hms3os6be5meadi 2461961 2461960 2026-09-29T22:01:27Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors 2461961 wikitext text/x-wiki 2s4mzbmhbmpprl20d6ybcrgxrs2qxkg 2461962 2461961 2026-09-29T22:01:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043 2461962 wikitext text/x-wiki hw3hx3edceerpuytbs56xd4xknyrlc7 2461963 2461962 2026-09-29T22:01:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2461963 wikitext text/x-wiki tm8o56rfww7i212uc1dcqn7ajh9zvtk 2461964 2461963 2026-09-29T22:02:44Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043 2461964 wikitext text/x-wiki fpi4tneubwe9nmigep8o8uo93o2cl2s 2461965 2461964 2026-09-29T22:03:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043 2461965 wikitext text/x-wiki qhfennysond1f90hx6mr9vauntzyk11 2461966 2461965 2026-09-29T22:03:56Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm 2461966 wikitext text/x-wiki hycl6qskzf88ralvfhrfhhs61vp5avf 2461968 2461966 2026-09-29T22:05:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" 2461968 wikitext text/x-wiki irmqx49ckr3hp1rx38s0lu47fwxmnas 2461969 2461968 2026-09-29T22:05:52Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" 2461969 wikitext text/x-wiki 8f5ncm6ljmeauaptckryioarnqzl26f 2461970 2461969 2026-09-29T22:05:53Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2461970 wikitext text/x-wiki 7lhf337hqkhon6btow4tzey04n691s4 2461971 2461970 2026-09-29T22:05:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors 2461971 wikitext text/x-wiki h8kjk6db709n0hjvzw4we9kuvlferbl 2461972 2461971 2026-09-29T22:05:56Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors 2461972 wikitext text/x-wiki 56rb1dtdxku9x46amz28d4z37yx3pw1 2461973 2461972 2026-09-29T22:05:57Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044 2461973 wikitext text/x-wiki 29f534cflfynrp4zdsbskl4nx3lx6yt 2461974 2461973 2026-09-29T22:06:12Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044 2461974 wikitext text/x-wiki skahpg8v486rkywq7srsvwg4fwgrhqo 2461975 2461974 2026-09-29T22:06:49Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044 2461975 wikitext text/x-wiki 03y5th783q5etztp3du4i94ces7e694 2461976 2461975 2026-09-29T22:07:12Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm 2461976 wikitext text/x-wiki 2wjg6f6zjwv65v1mce8er1r3hm7mlhi 2461977 2461976 2026-09-29T22:19:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage 2461977 wikitext text/x-wiki 7aa0dr5e1yc5tyjc74hl0ma1uhyr4as 2461978 2461977 2026-09-29T22:21:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage 2461978 wikitext text/x-wiki 5622y1sgf77685vzscbpantymj9627y 2461979 2461978 2026-09-29T22:23:48Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage 2461979 wikitext text/x-wiki dx8bzwpjxebsgq2cqq2ft3j09zwduvc 2461980 2461979 2026-09-29T22:26:19Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage 2461980 wikitext text/x-wiki pg0ug59qlci5ybilfl939a8k311cyiz 2461981 2461980 2026-09-29T22:39:29Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm 2461981 wikitext text/x-wiki eah0pcdi0nnhh7zp6wa43xeb9vrlpls 2461982 2461981 2026-09-29T22:42:20Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm 2461982 wikitext text/x-wiki chjwhnkapx11rmll59gp2x91c9mnega 2461990 2461982 2026-09-30T02:00:34Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2461990 wikitext text/x-wiki q9ua3a01mosd4x36qut5wrku03kbfgq 2461991 2461990 2026-09-30T02:08:14Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) 2461991 wikitext text/x-wiki fbbxxw0qjyl8hgju94t81q8x3ny22kp 2461998 2461991 2026-09-30T06:32:17Z Stashbot 7414 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2461998 wikitext text/x-wiki gotcgygilrnabbg7jc4wzzyblk4bp2v 2461999 2461998 2026-09-30T06:33:53Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet 2461999 wikitext text/x-wiki j4y0il2eojb6ydb2twnrpgy2kk559mu 2462000 2461999 2026-09-30T06:48:57Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet 2462000 wikitext text/x-wiki qt9j5lnwu8k254isvh8dtc666td6sl8 2462001 2462000 2026-09-30T06:51:33Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2462001 wikitext text/x-wiki 8x42qx5vg6gfue2wmnlgzwfa2vfesse 2462002 2462001 2026-09-30T07:03:36Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1346012|Search: optimize morelike queries]] 2462002 wikitext text/x-wiki oe72k77wt9ndiv1pdr6efy4bb78rlhw 2462003 2462002 2026-09-30T07:07:58Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1346012|Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2462003 wikitext text/x-wiki 25m4igtpppt9tqedm1jmimkwh11zp22 2462004 2462003 2026-09-30T07:09:43Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2462004 wikitext text/x-wiki 0okd6rdc5g7bqxzhbn4twqkzq4gcc4k 2462005 2462004 2026-09-30T07:12:41Z Stashbot 7414 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2462005 wikitext text/x-wiki 4hz3mevezcv6itym0ss4v58h1iem12l 2462006 2462005 2026-09-30T07:13:26Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2462006 wikitext text/x-wiki o07majruocu76ct1ulqj0ops01fqey7 2462007 2462006 2026-09-30T07:15:03Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346012|Search: optimize morelike queries]] (duration: 11m 27s) 2462007 wikitext text/x-wiki q2mwd70e4avkppbys79j96ecy7uh7kd 2462008 2462007 2026-09-30T07:18:17Z Stashbot 7414 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2462008 wikitext text/x-wiki jmgi4ka94zf4agnqezpw1qaoef1psc8 2462009 2462008 2026-09-30T07:18:39Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2462009 wikitext text/x-wiki kf449gqyp67nuo56jjcl886u9e3i3cb 2462010 2462009 2026-09-30T07:35:39Z Stashbot 7414 moritzm: installing pcre2/openssl security updates 2462010 wikitext text/x-wiki lev8roruaxt5uitrkbw97d6oneblt5t 2462011 2462010 2026-09-30T07:44:27Z Stashbot 7414 elukey@dns1004: START - running authdns-update 2462011 wikitext text/x-wiki ahaf92shylxk2n8ftrnz7gojzqyfa5d 2462012 2462011 2026-09-30T07:44:55Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2462012 wikitext text/x-wiki c1du0gzp5fh4ufjnadexrhlf4fe5tjc 2462013 2462012 2026-09-30T07:44:59Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . 2462013 wikitext text/x-wiki di3t7cmp2ouw810rthhszi9ferm2cw7 2462014 2462013 2026-09-30T07:46:54Z Stashbot 7414 elukey@dns1004: END - running authdns-update 2462014 wikitext text/x-wiki 8r6ms4oil9mpmb3pbba53otxcqytann 2462015 2462014 2026-09-30T07:49:49Z Stashbot 7414 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2462015 wikitext text/x-wiki skl9dcwy0965msp33k8lc7lzkbmlg1k 2462019 2462015 2026-09-30T08:22:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw 2462019 wikitext text/x-wiki id4xg8f0q9lt9oamlhujheygb04ndrs 2462020 2462019 2026-09-30T08:24:25Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw 2462020 wikitext text/x-wiki cpgn8yubfq4t6ovpe9412f7kf08x6a7 2462021 2462020 2026-09-30T08:24:57Z Stashbot 7414 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca 2462021 wikitext text/x-wiki ogdliyingkg28o4yzkfqrgb0sb4oa8z 2462022 2462021 2026-09-30T08:25:31Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad 2462022 wikitext text/x-wiki btzxbtus2bgaudhnsqifd9meoqnvi4n 2462024 2462022 2026-09-30T08:27:32Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad 2462024 wikitext text/x-wiki fitm26kx3e73kfohezykjx5om9r17ca 2462034 2462024 2026-09-30T09:01:33Z Stashbot 7414 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' 2462034 wikitext text/x-wiki 07b8u3ujw0w2eljnvgz1oub5hkh3tag 2462036 2462034 2026-09-30T09:04:18Z Stashbot 7414 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' 2462036 wikitext text/x-wiki 6cu47soaihnhr5ab0u21q7ipao705oh 2462038 2462036 2026-09-30T09:06:18Z Stashbot 7414 btullis@dns1004: START - running authdns-update 2462038 wikitext text/x-wiki 17kdfgjaazwb6cip5i62v90oa1ibsiq 2462039 2462038 2026-09-30T09:08:46Z Stashbot 7414 btullis@dns1004: END - running authdns-update 2462039 wikitext text/x-wiki 8wr7y4ejlk778oyta73zoetu4c0mf43 2462042 2462039 2026-09-30T09:28:12Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw 2462042 wikitext text/x-wiki m0yl0m1iekffine3yslo0iy9gvgzgli 2462045 2462042 2026-09-30T09:32:48Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2462045 wikitext text/x-wiki 6m4i6xiyq6gz30khp9aqww03qkcscrb 2462046 2462045 2026-09-30T09:33:46Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2462046 wikitext text/x-wiki 69m61dvyub1uaoimiu55h9mmer2r0j0 2462047 2462046 2026-09-30T09:33:47Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw 2462047 wikitext text/x-wiki 5v5lx1cik1wmc1o1j3ghyk3krll02ca 2462048 2462047 2026-09-30T09:38:50Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad 2462048 wikitext text/x-wiki ftdpqup0p2p6lam88jxhwax79n0t4ih 2462050 2462048 2026-09-30T09:39:58Z Stashbot 7414 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad 2462050 wikitext text/x-wiki 4xfbe8gwx0f8hjsz2lxm8vdtg1a6i4l 2462054 2462050 2026-09-30T10:03:11Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad 2462054 wikitext text/x-wiki crvweh90axytcwawhz9vzls90gxw0cz 2462055 2462054 2026-09-30T10:05:25Z Stashbot 7414 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2462055 wikitext text/x-wiki 0sdrgwav0yqa69nti012ci4s9iudmvl 2462056 2462055 2026-09-30T10:06:32Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2462056 wikitext text/x-wiki kgzr4wgqidbmlo1avabulw9u6azbp5a 2462057 2462056 2026-09-30T10:06:33Z Stashbot 7414 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad 2462057 wikitext text/x-wiki hpucko5evqpfja14tt8glbh376i992p 2462059 2462057 2026-09-30T10:27:03Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2462059 wikitext text/x-wiki jgczpu9410sn73hdgqnz7p1jvfdxq4v 2462060 2462059 2026-09-30T10:27:11Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2462060 wikitext text/x-wiki 1tsyaugx02hmr4fajirm10isy3i44yj 2462062 2462060 2026-09-30T10:32:01Z Stashbot 7414 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - T438454 2462062 wikitext text/x-wiki mjg4t4opj321hx9u9upvele17axk82f 2462063 2462062 2026-09-30T10:33:13Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker1384.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2462063 wikitext text/x-wiki k7js01yg1de09i4adnl2bvoa8fhfsrs 2462064 2462063 2026-09-30T10:33:17Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet 2462064 wikitext text/x-wiki j3jgt0eoj3o4q6b6iippqv0spyxvh4w 2462065 2462064 2026-09-30T10:33:49Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet 2462065 wikitext text/x-wiki 2o5qebi7fzws4n1srcn86cbf9k3rzjx 2462066 2462065 2026-09-30T10:38:28Z Stashbot 7414 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service 2462066 wikitext text/x-wiki 5l7ltwrpg2lprell9sx6e8hr0ts91zx 2462067 2462066 2026-09-30T10:40:48Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet 2462067 wikitext text/x-wiki 48c9nolisavdhkdoqgvst9pvluk8co3 2462068 2462067 2026-09-30T10:40:49Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet 2462068 wikitext text/x-wiki qjldhzyhqkzqfl3lv9z873laiwux27n 2462069 2462068 2026-09-30T10:40:50Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker1384.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2462069 wikitext text/x-wiki 7ds4b7eajpm7730lhwfs5x37e9t2ns5 2462071 2462069 2026-09-30T10:56:23Z Stashbot 7414 cgoubert@deploy1003: Started scap sync-world: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 2462071 wikitext text/x-wiki gxhgtqzuwy5mvrk98tbpcy8y8h8e8pg 2462072 2462071 2026-09-30T10:59:36Z Stashbot 7414 cgoubert@deploy1003: cgoubert: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2462072 wikitext text/x-wiki qrzgoasig5sbmc8d4bgk2a1rh43xhh9 2462074 2462072 2026-09-30T11:01:46Z Stashbot 7414 cgoubert@deploy1003: cgoubert: Continuing with deployment 2462074 wikitext text/x-wiki 9lvy32m25evquu6unitpq5i58ibo9j8 2462075 2462074 2026-09-30T11:07:10Z Stashbot 7414 cgoubert@deploy1003: Finished scap sync-world: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 (duration: 11m 12s) 2462075 wikitext text/x-wiki ji7si6bytn4a8fwbc8he07jojrwzylw 2462079 2462075 2026-09-30T11:22:44Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply 2462079 wikitext text/x-wiki eimo8vg4jmrylj8r7pfkxcniucnnrnl 2462080 2462079 2026-09-30T11:22:47Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply 2462080 wikitext text/x-wiki 86qp7ch4418tubx270e511xvuc7tujs 2462081 2462080 2026-09-30T11:22:52Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply 2462081 wikitext text/x-wiki oq08apqquqmnwmikbp9cvu26d1i87e0 2462082 2462081 2026-09-30T11:22:55Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply 2462082 wikitext text/x-wiki 157iopgmr71gqtl3ucrquvqh7kefw73 2462083 2462082 2026-09-30T11:23:35Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2462083 wikitext text/x-wiki kg5fre7qmyd2jsgr2x01wsh70uog7cv 2462084 2462083 2026-09-30T11:24:02Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply 2462084 wikitext text/x-wiki cnru10d0uhl9672onjdjwv8ufwmzi2p 2462085 2462084 2026-09-30T11:24:03Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2462085 wikitext text/x-wiki n99dy28suzene8w2grxx3peo3ynhzw2 2462086 2462085 2026-09-30T11:24:07Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2462086 wikitext text/x-wiki mi9m4b35wxzpm4srsw3a6jx8wq7887m 2462087 2462086 2026-09-30T11:24:32Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply 2462087 wikitext text/x-wiki 1sooumbwanay1cs6e0b1lp28dpjgylc 2462088 2462087 2026-09-30T11:24:41Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2462088 wikitext text/x-wiki ea7xbv9b7e5x74o3v107x9ixh7htmft 2462089 2462088 2026-09-30T11:24:47Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply 2462089 wikitext text/x-wiki kdstc223mjtfr1fgtr72dalg7so6m98 2462090 2462089 2026-09-30T11:25:17Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply 2462090 wikitext text/x-wiki 1nseas8reoiuimjpuhgwmexcu31hq21 2462091 2462090 2026-09-30T11:27:00Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2462091 wikitext text/x-wiki o487a1cjgol2egcvv7th4c4rvms4e2c 2462092 2462091 2026-09-30T11:29:49Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2462092 wikitext text/x-wiki 0oon7l0mnfo67unl5e48d6vdpqbpxbb 2462093 2462092 2026-09-30T11:32:33Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet 2462093 wikitext text/x-wiki 610fmip6bzcqwtzuj6r6dczrbo8vjsc 2462094 2462093 2026-09-30T11:32:42Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet 2462094 wikitext text/x-wiki fhaqogm7nyxuyzb0d8e8rhvbv2muas2 2462095 2462094 2026-09-30T11:32:50Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet 2462095 wikitext text/x-wiki r9s7gsrzhw6gwe9v7u6z0fnk6ulc8tv 2462096 2462095 2026-09-30T11:32:57Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet 2462096 wikitext text/x-wiki k93cms8ha5r33ylwbvdm3gnzwlat9t7 2462097 2462096 2026-09-30T11:33:05Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet 2462097 wikitext text/x-wiki 5r0l3ij0iftvq589m31ocffgwir522o 2462098 2462097 2026-09-30T11:33:30Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet 2462098 wikitext text/x-wiki cdu66jfck7xny82luwy0husy0het9f3 2462099 2462098 2026-09-30T11:34:10Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet 2462099 wikitext text/x-wiki 71t2c4hkxl6u6be5hd9km0pcjb2r7qj 2462100 2462099 2026-09-30T11:36:03Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2462100 wikitext text/x-wiki 80098uhmd5uge25jx3t23tg51bazncm 2462103 2462100 2026-09-30T11:39:46Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet 2462103 wikitext text/x-wiki 0eci50wx5k7l7shxk70igzhneg45e6j 2462104 2462103 2026-09-30T11:41:11Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2462104 wikitext text/x-wiki ljorp0bh9tjgpyqxo3ml7jpgl24od9t 2462105 2462104 2026-09-30T11:55:03Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet 2462105 wikitext text/x-wiki 92pkrz6jg8krnqaboimz43r43lwvhlg 2462106 2462105 2026-09-30T11:55:19Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet 2462106 wikitext text/x-wiki qv0bi1yq3eacjclrwj2bfdf5d69ysfc 0 17290 2461843 2461563 2026-09-29T18:55:20Z Stashbot 7414 James_F: Zuul: [fresh] Make shellcheck voting and required before tests, for T439219 2461843 wikitext text/x-wiki jgq5exqft9ld810h2pvdwiif44r8yzr 2461947 2461843 2026-09-29T21:37:11Z Stashbot 7414 Southparkfan: Beta Cluster: remove $wmgRedisPassword from PS.php, superseded by $wmgRedisLockPassword - T436480 2461947 wikitext text/x-wiki 08vpujk1f4cyrezsd0bjtbo4a514lyl 2461948 2461947 2026-09-29T21:52:21Z Stashbot 7414 Southparkfan: decommission deployment-rdb01 - T436480 2461948 wikitext text/x-wiki amtzf3ta8gup194rbje50kvw2evc54f 12 18681 2461754 2456083 2026-09-29T15:54:07Z Raymond Ndibe 16522 added reference to the --webservice option in jobs-api 2461754 wikitext text/x-wiki rznfzo5fwbr1340m0oou4cp4pktlems 0 24248 2462070 2460091 2026-09-30T10:42:28Z Tiziano Fogli 37623 /* Upgrade steps: */ 2462070 wikitext text/x-wiki odedll1b4e0088oxsmu1al45snk1sk4 0 26677 2462061 2401773 2026-09-30T10:27:13Z HNowlan (WMF) 15760 Link runbook 2462061 wikitext text/x-wiki smfyphs9496z22regxvu48k0q564g7q 498 30942 2461680 2461531 2026-09-29T12:42:19Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2461680 wikitext text/x-wiki 6krcokisxfh4ij6n9nklu6t01fxb1vq 2461967 2461680 2026-09-29T22:04:44Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) 2461967 wikitext text/x-wiki 8xfox04hjrvlnsl2crcrg27i34sxa4y 2461986 2461967 2026-09-30T01:16:46Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add 2461986 wikitext text/x-wiki oaycyl743c1dtjk67x9zntjf5ldsbc5 2461987 2461986 2026-09-30T01:25:33Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) 2461987 wikitext text/x-wiki ctnyay5v2f2c76ory7bn9iu8jjqbflk 2461988 2461987 2026-09-30T01:28:29Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2461988 wikitext text/x-wiki 39p0pny457z7tms1s95aoz8uogrs6v6 2462073 2461988 2026-09-30T10:59:52Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) 2462073 wikitext text/x-wiki azygl91wigceyl21lqmomoiczq8qz1a 0 33280 2461913 2452923 2026-09-29T20:26:49Z JHathaway (WMF) 28372 /* ops-limited group */ 2461913 wikitext text/x-wiki psrho3f1uare18la7idudkvtr1bvgcg 12 441749 2461719 2387392 2026-09-29T13:58:53Z DKertesz-WMF 59186 /* Catalog compiler local run (pcc utility) */ replace link to generate an API token in Jenkins 2461719 wikitext text/x-wiki pldgvrdt6hdas3jhfi1ihv05an9757z 0 443307 2461823 2461520 2026-09-29T18:27:28Z BCornwall-WMF 31521 /* Architecture */ system architecture sections 2461823 wikitext text/x-wiki m8klqxgilxktg0ycg6ly7jtywlf38sy 2461825 2461823 2026-09-29T18:27:58Z BCornwall-WMF 31521 /* Architecture */ ce: promote active v passive heading 2461825 wikitext text/x-wiki 4xed6pjknos571ubxj9bqtwzyx579g8 2461844 2461825 2026-09-29T18:56:40Z BCornwall-WMF 31521 /* Atomic rollout */ ce: clarify example location 2461844 wikitext text/x-wiki 6yz4fuibdwusj251jxabfawgwvjza8l 2461882 2461844 2026-09-29T19:37:56Z BCornwall-WMF 31521 /* Architecture */ move intro paragraph to active/passive docs 2461882 wikitext text/x-wiki 5sy5k0y9sl4ki2ugu96plp28qzuyitl 2461887 2461882 2026-09-29T19:42:48Z BCornwall-WMF 31521 ce: intro grammar 2461887 wikitext text/x-wiki 4hzedwhl3zggpwphc8as8rt86ebiibp 2461888 2461887 2026-09-29T19:45:58Z BCornwall-WMF 31521 /* CLI utilities */ ce: clean up 2461888 wikitext text/x-wiki sh80xlrhr6rgiohp1gy4vyufhdsxkyb 0 444231 2461984 1834058 2026-09-30T00:48:39Z Mutante 20 replace broken external link to arch, which we don't use, with internal redirect to the server lifecycle page, based on comment on talk page by BCornwall 2461984 wikitext text/x-wiki 14qqpewf918zta5babczrm8rm8y8toj 0 444398 2461740 2460208 2026-09-29T14:39:15Z Milimetric 612 /* MySQL wrapper */ 2461740 wikitext text/x-wiki 8pvnaldqne46vorht71vqumz07k4u9d 0 446037 2462077 2418356 2026-09-30T11:14:14Z HNowlan (WMF) 15760 move reporting documentation to its own page 2462077 wikitext text/x-wiki 0vwlumi82qc6w17zjrrzsay8dig40tk 498 447870 2461832 2335513 2026-09-29T18:39:29Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 3d16b7ab07 (upgrade dependencies) 2461832 wikitext text/x-wiki 6b01fnr22itny2v8lva4kcebhyt3m05 2461834 2461832 2026-09-29T18:40:31Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 2bbb24b895 (Python 3.14) 2461834 wikitext text/x-wiki 3hkyiovj84dwbom52q7q4niqdoy6dgk 2461842 2461834 2026-09-29T18:51:56Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 1f97dafd49 (Toolforge Components Service, push-to-deploy) 2461842 wikitext text/x-wiki bj0e7rcxax3vngfwrhclpbc8uuinwjd 0 448981 2461911 2397560 2026-09-29T20:20:37Z CStone (WMF) 38439 /* Failures and retry logic */ 2461911 wikitext text/x-wiki 7qcsm6w3kfn1y05prsvfpq3ylc3mztl 2461989 2461911 2026-09-30T01:55:27Z CStone (WMF) 38439 /* Failures and retry logic */ 2461989 wikitext text/x-wiki 95orjlr8cs67f6yzjcoakftpmxz86e0 12 449113 2461742 2452173 2026-09-29T14:49:12Z Raymond Ndibe 16522 added documentation for the --webservice option 2461742 wikitext text/x-wiki jfdrok5pgnd3szukos0c7nxxcu782tn 2461902 2461742 2026-09-29T20:02:07Z Tacsipacsi 7034 /* Creating webservice jobs */ copy edit 2461902 wikitext text/x-wiki t0p40zsdb7b1i9zfch7rucmh39nwy4u 2462101 2461902 2026-09-30T11:36:45Z DCaro (WMF) 18271 /* Creating webservice jobs */ fixed a few typos and reworded a couple phrases 2462101 wikitext text/x-wiki c0f90286v1aflhtrvsgesq0h6avflu6 2462102 2462101 2026-09-30T11:38:44Z DCaro (WMF) 18271 /* Creating webservice jobs */ fixed typo 2462102 wikitext text/x-wiki fub8oix446ufwthc9z6ynmw7hw7lbxj 0 449160 2461983 2461506 2026-09-30T00:01:40Z Dexbot 30554 Bot: Updating the report 2461983 wikitext text/x-wiki erwxdkmkjwcp4vma5nn2uue7lbbelpt 0 450784 2461753 2130537 2026-09-29T15:47:37Z Geno (WMF) 20938 Add links to architecture and storage documentation in mediawiki 2461753 wikitext text/x-wiki km3mynxku1pywm59h199skmw8umo1dd 828 452251 2461752 2400488 2026-09-29T15:45:39Z Raymond Ndibe 16522 stop using variant in generateJobsFrameworkImageTable function 2461752 Scribunto text/plain qhn2wzd38qo3yw3jfcrnxnevfp8yxxg 0 453597 2461748 2456513 2026-09-29T15:22:51Z Raymond Ndibe 16522 added --webservice support changelog 2461748 wikitext text/x-wiki fv0zxt5txo4tq68wqtistx99t8swozh 116 453906 2461709 2461510 2026-09-29T13:45:19Z Gitlabaccountapprovalbot 37332 monathierse was rejected. 2461709 wikitext text/x-wiki lzg8oa424oo8hyztvw8ooje6yd9fq3x 0 459342 2461741 2430115 2026-09-29T14:46:39Z BKing (WMF) 29017 /* Getting Started With Your New Cluster */ 2461741 wikitext text/x-wiki 7ssgki1z33v4accrnpgt1hvyob9il1s 0 459699 2461746 2461535 2026-09-29T15:13:55Z Stashbot 7414 ejegg: payments-wiki upgraded from 49e656bc to dae59de9 2461746 wikitext text/x-wiki i11z3j3b1bb1mkoucaudcxl9xja5qmn 2461890 2461746 2026-09-29T19:49:53Z Stashbot 7414 eileen: civicrm upgraded from a7d8ba32 to 37e1be22 2461890 wikitext text/x-wiki nc7g6pvadmsj7o5vvrly8p6johpdq5j 2461891 2461890 2026-09-29T19:51:26Z Stashbot 7414 eileen: civicrm upgraded from 37e1be22 to 6144ab57 2461891 wikitext text/x-wiki exg131faduz945a6p50g67ltleiaobq 2461992 2461891 2026-09-30T02:20:57Z Stashbot 7414 eileen: civicrm upgraded from 6144ab57 to 7ff3e786 2461992 wikitext text/x-wiki jpkltr2o4o4xieir7946wgbud3fyknh 0 460515 2461726 2453929 2026-09-29T14:07:05Z Ottomata 88 /* See also */ added LAC vs LAS 2461726 wikitext text/x-wiki 3hd80zfih1iyoc7gb75jnhnne4ju5pn 0 460710 2461993 2461522 2026-09-30T02:54:38Z AWesterinen-WMF 54189 /* Query hints */ 2461993 wikitext text/x-wiki mysv8errk5bhyilx8eni166pzoefqou 2461994 2461993 2026-09-30T03:18:39Z AWesterinen-WMF 54189 /* wikibase:isSomeValue() */ 2461994 wikitext text/x-wiki rx974irovjsf070wbjpqa9z4x6nuzsz 2461995 2461994 2026-09-30T04:32:33Z AWesterinen-WMF 54189 /* bd:slice service */ 2461995 wikitext text/x-wiki 9x35ifmoepoax2o93gb7rb1hgrowfvm 2461996 2461995 2026-09-30T04:33:19Z AWesterinen-WMF 54189 /* bd:slice service */ 2461996 wikitext text/x-wiki oil5eb0tup5rzjxyna1ivuu3lc4es3y 2461997 2461996 2026-09-30T04:38:56Z AWesterinen-WMF 54189 /* bd:slice service */ 2461997 wikitext text/x-wiki ow2x7p6jl78da1uxp6549ysbia5wwx7 0 460719 2462031 2459386 2026-09-30T08:51:47Z Mvolz (WMF) 2813 2462031 wikitext text/x-wiki bm7z7nrbgli06h20uknlwphrxfw6zbs 12 460735 2461759 2461395 2026-09-29T16:14:40Z FNegri-WMF 32595 /* What you receive */ update email address, we decided to use your-tool.alerts instead of tools.your-tool 2461759 wikitext text/x-wiki dpd77i5s1rbu3ua9yse0bq1haklwpg2 1 460736 2461985 2461465 2026-09-30T00:50:30Z Mutante 20 /* Don't think this belongs here */ Reply 2461985 wikitext text/x-wiki exvuixsc2290ixaxsjqlyjsyep6vx5p 1 460744 2461725 2026-09-29T14:06:25Z Ottomata 88 /* LAC vs "LAS" tradeoffs */ new section 2461725 wikitext text/x-wiki ouvcxw9qhmeymc21kuz37a84pjnhkht 0 460746 2462076 2026-09-30T11:12:20Z HNowlan (WMF) 15760 Move incident follow-up to its own page 2462076 wikitext text/x-wiki 5ctdqhat26x3q0zeh10kqwp6nq9s3f6 2462078 2462076 2026-09-30T11:17:03Z HNowlan (WMF) 15760 /* Anti-patterns */ 2462078 wikitext text/x-wiki evsar5p91ak2vnlj0xpkg41eh3f82rq