Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.22 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2463497 2463428 2026-10-05T12:51:54Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2463497 wikitext text/x-wiki 37p7hv2dx1y5vhwrxjoxreqradqyb2u 2463507 2463497 2026-10-05T12:56:40Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder 2463507 wikitext text/x-wiki 2um9ukdn9k2jbqvgtxhqg2pup3gk3b1 2463529 2463507 2026-10-05T13:14:28Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission 2463529 wikitext text/x-wiki 0yggd0ptg5ozklct66taj4j8s7dxkoy 2463551 2463529 2026-10-05T13:28:26Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission 2463551 wikitext text/x-wiki hxjodg0kz9oogsffykc7hd3hcgvzzgb 2463617 2463551 2026-10-05T14:16:12Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api 2463617 wikitext text/x-wiki 6o78a279eaeupzyfqfkgkic5mwmg1x1 2463622 2463617 2026-10-05T14:17:25Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api 2463622 wikitext text/x-wiki a6mzyvv6mtsdfw18l6qz3gpa6furmau 2463693 2463622 2026-10-05T14:54:21Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T439237) 2463693 wikitext text/x-wiki 8ypvw56xfetqidzv2n5m4j62s4hiviy 2463706 2463693 2026-10-05T15:03:09Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers (T439237) 2463706 wikitext text/x-wiki s80khc13u3bft0v23tsuy78t7i27072 2463783 2463706 2026-10-05T16:57:32Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T439237) 2463783 wikitext text/x-wiki qcy495x6d5bpy4q3eblnf2ev8pzlpk4 2463795 2463783 2026-10-05T17:19:27Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T439237) 2463795 wikitext text/x-wiki 5myln67w1xx9ky7pdmlri63qinml3xa 2463823 2463795 2026-10-05T18:06:33Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2463823 wikitext text/x-wiki 75s2tflzo1myxjci5vq21m02uqtnjue 2463829 2463823 2026-10-05T18:18:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2463829 wikitext text/x-wiki tj530kut62yavl68gxsrz40bgtbvv01 0 4108 2463487 2463422 2026-10-05T12:48:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1349752]] to Monday, October 05 UTC afternoon backport window 2463487 wikitext text/x-wiki 0obq94jzn1lmn2ay87qh2pda4568okr 2463499 2463487 2026-10-05T12:52:21Z ScheduleDeploymentBot 37566 Add [[gerrit:1351194]] to Monday, October 05 UTC afternoon backport window 2463499 wikitext text/x-wiki n85oojkgmqwjbd7k35iozdnkxq107gu 2463555 2463499 2026-10-05T13:33:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1351316]] to Tuesday, October 06 UTC morning backport window 2463555 wikitext text/x-wiki n44koag5jfvhxnqmrgzz2si7377rgm3 2463597 2463555 2026-10-05T13:58:42Z LToscano (WMF) 7191 /* Tuesday, October 6 */ 2463597 wikitext text/x-wiki 2g6jtpaecygwjihp9q9f83ogaocam56 2463743 2463597 2026-10-05T15:28:00Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-10-05}} */ UTC-late infra window 2463743 wikitext text/x-wiki 4lccr58kecm17nzn1k9z5e7tu9n90xj 2463750 2463743 2026-10-05T15:30:31Z ScheduleDeploymentBot 37566 Add [[gerrit:1351378]] to Tuesday, October 06 UTC morning backport window 2463750 wikitext text/x-wiki 1eio25oedy33rw5f67icd3i10pah2fq 2463751 2463750 2026-10-05T15:30:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1351379]] to Tuesday, October 06 UTC morning backport window 2463751 wikitext text/x-wiki 9dcnoeook7u958mdrbhzrf9ykht8xsq 2463879 2463751 2026-10-05T22:11:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1351518]] to Tuesday, October 06 UTC morning backport window 2463879 wikitext text/x-wiki ju45k7ctqlagfslgfsvnv6m6g57o1nm 2463907 2463879 2026-10-06T07:08:37Z ScheduleDeploymentBot 37566 Add [[gerrit:1351826]] to Tuesday, October 06 UTC morning backport window 2463907 wikitext text/x-wiki gn51eiw254anxr9u4jlzum8d93v1ss7 498 6726 2463469 2463426 2026-10-05T12:36:52Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2463469 wikitext text/x-wiki kuy2wkmr2epk8z9ffcy9rbkla51lcys 2463470 2463469 2026-10-05T12:37:28Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder 2463470 wikitext text/x-wiki 7q7s2zqpnn276lsgtk686c3q0xmui1e 2463480 2463470 2026-10-05T12:45:45Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2463480 wikitext text/x-wiki 0j5guva2624nkqquy3dz7cmnvn248f4 2463493 2463480 2026-10-05T12:50:01Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder 2463493 wikitext text/x-wiki skxm4wgxtdbsipw774avrk9eqbwl4c4 2463510 2463493 2026-10-05T12:59:00Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission 2463510 wikitext text/x-wiki 3sccf7esx4108tpovsqlyj9n8w4fh4l 2463523 2463510 2026-10-05T13:11:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission 2463523 wikitext text/x-wiki 75pa3pyi8fbhgl17ooulp5jf9w6y7iq 2463607 2463523 2026-10-05T14:06:20Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api 2463607 wikitext text/x-wiki e4xd0hhlvaswq4b3f1qbfebqp57acno 2463608 2463607 2026-10-05T14:07:19Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api 2463608 wikitext text/x-wiki p64l32yq7cb0qlslqezbpz23wnw59v2 2463810 2463608 2026-10-05T17:49:49Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2463810 wikitext text/x-wiki d8ari7xh31eco8e2ux6b1jnriz7m7i5 2463811 2463810 2026-10-05T17:50:38Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api 2463811 wikitext text/x-wiki rjdz7fwtl7fbfsdf9dn5qvc67egm4gc 2463814 2463811 2026-10-05T17:54:00Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2463814 wikitext text/x-wiki f39vu2zjfejsbrju9kxafftfym4b5a3 2463815 2463814 2026-10-05T17:54:18Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api 2463815 wikitext text/x-wiki ei8kxpexa1ng2v6xadqr74z56w855qc 2463816 2463815 2026-10-05T17:55:51Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2463816 wikitext text/x-wiki 9vpfqkyji1kaa3jzjfwxn7p3ti1qgsr 2463822 2463816 2026-10-05T18:06:12Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2463822 wikitext text/x-wiki czz24ltwsr27at2x7ieigdf3fo591ve 0 7919 2463450 2463448 2026-10-05T12:00:16Z Stashbot 7414 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org 2463450 wikitext text/x-wiki po62xfwm9zde6ql6ghyyckm5n6t824o 2463451 2463450 2026-10-05T12:00:26Z Stashbot 7414 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org 2463451 wikitext text/x-wiki acrvhqgv1sme96ptvfiharhiytsge6w 2463453 2463451 2026-10-05T12:00:42Z Stashbot 7414 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org 2463453 wikitext text/x-wiki gdsgfm7vtfwmsj7vm38rbl91fr148w1 2463454 2463453 2026-10-05T12:01:21Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for [[gerrit:1350233|instrument(AccountSetup): expose motivation data for control (T439887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463454 wikitext text/x-wiki geess0b5y99mx9z74qh5tuc3c2dfyce 2463457 2463454 2026-10-05T12:12:30Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage 2463457 wikitext text/x-wiki 5io8iaep11kevn2jhosdkks32fdwhmo 2463461 2463457 2026-10-05T12:16:43Z Stashbot 7414 moritzm: installing mesa security updates 2463461 wikitext text/x-wiki pyidzkz0tbcen8nrbodjlaevbpu1dfu 2463462 2463461 2026-10-05T12:17:26Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage 2463462 wikitext text/x-wiki jj89goqpzediedf7yuu7x7pl8uip1dj 2463463 2463462 2026-10-05T12:26:20Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment 2463463 wikitext text/x-wiki 2vjy29asrg2embbwovpd3pczirhnpqb 2463466 2463463 2026-10-05T12:31:45Z Stashbot 7414 moritzm: installing sqlite3 security updates 2463466 wikitext text/x-wiki 905b31gj2f6vv6av7ppaktyedp6doha 2463467 2463466 2026-10-05T12:35:44Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2463467 wikitext text/x-wiki 1q0vh3pdgtp3abcyetvckkxruyxqrbz 2463472 2463467 2026-10-05T12:38:26Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1350233|instrument(AccountSetup): expose motivation data for control (T439887)]] (duration: 55m 26s) 2463472 wikitext text/x-wiki 39rdzxgmgi6m4ak5umwmhmebttj83ta 2463474 2463472 2026-10-05T12:38:36Z Stashbot 7414 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" 2463474 wikitext text/x-wiki 88z51b0nwznwrcswmxjxnhmn3oheeft 2463475 2463474 2026-10-05T12:38:37Z Stashbot 7414 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie 2463475 wikitext text/x-wiki cdiu3cq7jvvi1lorqwqrfd4rd11r4tv 2463476 2463475 2026-10-05T12:39:16Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463476 wikitext text/x-wiki 2tc8n7jadtkq3hls58qlxtcve8dxdne 2463477 2463476 2026-10-05T12:45:05Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463477 wikitext text/x-wiki sq2lutzerbwz7ucb2hj1n3jjtl0ayio 2463478 2463477 2026-10-05T12:45:10Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet 2463478 wikitext text/x-wiki g74azwd5dog67vou4npjfspa8xxu9z4 2463479 2463478 2026-10-05T12:45:23Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2463479 wikitext text/x-wiki 5oley8jmi0he9b9ivews8yer56mej9i 2463481 2463479 2026-10-05T12:46:02Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet 2463481 wikitext text/x-wiki qnao5lr6o7ng8rqwae50t1iz21newbf 2463482 2463481 2026-10-05T12:46:04Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet 2463482 wikitext text/x-wiki s10z229lx50jgx6cklhnlmweb0hopwb 2463483 2463482 2026-10-05T12:46:38Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2463483 wikitext text/x-wiki ataoi8m61vh59v1a8heh1nlq28g1sbi 2463484 2463483 2026-10-05T12:46:39Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet 2463484 wikitext text/x-wiki qvym8jbqemx8mdbo4swu643lw2eatlw 2463485 2463484 2026-10-05T12:47:33Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet 2463485 wikitext text/x-wiki k10vyvr1q869v8lk9iovvr59f28bbzn 2463486 2463485 2026-10-05T12:48:16Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2463486 wikitext text/x-wiki mwz4l05v6gki6dsbyzxxfyhnmaluxty 2463488 2463486 2026-10-05T12:48:41Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet 2463488 wikitext text/x-wiki 262qra5ec6c5s6a838su18ljnzqiz9e 2463489 2463488 2026-10-05T12:48:44Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet 2463489 wikitext text/x-wiki 81bdtgo2kuqsdvxqcvhanwrz47fcp21 2463490 2463489 2026-10-05T12:49:25Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet 2463490 wikitext text/x-wiki h0k364wdufgm32kbx1zshxwi4vbjtuf 2463491 2463490 2026-10-05T12:49:36Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm 2463491 wikitext text/x-wiki qtrbmttreah4zb9jmqfmi81g3kepopa 2463492 2463491 2026-10-05T12:50:00Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet 2463492 wikitext text/x-wiki n49dkx2aa6i6pnoagfrmbjtkq7h6j8q 2463494 2463492 2026-10-05T12:50:53Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.dns.netbox 2463494 wikitext text/x-wiki jd0qcvruazo13iozsq0dlojxtf7vlwy 2463498 2463494 2026-10-05T12:51:56Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463498 wikitext text/x-wiki ho0i82igt0u9rw5n54gnffmguupiwj2 2463501 2463498 2026-10-05T12:55:10Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" 2463501 wikitext text/x-wiki o0z4k1a39hkup27nb3ledry9j4m7lzh 2463502 2463501 2026-10-05T12:55:23Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" 2463502 wikitext text/x-wiki k0ley00qkmsmyn3ohf9vkaoq5rsgda5 2463503 2463502 2026-10-05T12:55:24Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2463503 wikitext text/x-wiki 7ifihbnrci3icknqlgnf0puswx2jxi4 2463504 2463503 2026-10-05T12:55:25Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet 2463504 wikitext text/x-wiki rb80jz0fmaicwhq3ulntg2xwpju45az 2463505 2463504 2026-10-05T12:56:18Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet 2463505 wikitext text/x-wiki h06g3eua6few0x7us9c5aj7s3en65z0 2463506 2463505 2026-10-05T12:56:28Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2463506 wikitext text/x-wiki s1gtkv6ch9aoxyy0n6owauamdsxoeug 2463508 2463506 2026-10-05T12:57:59Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm 2463508 wikitext text/x-wiki 0kxcli7o9fzmr1jabep2vma96of32t2 2463509 2463508 2026-10-05T12:58:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2463509 wikitext text/x-wiki 747lml6dfr85hbfw7m3hntzlox4mgi5 2463512 2463509 2026-10-05T12:59:51Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2463512 wikitext text/x-wiki r8t3bds3hojiip95jlnx0ym26telpld 2463513 2463512 2026-10-05T13:01:17Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2463513 wikitext text/x-wiki tty2xv5b5eq3atk14nm9f1nygsk1j7k 2463514 2463513 2026-10-05T13:01:50Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.dns.netbox 2463514 wikitext text/x-wiki 1lfe7bswcslub8wxppp2ku042va2qhi 2463515 2463514 2026-10-05T13:04:50Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet 2463515 wikitext text/x-wiki bn8nf76aiz0yiz3jc9j51oa4018k4f2 2463516 2463515 2026-10-05T13:05:38Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" 2463516 wikitext text/x-wiki 4rkk8pwl9wlkba5yjl29wgodb6d1y0d 2463517 2463516 2026-10-05T13:05:50Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" 2463517 wikitext text/x-wiki et6o33t2m9x9im0079aswv92mk72sbj 2463518 2463517 2026-10-05T13:05:51Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2463518 wikitext text/x-wiki owmt6n7ojzms9rw4lfln61ecrxq0ib5 2463519 2463518 2026-10-05T13:05:54Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1349752|throttle.php: Lift IP cap for edit-a-thon (T439991)]] 2463519 wikitext text/x-wiki csxmnk4kjollpuarqnrvtwgiqdt7ldm 2463520 2463519 2026-10-05T13:05:55Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet 2463520 wikitext text/x-wiki 5u7wqby8yzejs26b7q55du3n6bq9aai 2463522 2463520 2026-10-05T13:10:21Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet 2463522 wikitext text/x-wiki sf1pg1awj6pzx856c7vaqom1nwtugi3 2463524 2463522 2026-10-05T13:11:55Z Stashbot 7414 zabe@deploy1003: zabe, tryvix1509: Backport for [[gerrit:1349752|throttle.php: Lift IP cap for edit-a-thon (T439991)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463524 wikitext text/x-wiki lohit47lup569enejuaqa8n27qpjht1 2463526 2463524 2026-10-05T13:12:03Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet 2463526 wikitext text/x-wiki k58w7tascjvwwk5n2ta7e4uydr2z5m9 2463527 2463526 2026-10-05T13:12:07Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet 2463527 wikitext text/x-wiki 8eh1vcn60bfu37obwf5bbmu6x2p0xij 2463528 2463527 2026-10-05T13:12:54Z Stashbot 7414 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment 2463528 wikitext text/x-wiki pimrehxp3qtrrw50w53su94vhzb2ktj 2463530 2463528 2026-10-05T13:14:50Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2463530 wikitext text/x-wiki 3qmk34ps13pye3ebtkxpij1pbmjizbg 2463532 2463530 2026-10-05T13:19:01Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2463532 wikitext text/x-wiki 283syuf8f24n0s2a2ul7rs6ggjq4l6r 2463533 2463532 2026-10-05T13:19:13Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet 2463533 wikitext text/x-wiki qmlka6j22j3udf120y3vu1d19822pmc 2463534 2463533 2026-10-05T13:19:16Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet 2463534 wikitext text/x-wiki 3oclkg6u9swz21a4z8yi0p6dxw8xx8i 2463535 2463534 2026-10-05T13:20:09Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2463535 wikitext text/x-wiki 6hubilwqkrjszos8wvsns9359py36ic 2463536 2463535 2026-10-05T13:20:13Z Stashbot 7414 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463536 wikitext text/x-wiki tev9vlkbnyd5bwhclmawygur8scdil3 2463537 2463536 2026-10-05T13:20:35Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet 2463537 wikitext text/x-wiki 38ceqf31g75n8hn8m6915oy3je4kx31 2463538 2463537 2026-10-05T13:20:36Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet 2463538 wikitext text/x-wiki kb93wrxy1f8np6g53gyjs5oge7urf7w 2463539 2463538 2026-10-05T13:20:41Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1349752|throttle.php: Lift IP cap for edit-a-thon (T439991)]] (duration: 14m 48s) 2463539 wikitext text/x-wiki e6wej5vs4icwuycrn8my27b93dk3x4i 2463540 2463539 2026-10-05T13:21:15Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346053|Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793|Update pulsating blue dot behavior for anonymous bookmark (T438779)]] 2463540 wikitext text/x-wiki squf3dd5rj8xnpdxucyg9rw6rez9dfd 2463541 2463540 2026-10-05T13:21:18Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2463541 wikitext text/x-wiki dyf6h8zeilnfa5e0tacwhq2eudedakf 2463542 2463541 2026-10-05T13:22:09Z Stashbot 7414 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 (T439667) 2463542 wikitext text/x-wiki clq29516cjzlfc229ikyqml1hsdigt6 2463543 2463542 2026-10-05T13:24:38Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2463543 wikitext text/x-wiki h4nh5wuz2fn85f399v9mqlpu6qj02qc 2463544 2463543 2026-10-05T13:24:55Z Stashbot 7414 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 (T439667) 2463544 wikitext text/x-wiki 42nku5kh1w4pu3hbqgxm61iqrdzkcvs 2463545 2463544 2026-10-05T13:25:29Z Stashbot 7414 zabe@deploy1003: jdlrobson, zabe, annet: Backport for [[gerrit:1346053|Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793|Update pulsating blue dot behavior for anonymous bookmark (T438779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463545 wikitext text/x-wiki kb9mlaelfp5o2hrgf3r421tvidxo67p 2463546 2463545 2026-10-05T13:25:35Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2463546 wikitext text/x-wiki babw2r3h27gce3j7ch0tvx1may3lj02 2463547 2463546 2026-10-05T13:26:18Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet 2463547 wikitext text/x-wiki 4qvnhg0iv7uvjtfv2m388v4mtusafie 2463548 2463547 2026-10-05T13:26:29Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet 2463548 wikitext text/x-wiki 7hcjt4zqjqzg1r40zox1qsp7io1ramh 2463549 2463548 2026-10-05T13:27:07Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet 2463549 wikitext text/x-wiki 03g4tja0v7ryp2xchjiis9jutrf3761 2463550 2463549 2026-10-05T13:27:27Z Stashbot 7414 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment 2463550 wikitext text/x-wiki sjgw15io8cyrutavo9269sqqqhnppca 2463552 2463550 2026-10-05T13:28:44Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463552 wikitext text/x-wiki kw75xue8tcifu0khlfpb2m2exaaf1zm 2463553 2463552 2026-10-05T13:32:38Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346053|Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793|Update pulsating blue dot behavior for anonymous bookmark (T438779)]] (duration: 11m 23s) 2463553 wikitext text/x-wiki lxo3zn0ldenw2mxtvjfsgtv7vz9rt7q 2463554 2463553 2026-10-05T13:32:59Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1351194|Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753)]] 2463554 wikitext text/x-wiki ezhpdp22b0aq1hw80kd0pn1sce6pjtj 2463556 2463554 2026-10-05T13:34:26Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463556 wikitext text/x-wiki 13vbjkrw1ly7byphb3tb6ftxqnpqw1a 2463557 2463556 2026-10-05T13:34:31Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet 2463557 wikitext text/x-wiki qir8terfaqk6p07l19kgbdrr11048g3 2463558 2463557 2026-10-05T13:35:29Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.dns.netbox 2463558 wikitext text/x-wiki c8cajg0pqgho97248d5nmtz8yd2s1up 2463559 2463558 2026-10-05T13:36:33Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet 2463559 wikitext text/x-wiki kn4hc96rrecdgr25hqyl3e0k0p9gacj 2463560 2463559 2026-10-05T13:36:58Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1351194|Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463560 wikitext text/x-wiki 8pomdg0a7sxih08nphu1reln3mw2nuz 2463561 2463560 2026-10-05T13:37:08Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet 2463561 wikitext text/x-wiki 3iqu37gwn089ro05pqw3fwvwsyv9db1 2463562 2463561 2026-10-05T13:37:15Z Stashbot 7414 elukey: restart pybal on lvs201[34] to pick up a new LVS service 2463562 wikitext text/x-wiki ik6yuhugs6tfgzlz585095cl1o5uqk6 2463563 2463562 2026-10-05T13:37:38Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with deployment 2463563 wikitext text/x-wiki 0dvjd6tmnvkgafbwzu0dh6pum6ppr7q 2463564 2463563 2026-10-05T13:38:02Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet 2463564 wikitext text/x-wiki mdddfk1x67d196tr3yh5krvl0hb2mao 2463565 2463564 2026-10-05T13:38:16Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet 2463565 wikitext text/x-wiki 5j31mmr473lsl7jzyo4lt1zu5i7p8sq 2463566 2463565 2026-10-05T13:38:24Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet 2463566 wikitext text/x-wiki 7fp4bl8vmmzeq2focyzhop1944cafoy 2463567 2463566 2026-10-05T13:39:23Z Stashbot 7414 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" 2463567 wikitext text/x-wiki ol2fyzzpfbapxq7b307ucd38xr3eqa3 2463568 2463567 2026-10-05T13:39:24Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet 2463568 wikitext text/x-wiki ecv6ql737nbv4idsefub1uc6b9fejqs 2463569 2463568 2026-10-05T13:39:37Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" 2463569 wikitext text/x-wiki 8d345nm48cm8itqtl9w78r01uy2jzij 2463570 2463569 2026-10-05T13:39:38Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2463570 wikitext text/x-wiki 967u572pee32ki1hlbkdtzmxbt88g2u 2463571 2463570 2026-10-05T13:39:39Z Stashbot 7414 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet 2463571 wikitext text/x-wiki qu5n88tgdupbvfk6ggl1qqz6j7zye4r 2463572 2463571 2026-10-05T13:39:58Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet 2463572 wikitext text/x-wiki tuggq9vo82s9cjqpveat50ue50jco6p 2463573 2463572 2026-10-05T13:40:29Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463573 wikitext text/x-wiki mo1wb92vgdak9yrfms51fqfmgb83417 2463577 2463573 2026-10-05T13:42:10Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2463577 wikitext text/x-wiki gzjjiulrodhpdtxlszhugaraukacenx 2463578 2463577 2026-10-05T13:42:41Z Stashbot 7414 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service 2463578 wikitext text/x-wiki kichu3ciy6fihy90yo12sjd50kgx61k 2463579 2463578 2026-10-05T13:42:58Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351194|Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753)]] (duration: 09m 59s) 2463579 wikitext text/x-wiki 53cxedhuoxkfnrgjegguw3rvolbr44v 2463580 2463579 2026-10-05T13:45:57Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463580 wikitext text/x-wiki 462hw940tvd30yxblbr1j66i8hwpr88 2463581 2463580 2026-10-05T13:46:31Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet 2463581 wikitext text/x-wiki 0byqi74n7neglqqign01crxy3sresgv 2463582 2463581 2026-10-05T13:48:52Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet 2463582 wikitext text/x-wiki b68tt1von31bkgp55cs6sb2fywvp2bj 2463583 2463582 2026-10-05T13:49:28Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet 2463583 wikitext text/x-wiki fo1qeqjlp5tq16npho9216j3oy5wvtm 2463584 2463583 2026-10-05T13:50:20Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet 2463584 wikitext text/x-wiki b8llxbj319kg2weyyf8rzz7vvdtqsye 2463585 2463584 2026-10-05T13:51:12Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet 2463585 wikitext text/x-wiki 1mxx4atdplo9udo7dvct3fh36ihswr1 2463586 2463585 2026-10-05T13:51:15Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet 2463586 wikitext text/x-wiki 21453ymcqso0ukeolmgi0q7j1dobk3v 2463587 2463586 2026-10-05T13:53:25Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet 2463587 wikitext text/x-wiki 45n56zj71tn3aukznysfcy86i78x8dy 2463588 2463587 2026-10-05T13:54:01Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet 2463588 wikitext text/x-wiki 4tazvrcedn3hv3x2rx100d55j41tv1z 2463589 2463588 2026-10-05T13:54:54Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2463589 wikitext text/x-wiki 3zmx1adkd6jjltwviuo7jjuf9v286rh 2463591 2463589 2026-10-05T13:56:21Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2463591 wikitext text/x-wiki 9lizk2p5svdl0k46m98by9kwi5l80hf 2463595 2463591 2026-10-05T13:57:18Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463595 wikitext text/x-wiki isb9x6pz91rmlk1juf08ass9w3apixx 2463596 2463595 2026-10-05T13:57:19Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2463596 wikitext text/x-wiki c6losxj1gj04pf3osd3cl7nqtxxh6f6 2463598 2463596 2026-10-05T13:58:53Z Stashbot 7414 logmsgbot: dreamyjazz Deployed security patch for T439968 2463598 wikitext text/x-wiki rm5m7nfeptk6iamizdd7hgn3u84ircv 2463599 2463598 2026-10-05T14:03:05Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463599 wikitext text/x-wiki c3ilyth8aym3ldngb61xfbz9pqcprg0 2463600 2463599 2026-10-05T14:03:19Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync 2463600 wikitext text/x-wiki 61wph3ff4rjrpsqmsb8aqvfbim8q3si 2463601 2463600 2026-10-05T14:03:39Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet 2463601 wikitext text/x-wiki adjnrylawfa6fjhu461iwt7uydgxapr 2463602 2463601 2026-10-05T14:03:42Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet 2463602 wikitext text/x-wiki m69rhgfz0t0slfpupav8nyp06bsblh3 2463603 2463602 2026-10-05T14:03:47Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync 2463603 wikitext text/x-wiki kw6u1woe4h4t90zt68q890vc817nu6t 2463604 2463603 2026-10-05T14:03:52Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet 2463604 wikitext text/x-wiki 2yxbwfn6m35zi5henqbbupc866fi5lq 2463605 2463604 2026-10-05T14:04:28Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet 2463605 wikitext text/x-wiki 5r3femy8f0uz42azqexup8nklb4l2e8 2463609 2463605 2026-10-05T14:07:22Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm 2463609 wikitext text/x-wiki aj3uqs4wljcic5zpub6wyjpitit4zfb 2463610 2463609 2026-10-05T14:07:43Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm 2463610 wikitext text/x-wiki p6i852vx8szdxgf0os8pfx69co9297w 2463611 2463610 2026-10-05T14:09:20Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351207|search: test impact of the MLR window size (T438984)]] 2463611 wikitext text/x-wiki 0meqgtkamp8iwau1r5zeyg07owv5526 2463612 2463611 2026-10-05T14:10:00Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463612 wikitext text/x-wiki j1ctz13os1ublatsndhcoixouwcaqyf 2463613 2463612 2026-10-05T14:12:11Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2463613 wikitext text/x-wiki 4unnm5bzeuemabewnf058r477ue1ghq 2463614 2463613 2026-10-05T14:12:27Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2463614 wikitext text/x-wiki lzqvgcfe0txzn9mkevh2he5l4l6je4x 2463615 2463614 2026-10-05T14:13:23Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351207|search: test impact of the MLR window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463615 wikitext text/x-wiki qtyqg97sl5l4qsh6dey4vs7wvnc7pbg 2463616 2463615 2026-10-05T14:15:15Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2463616 wikitext text/x-wiki 3jp0uv8ssgmcr3kb7k5a68jbd8ieeko 2463618 2463616 2026-10-05T14:16:19Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463618 wikitext text/x-wiki 17yg6rt2tzxlpwjo35ran9svj77knkm 2463619 2463618 2026-10-05T14:16:54Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet 2463619 wikitext text/x-wiki 55vog3ya8qgxgzfpug12qfneltpkm0e 2463620 2463619 2026-10-05T14:16:57Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet 2463620 wikitext text/x-wiki jpk0tl3z7kkqxgzsi2x94e7jdajm7yd 2463621 2463620 2026-10-05T14:17:06Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet 2463621 wikitext text/x-wiki 0tazdtf42geuya6q73kenf51uql7uky 2463623 2463621 2026-10-05T14:18:16Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet 2463623 wikitext text/x-wiki kvey3f8kaykmc8tiplmng307q0yc5is 2463624 2463623 2026-10-05T14:18:33Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463624 wikitext text/x-wiki prr68buvoay3t6nj7p8235lsm17fwpq 2463625 2463624 2026-10-05T14:20:07Z Stashbot 7414 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm 2463625 wikitext text/x-wiki trcru5ntmzvmyv0cbpixsh4kamlxo2k 2463626 2463625 2026-10-05T14:20:23Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm 2463626 wikitext text/x-wiki eyvg2bdr0frrw1mpppuq92ygx9vk4bq 2463627 2463626 2026-10-05T14:20:32Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351207|search: test impact of the MLR window size (T438984)]] (duration: 11m 12s) 2463627 wikitext text/x-wiki 6t9dlc6u5uh18d2od8wnj7ho7tc56k3 2463628 2463627 2026-10-05T14:24:51Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463628 wikitext text/x-wiki tirra29uw9mof2um7mdcgor30f52kve 2463629 2463628 2026-10-05T14:25:27Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet 2463629 wikitext text/x-wiki cpwyrob36w2roc0elj0kw7okudplnt9 2463630 2463629 2026-10-05T14:25:29Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet 2463630 wikitext text/x-wiki jmt6npm7dx2eso3v5h2iyh1kgk1rcvb 2463631 2463630 2026-10-05T14:25:38Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet 2463631 wikitext text/x-wiki qrzwbm6mi5yb3dlbpn0kfa7jarqcxdk 2463632 2463631 2026-10-05T14:26:16Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet 2463632 wikitext text/x-wiki o9z00emwgn6898enwjef4eromsaq9qs 2463634 2463632 2026-10-05T14:27:36Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463634 wikitext text/x-wiki ryw2en0b9yasbmuxjppczy8lqpvou9d 2463635 2463634 2026-10-05T14:28:39Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351208|Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222|search: test impact of MLR (T438984)]] 2463635 wikitext text/x-wiki jnreinl62kix0ql1pqc6vcup72ee63i 2463636 2463635 2026-10-05T14:30:18Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2463636 wikitext text/x-wiki pbl40pqsourxk5709ct95ze2h8n9o7p 2463637 2463636 2026-10-05T14:30:36Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2463637 wikitext text/x-wiki am72gtqx5atl2fcwg68an93yubeltbs 2463638 2463637 2026-10-05T14:32:28Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage 2463638 wikitext text/x-wiki mywag3jn6qig21klo6snt6pvkm11a2s 2463639 2463638 2026-10-05T14:32:44Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351208|Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222|search: test impact of MLR (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463639 wikitext text/x-wiki kdreybcvvog3u1lz30oqohqikgil9bk 2463640 2463639 2026-10-05T14:33:24Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2463640 wikitext text/x-wiki 1e78feb5c37cperxa01e3iujvmibusg 2463641 2463640 2026-10-05T14:33:25Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463641 wikitext text/x-wiki e7kzibkmbb8brabi8hamqwno7edmbvg 2463642 2463641 2026-10-05T14:33:59Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet 2463642 wikitext text/x-wiki en0m0ubntjt2jz6jmepouwzbc04t0if 2463643 2463642 2026-10-05T14:34:53Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet 2463643 wikitext text/x-wiki 4ngdz4g1qt4c1fmqnxbceq2ijg9ljmy 2463644 2463643 2026-10-05T14:35:28Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet 2463644 wikitext text/x-wiki 6evl37gqnnnbnlo4be5a79cnuc21ede 2463645 2463644 2026-10-05T14:35:31Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet 2463645 wikitext text/x-wiki ohxp1niynur48sszugihxho3mmdr7cb 2463646 2463645 2026-10-05T14:35:41Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet 2463646 wikitext text/x-wiki 8j49l52dgho9ncdilbmm0jdf6blok1q 2463647 2463646 2026-10-05T14:36:06Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox 2463647 wikitext text/x-wiki 8zbht4z4obukz5pt6mtlhw1swx2j9zv 2463648 2463647 2026-10-05T14:36:20Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet 2463648 wikitext text/x-wiki 6t8lqkiin3luasd2xtiery5igilhmct 2463649 2463648 2026-10-05T14:36:39Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463649 wikitext text/x-wiki bcfiwmh3d4kgj03xra3si2gg4v2fily 2463650 2463649 2026-10-05T14:37:16Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage 2463650 wikitext text/x-wiki anp56g2p8hfx7dcq9zkt4sacva20cs9 2463651 2463650 2026-10-05T14:37:42Z Stashbot 7414 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2463651 wikitext text/x-wiki 1n8d3w3ylcjbpsfzvcv45fd2t11gpr8 2463652 2463651 2026-10-05T14:38:04Z Stashbot 7414 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2463652 wikitext text/x-wiki lzghfspy60d82hvplp7591z58e38zdo 2463653 2463652 2026-10-05T14:38:19Z Stashbot 7414 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2463653 wikitext text/x-wiki svtglegv4cincd6ajxm17u3eqk0ntzc 2463654 2463653 2026-10-05T14:38:43Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351208|Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222|search: test impact of MLR (T438984)]] (duration: 10m 04s) 2463654 wikitext text/x-wiki hlgyacokc1r1q11j4hljmmylyl2hvw4 2463655 2463654 2026-10-05T14:38:46Z Stashbot 7414 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2463655 wikitext text/x-wiki 8z3t16y37222sgccbgj9isgc9pitic0 2463656 2463655 2026-10-05T14:42:49Z Stashbot 7414 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 (T439667) 2463656 wikitext text/x-wiki 2houvikuyakq1z8jqi747z9mdgmbspp 2463657 2463656 2026-10-05T14:43:03Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2463657 wikitext text/x-wiki n5ik8cxoi3iiom8kwosb0v3czh13qn7 2463658 2463657 2026-10-05T14:43:40Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2463658 wikitext text/x-wiki ijjwp8y0ax05h09zvm6x5r7vbbcpxiq 2463659 2463658 2026-10-05T14:43:41Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2463659 wikitext text/x-wiki 2wms4t05gf3ee91x3br0yh13cx1eeqa 2463660 2463659 2026-10-05T14:44:24Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2463660 wikitext text/x-wiki 2e6iiaao8dmgzpciei9tgkv8bnmfs2i 2463661 2463660 2026-10-05T14:44:32Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s) 2463661 wikitext text/x-wiki apj03lwst00aelb80ia6f3l501w6t2m 2463662 2463661 2026-10-05T14:45:25Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463662 wikitext text/x-wiki 711l3yj0x0zihy1sux19zscmt42aucq 2463663 2463662 2026-10-05T14:45:39Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2463663 wikitext text/x-wiki 7abd812uz15u2j2h0wkhzxk1feunlr7 2463664 2463663 2026-10-05T14:45:42Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2463664 wikitext text/x-wiki 2g7j9a6yivcnkp4av2dn07xo50k2qpp 2463665 2463664 2026-10-05T14:45:43Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2463665 wikitext text/x-wiki aa8om2k1pr7g59s4n95rgxnuudfbkbh 2463666 2463665 2026-10-05T14:45:46Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2463666 wikitext text/x-wiki laamizaqbcwwbvu30yic8uht25z06cq 2463667 2463666 2026-10-05T14:46:00Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet 2463667 wikitext text/x-wiki k61pfk5bc21i2c86zxy25ki2ymitgb5 2463668 2463667 2026-10-05T14:46:15Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2463668 wikitext text/x-wiki 0cm9exf77ixbrw2mgwieidjdrsheich 2463669 2463668 2026-10-05T14:46:28Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts 2463669 wikitext text/x-wiki l0kwdc2zenge3p09lt5c92q08d9nl7o 2463670 2463669 2026-10-05T14:46:53Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2463670 wikitext text/x-wiki pme9x2lu8p1yfhnzlmutmkj2om8ys6y 2463671 2463670 2026-10-05T14:46:54Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet 2463671 wikitext text/x-wiki 6bkj9yv9udob8fj93uymv0yhxdcyt7i 2463672 2463671 2026-10-05T14:46:58Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2463672 wikitext text/x-wiki n93cirbulgxxomyw0cdgd739x1fo6ye 2463673 2463672 2026-10-05T14:47:08Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2463673 wikitext text/x-wiki b9u6c1hm9l3un8tu3r815dutaj67qik 2463674 2463673 2026-10-05T14:47:29Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet 2463674 wikitext text/x-wiki e94t5d48tjq0z976mz0nzxptnf63fiu 2463675 2463674 2026-10-05T14:47:32Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet 2463675 wikitext text/x-wiki oi6uky2pgxpeaxe6d9tk1cy3px0tucu 2463676 2463675 2026-10-05T14:47:40Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet 2463676 wikitext text/x-wiki t1atzj3rhgpny1vdgr3samp6mtr6ydb 2463677 2463676 2026-10-05T14:47:54Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463677 wikitext text/x-wiki i0l966r79k83kwchvcirsa8igqtd244 2463678 2463677 2026-10-05T14:48:05Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463678 wikitext text/x-wiki rnn08zix3yn5js5gy8rgdbs6exw3ylr 2463679 2463678 2026-10-05T14:48:15Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet 2463679 wikitext text/x-wiki r7wv1iw8gc2slk9w6lu8p2t5ufp3rev 2463680 2463679 2026-10-05T14:48:44Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply 2463680 wikitext text/x-wiki cgll7f64ecqpeik688d7rky475gmngu 2463681 2463680 2026-10-05T14:48:46Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351223|Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209|search: test impact of the function score window size (T438984)]] 2463681 wikitext text/x-wiki lj2o843ptef4n96fguh78t83fj94bk2 2463682 2463681 2026-10-05T14:48:53Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply 2463682 wikitext text/x-wiki 3rpi6rd6ak6fhvowmt7qd7zf9mgrkir 2463684 2463682 2026-10-05T14:49:23Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2463684 wikitext text/x-wiki 7nk6p6gddpx8lqlm60m9kviyos8xbg4 2463685 2463684 2026-10-05T14:49:26Z Stashbot 7414 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2463685 wikitext text/x-wiki 3pdbaoxh7eflfijwiqr3se2oor86fjx 2463686 2463685 2026-10-05T14:49:27Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2463686 wikitext text/x-wiki 3ithilw3cc1kzqj1iy5h3g807r6wxoi 2463687 2463686 2026-10-05T14:49:30Z Stashbot 7414 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2463687 wikitext text/x-wiki 9quay3k7aozmrkdpa3z9qlj7o9onrzm 2463688 2463687 2026-10-05T14:52:38Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463688 wikitext text/x-wiki skytn4owlwo5w2llwkomikrjdrb0swl 2463689 2463688 2026-10-05T14:52:57Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351223|Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209|search: test impact of the function score window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463689 wikitext text/x-wiki 3r6nzlgls8h9pcqsxkue4le1p58o55t 2463690 2463689 2026-10-05T14:53:18Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463690 wikitext text/x-wiki pkdrp2rjsjby6xygw41c7kwe9nb8cd0 2463691 2463690 2026-10-05T14:53:21Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463691 wikitext text/x-wiki 3v2r7qfzckkun6hqwbpz6t5hn2vn5js 2463692 2463691 2026-10-05T14:53:27Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2463692 wikitext text/x-wiki sxms1rm0c7voukgcbf7l5azf8jrz9oo 2463695 2463692 2026-10-05T14:58:58Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351223|Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209|search: test impact of the function score window size (T438984)]] (duration: 10m 12s) 2463695 wikitext text/x-wiki p77u1jb5sc8oayn1h1b1a5nl8gfidxp 2463696 2463695 2026-10-05T15:00:20Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm 2463696 wikitext text/x-wiki 070v9vyj8krgfqkh74m1ddfnukl73qi 2463697 2463696 2026-10-05T15:00:40Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463697 wikitext text/x-wiki tdnvh2lhdwxyu096z2vimjau24ruops 2463698 2463697 2026-10-05T15:01:15Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet 2463698 wikitext text/x-wiki 1740osj6wz76ossvsknqvp98656xzom 2463699 2463698 2026-10-05T15:01:18Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet 2463699 wikitext text/x-wiki 79qpcqjde22k3qva1v4cink4qey7u9f 2463700 2463699 2026-10-05T15:01:26Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet 2463700 wikitext text/x-wiki hixv2hxe3smyafqoprjoxm2yzyb9ggm 2463701 2463700 2026-10-05T15:02:06Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet 2463701 wikitext text/x-wiki glac2a8j4rfkutj99eo6tjuwrvri2b6 2463702 2463701 2026-10-05T15:02:17Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463702 wikitext text/x-wiki bieh0osdl5eh0acenzaqw9me2evz6k1 2463703 2463702 2026-10-05T15:02:23Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463703 wikitext text/x-wiki 75ddcidjnmtjk7rd0n92ruxu7oygcos 2463705 2463703 2026-10-05T15:02:28Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463705 wikitext text/x-wiki 70jo5019skgbpgdbjagnwhc7wvmyahy 2463707 2463705 2026-10-05T15:05:12Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351210|Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211|search: test impact of cross project searches (T438984)]] 2463707 wikitext text/x-wiki 16to3squ554gk9chwkw5vy6c19mqurp 2463708 2463707 2026-10-05T15:09:09Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351210|Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211|search: test impact of cross project searches (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463708 wikitext text/x-wiki 5130xjinc6f5bqvhakuftunfhl9wmq8 2463709 2463708 2026-10-05T15:10:07Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2463709 wikitext text/x-wiki l7xgyqs7bgazve58sjpojahzb9bjjzc 2463710 2463709 2026-10-05T15:10:09Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463710 wikitext text/x-wiki drfcjcle1m6ce8k2hi0pytd82qt5t9k 2463711 2463710 2026-10-05T15:10:44Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet 2463711 wikitext text/x-wiki bbxyej6qtzuagoyx6al2ree8qskcdh0 2463712 2463711 2026-10-05T15:11:39Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet 2463712 wikitext text/x-wiki 6uo9rjttfbi7yto6zv6k0pdgmpkbgis 2463713 2463712 2026-10-05T15:11:48Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet 2463713 wikitext text/x-wiki le77tytd7zbv0uxp0jwfwcbxlvyputt 2463716 2463713 2026-10-05T15:12:10Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2463716 wikitext text/x-wiki 73y0aa61cxunx9rdi4cshq0vf0drlcc 2463717 2463716 2026-10-05T15:12:11Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463717 wikitext text/x-wiki o28vccb6g63zat00d95irlhbo8j9qhg 2463718 2463717 2026-10-05T15:12:14Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463718 wikitext text/x-wiki tdz3wgzulm5mfbpo4p21298ur61m24k 2463719 2463718 2026-10-05T15:12:24Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet 2463719 wikitext text/x-wiki pa222b1a15y0ic57aiy0d1zaneaejjn 2463720 2463719 2026-10-05T15:12:25Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463720 wikitext text/x-wiki om1o05pitaon2j231dxreu3lgioues0 2463721 2463720 2026-10-05T15:12:29Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2463721 wikitext text/x-wiki eekorfbo0n8ip73h1giwtc8r7plhsy4 2463722 2463721 2026-10-05T15:12:36Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463722 wikitext text/x-wiki av05k6f523terfsxj3c3hznudlyyrhh 2463724 2463722 2026-10-05T15:14:25Z Stashbot 7414 elukey@dns1004: START - running authdns-update 2463724 wikitext text/x-wiki 744veb9e06j2x4xebeoewrjp2m8z0pa 2463725 2463724 2026-10-05T15:15:23Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351210|Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211|search: test impact of cross project searches (T438984)]] (duration: 10m 11s) 2463725 wikitext text/x-wiki t13eb1wkjgwk87k6ky39xrxuvwdprkv 2463726 2463725 2026-10-05T15:16:43Z Stashbot 7414 elukey@dns1004: END - running authdns-update 2463726 wikitext text/x-wiki m5uwr236lku7gpvw3f0bi681xagu0ga 2463727 2463726 2026-10-05T15:16:46Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm 2463727 wikitext text/x-wiki 21zoxlk2cv56wqrljrpbp1yzuzb3rfu 2463728 2463727 2026-10-05T15:17:26Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2463728 wikitext text/x-wiki kcmub9nx1sc7ul5hm1ico6ni26ticc3 2463729 2463728 2026-10-05T15:18:03Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2463729 wikitext text/x-wiki 9oc7skchr7480jrk9mpolmo2n9kxijv 2463731 2463729 2026-10-05T15:20:07Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463731 wikitext text/x-wiki huuw5arevdbokc11pvqvr017wad46xs 2463732 2463731 2026-10-05T15:20:42Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet 2463732 wikitext text/x-wiki soti1khgnaf1yf2q2k8y3ms3xg8k4kc 2463733 2463732 2026-10-05T15:21:36Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet 2463733 wikitext text/x-wiki hppym92ykda60gynmr964uias8qakcu 2463734 2463733 2026-10-05T15:22:11Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet 2463734 wikitext text/x-wiki 1zmbb41ts1774607ht8889d3ga5cjlu 2463735 2463734 2026-10-05T15:22:14Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet 2463735 wikitext text/x-wiki epu11tqcf4xlrh1ytxxpxv34hsimii5 2463736 2463735 2026-10-05T15:22:22Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet 2463736 wikitext text/x-wiki 1av8xkr8qh52p06a046w6embyqpyn7e 2463737 2463736 2026-10-05T15:22:27Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351212|Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213|search: measure impact of enwiki (T438984)]] 2463737 wikitext text/x-wiki 867d27tcokitxtnh3akmonw8biy79ft 2463738 2463737 2026-10-05T15:22:57Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet 2463738 wikitext text/x-wiki l6u1afzi14vipctp8mpivhk61bsibjd 2463739 2463738 2026-10-05T15:23:33Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463739 wikitext text/x-wiki g1ibmglr6keglzyz8lc4of8o0man5gy 2463740 2463739 2026-10-05T15:26:35Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351212|Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213|search: measure impact of enwiki (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463740 wikitext text/x-wiki 8vbivfhxvfvp6gd26axmx8q7py7qa45 2463742 2463740 2026-10-05T15:27:45Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2463742 wikitext text/x-wiki nh4qeib9ou8lgdoyfhot4sr05gmo2k2 2463744 2463742 2026-10-05T15:28:55Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2463744 wikitext text/x-wiki 1drcz22nqwupikhz41gt9xq6xnmvdcn 2463745 2463744 2026-10-05T15:29:11Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage 2463745 wikitext text/x-wiki r6o7o94q4p8xvb2el5gm3uf48adsdzt 2463746 2463745 2026-10-05T15:29:13Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2463746 wikitext text/x-wiki tnp9ysm77i4vlr46ol5jdm4jtvq506w 2463747 2463746 2026-10-05T15:29:30Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet 2463747 wikitext text/x-wiki 06ze5428q6f7gdbgqf2416oiu5gu5vq 2463748 2463747 2026-10-05T15:29:44Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2463748 wikitext text/x-wiki frfib1x3ii3k0oilggfy48sbixc66qh 2463749 2463748 2026-10-05T15:30:08Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. 2463749 wikitext text/x-wiki 6u0ny9is7m5g3pkxbgd6s0vr08cm2oz 2463752 2463749 2026-10-05T15:30:51Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. 2463752 wikitext text/x-wiki 3o7ra7dzt6gf1fjpphtztjgy9seom95 2463753 2463752 2026-10-05T15:32:05Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage 2463753 wikitext text/x-wiki iljt90eijdjyo6gf2j5eab284174lce 2463754 2463753 2026-10-05T15:32:10Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet 2463754 wikitext text/x-wiki 0nijalo5l0lio0ew9adwg4z6gwjb4v6 2463755 2463754 2026-10-05T15:32:22Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2463755 wikitext text/x-wiki aed7bveheiyr1hv1vo5ezhndibtia71 2463756 2463755 2026-10-05T15:32:25Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2463756 wikitext text/x-wiki q2uzta0kvy06wors03a67g5id6bg96y 2463757 2463756 2026-10-05T15:32:44Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet 2463757 wikitext text/x-wiki b8s701upyuzxf3zi5zullk586p0eu7c 2463758 2463757 2026-10-05T15:32:54Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. 2463758 wikitext text/x-wiki 2qdxcb124ldlx0lmgu0ht1lbfvtyecb 2463759 2463758 2026-10-05T15:32:59Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. 2463759 wikitext text/x-wiki 1am5z64bub3l5qrqlpnki0ve4dzq2x7 2463760 2463759 2026-10-05T15:33:38Z Stashbot 7414 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet 2463760 wikitext text/x-wiki krinzzxhptbtg1nm9i8uo2sttub2job 2463761 2463760 2026-10-05T15:35:05Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox 2463761 wikitext text/x-wiki 1qz6by517ln88aon99dzkzcz5ko1eyw 2463762 2463761 2026-10-05T15:47:14Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet 2463762 wikitext text/x-wiki q6ycdvvbryuv93xgp90ym5a8ba8sz1y 2463763 2463762 2026-10-05T15:47:17Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet 2463763 wikitext text/x-wiki mpite3ep8pn5yh5ke27w6a4mkim03bs 2463764 2463763 2026-10-05T15:47:22Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm 2463764 wikitext text/x-wiki tb4l7cl6x0weupp4c3q2ouwri610yxs 2463765 2463764 2026-10-05T15:47:39Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351212|Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213|search: measure impact of enwiki (T438984)]] (duration: 25m 12s) 2463765 wikitext text/x-wiki 6l0i3i37rqw04h6u23n42rrexj3163y 2463766 2463765 2026-10-05T15:51:17Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad 2463766 wikitext text/x-wiki nu9yfuhrvy1550y100n6gzkp34p1j9e 2463767 2463766 2026-10-05T15:54:38Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351214|Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215|search: test perf of moving completion out of eqiad (T438984)]] 2463767 wikitext text/x-wiki k1hkrb1z41fdr6jcvq5fehjed4ddc64 2463769 2463767 2026-10-05T15:58:38Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351214|Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215|search: test perf of moving completion out of eqiad (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463769 wikitext text/x-wiki 5mb2fuwh79erabwgebhp9e0q17xuvex 2463770 2463769 2026-10-05T15:59:55Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2463770 wikitext text/x-wiki brbdoy4qpbdbeqqa0bkeet8zmobwjsn 2463771 2463770 2026-10-05T16:05:22Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351214|Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215|search: test perf of moving completion out of eqiad (T438984)]] (duration: 10m 44s) 2463771 wikitext text/x-wiki otdhsr5upp40jwr0qws4ndftuumb7ji 2463774 2463771 2026-10-05T16:16:30Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad 2463774 wikitext text/x-wiki dnsxf02x1ntnmx9qt6bjfef4t31mgo3 2463775 2463774 2026-10-05T16:19:33Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351216|Revert "search: test perf of moving completion out of eqiad" (T438984)]] 2463775 wikitext text/x-wiki l3jrbp12lwl6e98pzatuyoad1paes1f 2463776 2463775 2026-10-05T16:19:46Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox 2463776 wikitext text/x-wiki 3hf8aaarlzw6ci0v1wl5m5jl5epltmb 2463778 2463776 2026-10-05T16:23:41Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351216|Revert "search: test perf of moving completion out of eqiad" (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463778 wikitext text/x-wiki p6hq7ezznzmurfgimaqwh3j7a50jzkx 2463779 2463778 2026-10-05T16:25:02Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2463779 wikitext text/x-wiki igelvrciybsxqeukuuuh8akcft95enp 2463780 2463779 2026-10-05T16:30:21Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351216|Revert "search: test perf of moving completion out of eqiad" (T438984)]] (duration: 10m 48s) 2463780 wikitext text/x-wiki qc5u2u6u8sntrzwu5n0bqxxgq7eiwzz 2463781 2463780 2026-10-05T16:32:59Z Stashbot 7414 dcausse: closing the "Distributed search queries experiment" window 2463781 wikitext text/x-wiki 7jwziid8f336qui4dqqq9axugicici0 2463782 2463781 2026-10-05T16:50:39Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru 2463782 wikitext text/x-wiki 0m3pjqj8lwgi3c6pags2p9pjdb1bnoz 2463784 2463782 2026-10-05T17:11:36Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2463784 wikitext text/x-wiki hzwgx8r4plkjy2263kablhdqctw33yk 2463785 2463784 2026-10-05T17:12:03Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2463785 wikitext text/x-wiki r5jak5bg5jbfmhtvfimqdkx4v9qulil 2463786 2463785 2026-10-05T17:13:21Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply 2463786 wikitext text/x-wiki he2k7bievxg90gv0h7fafcvulbc3erb 2463787 2463786 2026-10-05T17:14:27Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply 2463787 wikitext text/x-wiki rwsimfmaeqo3z84mp5s0r5baaruprhi 2463788 2463787 2026-10-05T17:15:09Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2463788 wikitext text/x-wiki pgvgwwsb3pzix2g91knfo6433aj9fch 2463789 2463788 2026-10-05T17:15:26Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2463789 wikitext text/x-wiki l6fgrnlzhrytqt43cetvf9vx5o71edz 2463790 2463789 2026-10-05T17:15:39Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply 2463790 wikitext text/x-wiki pt6w1oxg9y3fol69jacf8n7j04ti783 2463791 2463790 2026-10-05T17:15:55Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru 2463791 wikitext text/x-wiki t72ynsqjzr8uvamphvuldyhkm9ojut4 2463792 2463791 2026-10-05T17:16:09Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo 2463792 wikitext text/x-wiki kms00t7kbf72o9mwwt3vi9dw065ttu0 2463793 2463792 2026-10-05T17:16:47Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply 2463793 wikitext text/x-wiki nuhvu0qxwviabpe3qzr5ioprzpeqsct 2463796 2463793 2026-10-05T17:27:54Z Stashbot 7414 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes 2463796 wikitext text/x-wiki 8i0s4z9nxnrjmwqrhpl0j6sshr5tx5f 2463797 2463796 2026-10-05T17:27:56Z Stashbot 7414 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2463797 wikitext text/x-wiki l3nzgflezwa8wciapiljw1pw5bnzvmj 2463798 2463797 2026-10-05T17:28:17Z Stashbot 7414 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2463798 wikitext text/x-wiki gy1aehc2fh6m527n3eoq8xxg6qymnol 2463799 2463798 2026-10-05T17:29:05Z Stashbot 7414 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video 2463799 wikitext text/x-wiki s5q6jltfpwk8pxem5sbtox4k9uct59f 2463800 2463799 2026-10-05T17:29:25Z Stashbot 7414 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s) 2463800 wikitext text/x-wiki 8jhf0ypqdu29bgnq5nnh6ksq920gs2t 2463801 2463800 2026-10-05T17:33:46Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # T439577 2463801 wikitext text/x-wiki 9uvg9ebs916c240r2li1x332krww2t3 2463802 2463801 2026-10-05T17:39:53Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2463802 wikitext text/x-wiki m7iscmadv79rrywky2yupky5pcg540e 2463803 2463802 2026-10-05T17:40:08Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2463803 wikitext text/x-wiki owmi5vi9ki5hzmygeptixbak41m7jof 2463804 2463803 2026-10-05T17:41:07Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo 2463804 wikitext text/x-wiki b7ko8bqg8xxgc4gccur0uqkvrrf27zh 2463805 2463804 2026-10-05T17:42:43Z Stashbot 7414 sbassett: deployed csv file to security.wikimedia.org (miscweb) 2463805 wikitext text/x-wiki ny5lgt5dtv3h0tv2roykss1yaci0ea5 2463806 2463805 2026-10-05T17:43:03Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo 2463806 wikitext text/x-wiki sk92jw3099z7rw6r6dgff1950edo1s4 2463808 2463806 2026-10-05T17:46:01Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2463808 wikitext text/x-wiki 355uln1kxgznebu0m4imtl83sb15oey 2463809 2463808 2026-10-05T17:46:43Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2463809 wikitext text/x-wiki 9p96uoc5211ue2z24l27gvecjgvs6e7 2463812 2463809 2026-10-05T17:52:59Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2463812 wikitext text/x-wiki a4rhc2ugm3m3t2zt5uu6o4l5xsvszi6 2463813 2463812 2026-10-05T17:53:15Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2463813 wikitext text/x-wiki raxh46qxak9v95nqggcfq90zjbjvaef 2463817 2463813 2026-10-05T17:57:43Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2463817 wikitext text/x-wiki rdwz1b1rh4pdnr6jd5u5dzfhe6k0l0g 2463818 2463817 2026-10-05T17:58:41Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2463818 wikitext text/x-wiki i3bw04kl7cmvhw63he2lhmlkeg2c0ew 2463819 2463818 2026-10-05T17:58:47Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2463819 wikitext text/x-wiki 24f0yt1srxdltadmf5ehd11eu3so0x9 2463820 2463819 2026-10-05T17:58:58Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2463820 wikitext text/x-wiki 57z05yydc2o6f7v3oh0w2cfb687qrjc 2463825 2463820 2026-10-05T18:08:21Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo 2463825 wikitext text/x-wiki oksenz4r9lx6jjp1qp0514hg7vxula8 2463826 2463825 2026-10-05T18:14:50Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs 2463826 wikitext text/x-wiki thgajll9dtydp4l1nnt8xoq3eh753zj 2463830 2463826 2026-10-05T18:23:09Z Stashbot 7414 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2463830 wikitext text/x-wiki h5ys2m5docfmh0bq3ergqfbqzzntgos 2463831 2463830 2026-10-05T18:39:52Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs 2463831 wikitext text/x-wiki fmc0o8rft6t8d612e1be6wwav3xlapd 2463832 2463831 2026-10-05T18:44:21Z Stashbot 7414 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams 2463832 wikitext text/x-wiki 8eaiau9je1qllcmdikgrjwr6i5cgbkk 2463833 2463832 2026-10-05T18:45:40Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2463833 wikitext text/x-wiki lv9expn18mgs1q9u82mkjwdmmj1mj97 2463834 2463833 2026-10-05T18:46:27Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2463834 wikitext text/x-wiki gjgrk6u922u4ie8vwvffaffsccsqazo 2463835 2463834 2026-10-05T18:58:06Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2463835 wikitext text/x-wiki jme9fmz7m823iv0lx4z243en3tc18fp 2463836 2463835 2026-10-05T18:58:14Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2463836 wikitext text/x-wiki pcnw2glue9xcnhmqql61flq1hbs0iw5 2463837 2463836 2026-10-05T19:09:24Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams 2463837 wikitext text/x-wiki ee729fzvrvcgwxk7jewk2yp48mb7dwl 2463841 2463837 2026-10-05T19:33:33Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2463841 wikitext text/x-wiki idudituo74a9r0787z2hf7u005q4z13 2463842 2463841 2026-10-05T19:33:47Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2463842 wikitext text/x-wiki 0w5y463ruyl1hqp2ir4c05s1su1o5kf 2463843 2463842 2026-10-05T19:51:57Z Stashbot 7414 logmsgbot: dreamyjazz Deployed security patch for T439932 2463843 wikitext text/x-wiki fvyg9l5rvp5siwggklgj04c1ze6uuk8 2463846 2463843 2026-10-05T20:01:13Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2463846 wikitext text/x-wiki glgqtdloa23usa1eh9xtgpnw17dulpa 2463847 2463846 2026-10-05T20:01:27Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2463847 wikitext text/x-wiki rkriy66xgkp6fadmt0hd1lscuasqey5 2463849 2463847 2026-10-05T20:09:40Z Stashbot 7414 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox 2463849 wikitext text/x-wiki ixwughkwx6fon4lx85n46qxsp3f1ylu 2463852 2463849 2026-10-05T20:45:42Z Stashbot 7414 logmsgbot: dreamyjazz Deployed security patch for T439974 2463852 wikitext text/x-wiki lfpgevyvacq3r1at3j5fb15ibx89vmb 2463853 2463852 2026-10-05T20:50:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. 2463853 wikitext text/x-wiki iw4y6p8p9cyoy6rhv1sqlp9opi91kfk 2463854 2463853 2026-10-05T20:58:43Z Stashbot 7414 logmsgbot: dreamyjazz Deployed security patch for T439719 2463854 wikitext text/x-wiki cvth1jyjvrenrl45ffg388yhuilazp9 2463857 2463854 2026-10-05T21:02:50Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. 2463857 wikitext text/x-wiki pww3urpg52fxawfub4aphw9hpxxsckn 2463858 2463857 2026-10-05T21:05:09Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. 2463858 wikitext text/x-wiki nbrk7qcuwi2omze7az6aiucgypwuets 2463859 2463858 2026-10-05T21:10:47Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. 2463859 wikitext text/x-wiki j9zzpkmawh6py6abp2luz1x8jeed390 2463861 2463859 2026-10-05T21:33:52Z Stashbot 7414 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet 2463861 wikitext text/x-wiki fxwxbrj272a784cww9un9wrczme3vuq 2463862 2463861 2026-10-05T21:34:12Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet 2463862 wikitext text/x-wiki 05mmajazvdawisx7d0x2ttzbdf68nhv 2463863 2463862 2026-10-05T21:35:56Z Stashbot 7414 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet 2463863 wikitext text/x-wiki j47d5gvirkmdi2us0dlek36aqosb72x 2463864 2463863 2026-10-05T21:38:51Z Stashbot 7414 dzahn@cumin1004: START - Cookbook sre.dns.netbox 2463864 wikitext text/x-wiki 9x3xh7sac31mirlas3ii9suo2x3l1la 2463865 2463864 2026-10-05T21:39:26Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet 2463865 wikitext text/x-wiki m94o6ja5gchthx88d0udvymq7pw876q 2463866 2463865 2026-10-05T21:39:58Z Stashbot 7414 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet 2463866 wikitext text/x-wiki 6akod24z1jgdjm2zbgc0dar9n65m86n 2463867 2463866 2026-10-05T21:45:18Z Stashbot 7414 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" 2463867 wikitext text/x-wiki 180814btis1myvfkbkjncinxzvycydy 2463868 2463867 2026-10-05T21:45:35Z Stashbot 7414 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" 2463868 wikitext text/x-wiki 8nid8xqbq14bkkbdnn75aj4lnr5ct3f 2463869 2463868 2026-10-05T21:45:36Z Stashbot 7414 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2463869 wikitext text/x-wiki qvnhqiumnkkhpgzb7ws6kv4wgvceug0 2463870 2463869 2026-10-05T21:45:37Z Stashbot 7414 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet 2463870 wikitext text/x-wiki emdkpb8h6536dsxsl2emunnzuem8ihb 2463872 2463870 2026-10-05T21:48:33Z Stashbot 7414 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet 2463872 wikitext text/x-wiki d7hjvk55n3vibotspu2qwy6cca45gmh 2463873 2463872 2026-10-05T21:53:25Z Stashbot 7414 dzahn@cumin1004: START - Cookbook sre.dns.netbox 2463873 wikitext text/x-wiki jmovxbe81edi1xti4km95i8xulu8z1e 2463875 2463873 2026-10-05T22:00:17Z Stashbot 7414 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" 2463875 wikitext text/x-wiki isz7rmdyryvid8qg1xmhwztmxydqvv6 2463876 2463875 2026-10-05T22:00:50Z Stashbot 7414 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" 2463876 wikitext text/x-wiki rpr6oxo6g3r8mgoloefuhy7ec5lp1r4 2463877 2463876 2026-10-05T22:00:51Z Stashbot 7414 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2463877 wikitext text/x-wiki s9mtekr2dvgq6r1nygngygesuc4ju1r 2463878 2463877 2026-10-05T22:00:52Z Stashbot 7414 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet 2463878 wikitext text/x-wiki a64ydanzi2yrwb4znx7oce1i8lhdloe 2463881 2463878 2026-10-05T23:12:22Z Stashbot 7414 derenrich@deploy1003: Started scap sync-world: Backport for [[gerrit:1347724|Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] 2463881 wikitext text/x-wiki ounk6u9es1jvgygjxed6zxjhytqwu7g 2463883 2463881 2026-10-05T23:16:18Z Stashbot 7414 derenrich@deploy1003: derenrich: Backport for [[gerrit:1347724|Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463883 wikitext text/x-wiki quvkouy5xopds6any9vrc3c5exiw0uw 2463884 2463883 2026-10-05T23:18:58Z Stashbot 7414 derenrich@deploy1003: derenrich: Continuing with deployment 2463884 wikitext text/x-wiki 6ex33j1c1tby2v7kmkzdb44knzzicya 2463885 2463884 2026-10-05T23:24:21Z Stashbot 7414 derenrich@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347724|Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] (duration: 11m 59s) 2463885 wikitext text/x-wiki 9rnijvjh5chifcpk2cowdnema8mh6oy 2463888 2463885 2026-10-06T02:00:42Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2463888 wikitext text/x-wiki 5lndf22y45t2a3zpzda8neb0kalezyq 2463889 2463888 2026-10-06T02:08:20Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) 2463889 wikitext text/x-wiki tukvj3186m1div2vhoz0690agcd2i8w 2463892 2463889 2026-10-06T03:03:03Z Stashbot 7414 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs T438219 2463892 wikitext text/x-wiki ajm5n92l8s7omeoips8bqk46rvo3qoa 2463893 2463892 2026-10-06T03:28:03Z Stashbot 7414 mwpresync@deploy1003: Rolling back deployment 2463893 wikitext text/x-wiki 1uzsx37hxv2ky5o9fgidyytfrjsiyvg 2463894 2463893 2026-10-06T03:30:20Z Stashbot 7414 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs T438219 (duration: 27m 17s) 2463894 wikitext text/x-wiki 7cdf7wql4hv63opy4zr9xtpj21sjq7p 2463895 2463894 2026-10-06T04:01:07Z Stashbot 7414 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s) 2463895 wikitext text/x-wiki 0obdcgr0dtvemq2rcs5srbri36q51e9 2463896 2463895 2026-10-06T05:19:34Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7 2463896 wikitext text/x-wiki 9kqyiei3fmzh6er0g2emfgzysrxz566 2463897 2463896 2026-10-06T05:21:41Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7 2463897 wikitext text/x-wiki b0uga3awhurmn4ml9fioojaoup92fxm 2463898 2463897 2026-10-06T05:26:01Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet 2463898 wikitext text/x-wiki revpglvkgjekhdjmzm3x2c4yozwjyjv 2463899 2463898 2026-10-06T05:26:02Z Stashbot 7414 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet 2463899 wikitext text/x-wiki ak50zjww52x3x6nvglzriajodvwtpa0 2463900 2463899 2026-10-06T05:28:15Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning 2463900 wikitext text/x-wiki i5avhr0qy220jh4zrigdrv0l4pkr5oz 2463901 2463900 2026-10-06T05:31:12Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7 2463901 wikitext text/x-wiki euw1i03h0puduedbzccyj9yomjzp7md 2463902 2463901 2026-10-06T05:31:38Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7 2463902 wikitext text/x-wiki 1hmwj3ug7zl0t38c3mi084bk3c03okk 2463903 2463902 2026-10-06T05:36:42Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet 2463903 wikitext text/x-wiki 96uf99pr6j7zru48lj5xm357mpi4o82 2463904 2463903 2026-10-06T05:36:44Z Stashbot 7414 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet 2463904 wikitext text/x-wiki h50bft6bypfdin4ck4svvzh5fe6x5x3 2463905 2463904 2026-10-06T07:03:45Z Stashbot 7414 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343917|Config change for launch of stopping sending LL notifications into group1. (T437577)]] 2463905 wikitext text/x-wiki nultl52gr0ihptqfazhm3tfwx7kfg5f 2463906 2463905 2026-10-06T07:08:13Z Stashbot 7414 awight@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1343917|Config change for launch of stopping sending LL notifications into group1. (T437577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463906 wikitext text/x-wiki 1kcffs4kllhzuuojf9elftl0jjrmd4i 2463908 2463906 2026-10-06T07:08:53Z Stashbot 7414 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment 2463908 wikitext text/x-wiki pknaqp1fhw39l7adr3t7xxkqoa6hfk7 2463909 2463908 2026-10-06T07:15:01Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back 2463909 wikitext text/x-wiki sggbgoq1e8yjpxtixm5y40wboov9als 2463910 2463909 2026-10-06T07:21:15Z Stashbot 7414 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343917|Config change for launch of stopping sending LL notifications into group1. (T437577)]] (duration: 17m 31s) 2463910 wikitext text/x-wiki g8zf7hr214xrilzxh4q1wgajwvyz4xz 2463911 2463910 2026-10-06T07:25:10Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7 2463911 wikitext text/x-wiki pl6gxncnqnguy0guiphvun6vufn8ypu 2463912 2463911 2026-10-06T07:30:07Z Stashbot 7414 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 (T439667) 2463912 wikitext text/x-wiki 0m0luvbsxunsekonhzcubxmy1x71w04 2463913 2463912 2026-10-06T07:31:53Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351826|Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316|Find canonical pages on special page completion search (T440151)]] 2463913 wikitext text/x-wiki 8divb9mnyigy7i5gpy6815c6mvcl31e 2463914 2463913 2026-10-06T07:33:00Z Stashbot 7414 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test 2463914 wikitext text/x-wiki 0lqpr8wvwhfrib600n5r5btt78j3k8q 2463915 2463914 2026-10-06T07:34:02Z Stashbot 7414 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test 2463915 wikitext text/x-wiki abqqi945e1bn0j6i3aajryh10ndav2v 2463916 2463915 2026-10-06T07:38:02Z Stashbot 7414 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351826|Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316|Find canonical pages on special page completion search (T440151)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463916 wikitext text/x-wiki 4sx4f4nkn1vzixsxnwo1exhnyf4nagx 2463917 2463916 2026-10-06T07:38:32Z Stashbot 7414 dcausse@deploy1003: dcausse: Continuing with deployment 2463917 wikitext text/x-wiki 130bchfgbthhaqmkw4dxm7iewdxrn9i 2463918 2463917 2026-10-06T07:45:46Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351826|Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316|Find canonical pages on special page completion search (T440151)]] (duration: 13m 53s) 2463918 wikitext text/x-wiki 5hm4yrgunk08vl22reoi1z0k3hwecpt 2463919 2463918 2026-10-06T07:50:16Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1351378|Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379|Change lead section infobox exclusion (T439943)]], [[gerrit:1351518|[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] 2463919 wikitext text/x-wiki tpv888vg3nwf9dl1t96fs0sedj2z2jx 2463920 2463919 2026-10-06T07:54:31Z Stashbot 7414 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1351378|Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379|Change lead section infobox exclusion (T439943)]], [[gerrit:1351518|[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2463920 wikitext text/x-wiki gdm7o94e27c1l8a15wpxj4ht53nj2js 2463921 2463920 2026-10-06T07:55:12Z Stashbot 7414 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment 2463921 wikitext text/x-wiki azdamcbwumr9ggtjyt9jf0kaaaqzf0u 2463922 2463921 2026-10-06T08:00:34Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351378|Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379|Change lead section infobox exclusion (T439943)]], [[gerrit:1351518|[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] (duration: 10m 18s) 2463922 wikitext text/x-wiki o9rtxz61tnm7yk6gfckr62rglfijst7 2463923 2463922 2026-10-06T08:00:55Z Stashbot 7414 matthiasmullie: UTC morning backport window done 2463923 wikitext text/x-wiki fzr05okbwxwdub576nhrn5rbbn1pit6 2463924 2463923 2026-10-06T08:04:24Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back 2463924 wikitext text/x-wiki t39pd3pubpcqqqyi19z7va28im0jpzn 2463925 2463924 2026-10-06T08:10:24Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7 2463925 wikitext text/x-wiki l7f08scw9gycmv3qz99pkxl3ewlbwhw 2463926 2463925 2026-10-06T08:13:57Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2463926 wikitext text/x-wiki tmn5epesw0govkvfl6iggidtw40m988 2463927 2463926 2026-10-06T08:14:07Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync 2463927 wikitext text/x-wiki h0pkkygs9cd3qz5zb3pm79kch8pflhk 2463928 2463927 2026-10-06T08:14:10Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync 2463928 wikitext text/x-wiki ov4cc364hwagw5ba5kd7j8bsl2kadnc 2463929 2463928 2026-10-06T08:14:13Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync 2463929 wikitext text/x-wiki 0fvvsvu3l3bj1b6ax2o2hxjuf7dyqzc 2463931 2463929 2026-10-06T08:14:20Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync 2463931 wikitext text/x-wiki 6767816o13xhdd2jzbjpeqo1pku88y4 2463932 2463931 2026-10-06T08:14:34Z Stashbot 7414 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test 2463932 wikitext text/x-wiki eipa1txeaa1c92jtw0jm20cy4b5tjz5 2463933 2463932 2026-10-06T08:14:36Z Stashbot 7414 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test 2463933 wikitext text/x-wiki dcfkdqjqezd5le1ayg9mr1g56qgzuwl 2463934 2463933 2026-10-06T08:14:43Z Stashbot 7414 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test 2463934 wikitext text/x-wiki 4s6eax9c2uc2l8opthdiuzri1xyoxnn 2463935 2463934 2026-10-06T08:14:45Z Stashbot 7414 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test 2463935 wikitext text/x-wiki rw0jf8jiq82xj2rjb74ngj92v7jhs2c 2463937 2463935 2026-10-06T08:16:47Z Stashbot 7414 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test 2463937 wikitext text/x-wiki 9xqcsy3x9mkzfeg74ogva77mpwtnahi 2463938 2463937 2026-10-06T08:16:50Z Stashbot 7414 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test 2463938 wikitext text/x-wiki fpd1937p4d5w29ekyx0dqsdfawxxc4y 2463939 2463938 2026-10-06T08:17:57Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2463939 wikitext text/x-wiki dfkx49yuuwj4somsd7ig9klq25hcom8 2463940 2463939 2026-10-06T08:28:12Z Stashbot 7414 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs T438219 2463940 wikitext text/x-wiki 7fdxuiil29yczuzqm9vt7x9diho1dep 2463941 2463940 2026-10-06T08:28:54Z Stashbot 7414 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test 2463941 wikitext text/x-wiki 99ow8pd7sd20f1cjesk0z61p74bxrgv 2463942 2463941 2026-10-06T08:28:58Z Stashbot 7414 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test 2463942 wikitext text/x-wiki 878dmxhelnt7n8l3tg4o7pg1xokvfyu 2463943 2463942 2026-10-06T08:29:28Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning 2463943 wikitext text/x-wiki ms5lw5449y689d5n7cl7bw433np097i 2463944 2463943 2026-10-06T08:29:43Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning 2463944 wikitext text/x-wiki 7qo8sl1acmt8mq670656z4n7itzm7sw 2463945 2463944 2026-10-06T08:29:54Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning 2463945 wikitext text/x-wiki mev25ojqcix3ct2ekivnszd5yf2lnnd 2463946 2463945 2026-10-06T08:30:11Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning 2463946 wikitext text/x-wiki nyrnar3h631wxobt6zttpi9652lxslh 2463947 2463946 2026-10-06T08:30:40Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning 2463947 wikitext text/x-wiki o9lgnbqaai3haleqab1qu3h3dme90i4 2463948 2463947 2026-10-06T08:54:39Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2463948 wikitext text/x-wiki tj8mzx19t3csc7o6qyoyg4x8tkhnrc6 2463949 2463948 2026-10-06T08:56:46Z Stashbot 7414 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2463949 wikitext text/x-wiki 8ec27fcp0lxepvsl92dbmdr7vregyh9 2463950 2463949 2026-10-06T09:00:41Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync 2463950 wikitext text/x-wiki d0giwfs3ffl478wynwfev0wrc6jzysg 2463951 2463950 2026-10-06T09:00:44Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync 2463951 wikitext text/x-wiki 9c3f8d4hkbk6515ink8iwca1da5kagf 2463952 2463951 2026-10-06T09:02:33Z Stashbot 7414 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 (T439667) 2463952 wikitext text/x-wiki pz9kzrhs6e3rq98jw3wet75vv8goib8 2463953 2463952 2026-10-06T09:11:49Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2463953 wikitext text/x-wiki dadteo6qfrnqc280ilqsxx1nsne5asn 2463954 2463953 2026-10-06T09:12:09Z Stashbot 7414 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test 2463954 wikitext text/x-wiki ijvx6m32807o9hiimr6auut4jng4z9m 2463955 2463954 2026-10-06T09:12:48Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing 2463955 wikitext text/x-wiki 7palb88c0xbtv00ney7mtpbac17jw65 2463956 2463955 2026-10-06T09:13:01Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing 2463956 wikitext text/x-wiki s54lgapuhmjw4nrachsa3oovne9tj83 2463957 2463956 2026-10-06T09:13:30Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7 2463957 wikitext text/x-wiki jiuvfsjdi9npeualzms9bujxv8y6ccd 2463958 2463957 2026-10-06T09:13:47Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) 2463958 wikitext text/x-wiki 7x3vfz8deqx24mc4ivrxype9qbspfxq 2463959 2463958 2026-10-06T09:13:51Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet 2463959 wikitext text/x-wiki 6i9jqt9ormgygigmcb1865oqadijw50 2463960 2463959 2026-10-06T09:14:24Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet 2463960 wikitext text/x-wiki ln9tysarwpbyaz8y8xmss9awze4a3e9 2463961 2463960 2026-10-06T09:14:48Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet 2463961 wikitext text/x-wiki big8tdka04c5nquu2076vmh2ym3t77o 2463962 2463961 2026-10-06T09:14:50Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet 2463962 wikitext text/x-wiki tukcv4f8d7511u8ah8661lbymovql61 2463963 2463962 2026-10-06T09:15:03Z Stashbot 7414 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) 2463963 wikitext text/x-wiki azzfcn8wirepytf394wd1g3glzsq1zo 2463964 2463963 2026-10-06T09:15:37Z Stashbot 7414 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet 2463964 wikitext text/x-wiki kghbjrgtj03v9194itrwi6m5pxpn3qa 2463965 2463964 2026-10-06T09:15:39Z Stashbot 7414 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet 2463965 wikitext text/x-wiki s8z1m4x95d44yjil7fkw37x84m0fth4 2463968 2463965 2026-10-06T09:32:29Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2463968 wikitext text/x-wiki l2x648erlfwcal9cennen5x9zls368d 2463970 2463968 2026-10-06T09:45:22Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2463970 wikitext text/x-wiki oh94eyipx01unltoz7axdmanf2iw4c8 2463972 2463970 2026-10-06T09:45:38Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2463972 wikitext text/x-wiki slbfkpk70crzdwvyfpzyhkk7xgikcqf 2463973 2463972 2026-10-06T09:45:40Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2463973 wikitext text/x-wiki 3fq77nn32mht6vee10yxfd35n4znxky 2463974 2463973 2026-10-06T09:45:42Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2463974 wikitext text/x-wiki rwcaijiaojk5yw6y30cebc85fqzdyt9 2463975 2463974 2026-10-06T09:52:17Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back 2463975 wikitext text/x-wiki r59vm19asbxki9dfij8hb8lnvq4uvei 2463976 2463975 2026-10-06T09:53:00Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7 2463976 wikitext text/x-wiki cra9m78vmazz1t9znptm2iyjqf00v31 2463977 2463976 2026-10-06T09:55:38Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2463977 wikitext text/x-wiki i5fllfjlomqjgmkme31nsgv0b5c6wi4 2463978 2463977 2026-10-06T09:57:38Z Stashbot 7414 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 2463978 wikitext text/x-wiki 3h7ygywrudci5dcf74xbnsyygbiig3i 2463979 2463978 2026-10-06T09:57:58Z Stashbot 7414 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 00m 19s) 2463979 wikitext text/x-wiki i331ymy8460tgy6yy3sgxlcxxmf9c2q 2463980 2463979 2026-10-06T09:58:14Z Stashbot 7414 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 2463980 wikitext text/x-wiki 94sv9ce3f2dx1z1fv2rwwspr21g9oa8 2463981 2463980 2026-10-06T09:58:36Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7 2463981 wikitext text/x-wiki pq265vgtvms527gj57yxzriu53g5kls 2463982 2463981 2026-10-06T09:58:38Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2463982 wikitext text/x-wiki 2fyghslrli12j4vq67z6qf2ap2l2bc4 2463983 2463982 2026-10-06T09:58:48Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . 2463983 wikitext text/x-wiki tn6xnk4uio0xfryel87o02yzecka52c 2463984 2463983 2026-10-06T10:02:59Z Stashbot 7414 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m | tee gc_run_06102026` - T437297 2463984 wikitext text/x-wiki e2peklikhiqq9gujc2oyqnfomeljw2w 2463985 2463984 2026-10-06T10:08:40Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing 2463985 wikitext text/x-wiki 1ava9mkyl899p10ocnfoc5r4wgqxevv 2463986 2463985 2026-10-06T10:08:41Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache 2463986 wikitext text/x-wiki prwduw1pbg1fx956aay8yq8bgs0y3dr 2463987 2463986 2026-10-06T10:08:42Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2463987 wikitext text/x-wiki b4p9hmhkh1no0ek0wes0df3vnrgfos8 2463988 2463987 2026-10-06T10:08:43Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing 2463988 wikitext text/x-wiki bvx1wa8508tx3gqod41v0ciqvoaae2f 2463989 2463988 2026-10-06T10:09:21Z Stashbot 7414 moritzm: installing curl security updates 2463989 wikitext text/x-wiki lr8p2ykd8k5yxpne5fw4zc6ltfcppb0 2463990 2463989 2026-10-06T10:09:27Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . 2463990 wikitext text/x-wiki rrcz22omviftu27179yqnu2lo07tvh9 2463991 2463990 2026-10-06T10:09:36Z Stashbot 7414 marostegui: Upgrade pc1 to mariadb 10.11.19 T436082 2463991 wikitext text/x-wiki n7mo3sgrcdqcov0gnm7wmax2u66r1fd 2463992 2463991 2026-10-06T10:09:43Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . 2463992 wikitext text/x-wiki 6e4mke0vjbil7jyww7js9v6txabgaai 2463993 2463992 2026-10-06T10:10:11Z Stashbot 7414 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2463993 wikitext text/x-wiki 5ds8mlz9d566li6ee2jwj18fntyfzqv 2463994 2463993 2026-10-06T10:10:48Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading 2463994 wikitext text/x-wiki 95dytwi35xam565b4w00xrxhz1evg2g 2463995 2463994 2026-10-06T10:10:49Z Stashbot 7414 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2463995 wikitext text/x-wiki obhqagecitabgi35pcym5hfkx4aoty4 2463996 2463995 2026-10-06T10:10:50Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2463996 wikitext text/x-wiki d3tnfo7lsuf53uylrcnmc966fjpcow2 2463997 2463996 2026-10-06T10:10:57Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2463997 wikitext text/x-wiki hsftj34ordn5ufatlfm6m7ba1817n0i 2463998 2463997 2026-10-06T10:12:14Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage 2463998 wikitext text/x-wiki 2igxvt24m4nv42lxd4pvrexuwa8rw3m 2463999 2463998 2026-10-06T10:12:44Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2463999 wikitext text/x-wiki 8b22vjatcb5tvlm0hivy1beghc0iaro 2464000 2463999 2026-10-06T10:13:11Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1 2464000 wikitext text/x-wiki t7xnt9zvct6xhxwz8c1yl5v404gszex 2464001 2464000 2026-10-06T10:13:12Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache 2464001 wikitext text/x-wiki awcughqu85xgl2vmc80i08ftov3019z 2464002 2464001 2026-10-06T10:13:20Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2464002 wikitext text/x-wiki q0qol1d3thmq42ch3mzdv1bo8muzjpc 2464003 2464002 2026-10-06T10:13:21Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1 2464003 wikitext text/x-wiki gk1bhswjqczgwj0ryyofi4jbjnyw8ia 2464004 2464003 2026-10-06T10:14:53Z Stashbot 7414 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor (T406724) 2464004 wikitext text/x-wiki ipm3vx27ps1yku0ph4kvgvm5q1zc6iw 2464005 2464004 2026-10-06T10:18:33Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage 2464005 wikitext text/x-wiki 5i412oqp4iad1x3tj8om770r1ay0dai 2464006 2464005 2026-10-06T10:18:59Z Stashbot 7414 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2464006 wikitext text/x-wiki bse8zsxeg3h7liwut04gbxxf5pa644y 2464007 2464006 2026-10-06T10:19:51Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2464007 wikitext text/x-wiki g7vtc3659jv2u5yok4avg5w1dj6ifgy 2464008 2464007 2026-10-06T10:37:08Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" 2464008 wikitext text/x-wiki n9eoq16a4t1i1tngmorox466ik9fs2f 2464009 2464008 2026-10-06T10:37:24Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back 2464009 wikitext text/x-wiki knwat0rusnu87zk6sepzwwn8e0cqz4j 2464010 2464009 2026-10-06T10:38:07Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7 2464010 wikitext text/x-wiki ebmab4v37nycggysg511msgh1nteaue 2464011 2464010 2026-10-06T10:38:09Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" 2464011 wikitext text/x-wiki s3ob72v3v173j5a8351odpzuzehgfiq 2464012 2464011 2026-10-06T10:38:10Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm 2464012 wikitext text/x-wiki 8cwp29tbnxkgpakvnzpfd9lqsli71o0 2464013 2464012 2026-10-06T10:38:13Z Stashbot 7414 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2464013 wikitext text/x-wiki lttk7jpjho7en09cmxd55hzpm557xte 2464014 2464013 2026-10-06T10:40:08Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2464014 wikitext text/x-wiki jcz6lgoxy3awl4sijb8xlheauqnrmdo 2464015 2464014 2026-10-06T10:56:27Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage 2464015 wikitext text/x-wiki pf18a7k35eedibp2ic43hrc0myv4v56 2464016 2464015 2026-10-06T11:01:49Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage 2464016 wikitext text/x-wiki sopm4bnddoyv3t3rk3zifpeqjdr0bc9 2464017 2464016 2026-10-06T11:02:49Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.dns.netbox 2464017 wikitext text/x-wiki kyyb5n77qwsnaiuaffgemci60tgqcgt 2464018 2464017 2026-10-06T11:05:21Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2464018 wikitext text/x-wiki q8cqijs0wvjgycutgtp1s2yfhcux66w 2464019 2464018 2026-10-06T11:08:40Z Stashbot 7414 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply 2464019 wikitext text/x-wiki jweo05hs1gnsgiuadcben8m0335i1ad 2464020 2464019 2026-10-06T11:09:20Z Stashbot 7414 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply 2464020 wikitext text/x-wiki 04tkaajrciqvhlgiqm8ji36gcquv19w 2464021 2464020 2026-10-06T11:21:10Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" 2464021 wikitext text/x-wiki 7zuckrac816ah9jm82odhasl34snr2f 2464022 2464021 2026-10-06T11:21:33Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" 2464022 wikitext text/x-wiki 5h19gl3h7781bwzhj6mljjt6x9b2nrm 2464023 2464022 2026-10-06T11:21:35Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm 2464023 wikitext text/x-wiki g0bfaow7x1oiyh491aqak2d4shutff6 2464024 2464023 2026-10-06T11:33:21Z Stashbot 7414 moritzm: installing chromium security updates 2464024 wikitext text/x-wiki km1bcxx90rypwsrr5i30g5p7tcr9qm1 2464026 2464024 2026-10-06T11:41:05Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2464026 wikitext text/x-wiki lg8bcbphq44ov7aw9xob4uylfj4bqjl 2464027 2464026 2026-10-06T11:41:19Z Stashbot 7414 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2464027 wikitext text/x-wiki 7tse2j7qox8auzhmpudiw021q7x47t3 2464029 2464027 2026-10-06T11:52:01Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2464029 wikitext text/x-wiki 0a99j5u0p91udv6ml588xmydh1nmh7r 2464030 2464029 2026-10-06T11:56:18Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync 2464030 wikitext text/x-wiki mm8l31ltxw7swn0xpzvddvgd1tporam 2464031 2464030 2026-10-06T11:56:45Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync 2464031 wikitext text/x-wiki 03yjpvnpw3819auz8zccjnv8b270bsv 0 17290 2463828 2463421 2026-10-05T18:16:38Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/123 (T435368) 2463828 wikitext text/x-wiki cf7q1pxgsf07heqaxcv6pvemozsmrl0 0 21478 2463694 2461388 2026-10-05T14:55:27Z AndrewBogott (WMF) 51047 /* Quota management */ 2463694 wikitext text/x-wiki pqiiq32l2o4plggqo30bej6377a6an2 498 30942 2463456 2463291 2026-10-05T12:06:37Z Stashbot 7414 godog: set performance governor on cloudvirt in C8 cloudvirt[1048-1049,1051,1054-1057,1077].eqiad.wmnet - T439729 2463456 wikitext text/x-wiki o05npx6xlyyvpmm35hsiztujh8rtnll 2463531 2463456 2026-10-05T13:16:05Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add 2463531 wikitext text/x-wiki 2yhncu14afy5wl5nx6gqx72jvlw7f1h 2463590 2463531 2026-10-05T13:55:45Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) 2463590 wikitext text/x-wiki dgow0xdotj4kw61sm7lxrapzvyvpz9w 2463592 2463590 2026-10-05T13:56:29Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2463592 wikitext text/x-wiki jb0552inml64kjm71ix3w0nszkibpmu 2463593 2463592 2026-10-05T13:56:37Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) 2463593 wikitext text/x-wiki r8mbn3bmruwdf80mt8xx59ezmmvo8r1 2463594 2463593 2026-10-05T13:57:00Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2463594 wikitext text/x-wiki ax38mw5p7m6dsg2uix6mipwmfxrzg3o 2463633 2463594 2026-10-05T14:27:29Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) 2463633 wikitext text/x-wiki fuebsdv7cu5042xh0dtroio3cnw4vut 2463714 2463633 2026-10-05T15:12:01Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2463714 wikitext text/x-wiki pke9psokd8tpx77vjgoxot84lzfn6ce 2463715 2463714 2026-10-05T15:12:07Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) 2463715 wikitext text/x-wiki pjohmyn80ml4owtpkh1xq7rxg5tij0w 2463723 2463715 2026-10-05T15:13:31Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add 2463723 wikitext text/x-wiki 6lu4w82jklftvigg7uvkdm493c2d9ad 2463772 2463723 2026-10-05T16:07:04Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) 2463772 wikitext text/x-wiki 2u1cg4k83895rb0wyunlmmw1u30aoag 2463855 2463772 2026-10-05T21:00:18Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,magnum 2463855 wikitext text/x-wiki 47xd3fdnix5lrt3tjzs410ydccv9duf 2463856 2463855 2026-10-05T21:00:56Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,magnum 2463856 wikitext text/x-wiki g3bezwsuycllrfhimkm8obhnyzjtr7i 2463966 2463856 2026-10-06T09:21:21Z Stashbot 7414 godog: set performance governor on cloudvirt in F4 cloudvirt[1050,1052-1053,1058-1061,1076,1080] - T439729 2463966 wikitext text/x-wiki hego426tcvqppd74x3jnenon3b6am0i 2464025 2463966 2026-10-06T11:39:59Z Stashbot 7414 godog: set performance governor on cloudvirt in E4 cloudvirt[1062-1064,1068-1071,1075,1079] - T439729 2464025 wikitext text/x-wiki bvn5bofnkactslx4giqatcfadvli608 2464028 2464025 2026-10-06T11:41:41Z Stashbot 7414 godog: set performance governor on cloudvirt in D5 cloudvirt[1062-1064,1068-1071,1075,1079] - T439729 2464028 wikitext text/x-wiki 7ey0hg82k0zgawdwzz2v6ekls5bzlvc 0 334461 2463683 2409906 2026-10-05T14:49:01Z RCoccioli (WMF) 7416 /* PuppetDB is down */ Update section with proper hostnames 2463683 wikitext text/x-wiki bwpkgdhksj4nbao4maxz7uuvxg7sof4 498 443946 2463807 2463207 2026-10-05T17:45:34Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 9a129cb..772ee56: l10n updates (lt, ne, tt-cyrl) 2463807 wikitext text/x-wiki qisd9kwonc2uy3yqxgulgvl016x859h 0 447017 2463839 2455667 2026-10-05T19:25:26Z EEvans (WMF) 5035 added link to Upcoming SLOs 2463839 wikitext text/x-wiki qbt2zjbdumxpbbwed2l1y2jgn18ldr2 12 449113 2463452 2463254 2026-10-05T12:00:38Z DCaro (WMF) 18271 /* Internal log storage */ added a note about the new logs cli 2463452 wikitext text/x-wiki sdkzvvngjgh8eikkajaj2apyueq5si8 2463576 2463452 2026-10-05T13:42:07Z DCaro (WMF) 18271 /* Toolforge logs */ add task for deployment events 2463576 wikitext text/x-wiki aru3l5yfiaj09besdwu08wex213bqsa 0 449160 2463886 2463266 2026-10-06T00:01:12Z Dexbot 30554 Bot: Updating the report 2463886 wikitext text/x-wiki mtqyceyvtx1d66yaq5x53xu7w3kv32d 0 451075 2463850 2459571 2026-10-05T20:36:29Z WFan (WMF) 38087 2463850 wikitext text/x-wiki sxijxy80ku1b9rcvtml5ez5dvgj9uha 2463851 2463850 2026-10-05T20:37:18Z WFan (WMF) 38087 2463851 wikitext text/x-wiki jkqrlrwn1ton7zgllk222qspdd2gb9z 116 453426 2463887 2462579 2026-10-06T00:18:46Z Phabbanbot 37210 Dabmasterars was disabled by WMFOffice 2463887 wikitext text/x-wiki 6hkzhsmm0y79d6961ybqgb0luor7hiu 0 453597 2463455 2461748 2026-10-05T12:03:14Z DCaro (WMF) 18271 /* 2026 */ added logs-cli entry 2463455 wikitext text/x-wiki 1c1hmewiptu6wake6347jnjb0lnoj4f 0 453813 2463848 2461664 2026-10-05T20:07:43Z PFischer-WMF 38527 Second review pass: name the two request classes and state which SLI covers which, answer the pool counter question (T440163), fill in Operational per the template worksheet including the out-of-hours paging gap, give Ideal targets real numbers, refresh measurements 2463848 wikitext text/x-wiki dhuscb7kr6lfkkgl93n79wrpc11r67t 0 454129 2463880 2449294 2026-10-05T23:05:00Z MPopov (WMF) 5891 /* Configure the web/base stream in your local MediaWiki */ 2463880 wikitext text/x-wiki jdzt9zbducnb45005jxbrb28uo5m9s8 0 454546 2463777 2426629 2026-10-05T16:23:10Z Mingapedia 60255 Tell mebwhat lived and visual things that were happening in tje world in each century 2463777 wikitext text/x-wiki ebt38drl47fwgyrol91jdvwo3ul7gq3 0 455283 2463890 2202031 2026-10-06T02:23:19Z LSandergreen-WMF 46345 add "cases where joins and fields don't as you'd expect" 2463890 wikitext text/x-wiki d3682ge9e52zxqre5o004hwgc5mfzpk 2463891 2463890 2026-10-06T02:24:00Z LSandergreen-WMF 46345 /* Cases where joins and fields don't work as you'd expect */ 2463891 wikitext text/x-wiki 4gy6oe908peoz1qx7u3xxcrq19swiwz 0 455989 2463871 2461064 2026-10-05T21:46:42Z MPopov (WMF) 5891 /* HTTP header */ Example config for logged-in experiments 2463871 wikitext text/x-wiki p361sl9cu1gpwcqm06icyjg8sz2on1j 0 459511 2463882 2455718 2026-10-05T23:13:23Z PPelberg (WMF) 37902 Boldly adding, what I assume to be, the dates of the ST6 and SRE offsites based on what I'm seeing on Office wiki (https://office.wikimedia.org/wiki/Travel_and_Convening/Convening/Schedule). Please correct what I've mistaken! 2463882 wikitext text/x-wiki ct477di896w6j4adoowkfz7741tenoh 0 459560 2463860 2459297 2026-10-05T21:31:36Z DAdedoyin (WMF) 39150 Test 2463860 wikitext text/x-wiki c14wk49ydq5s5zhj2weuntirkhi68r1 0 459783 2463840 2427023 2026-10-05T19:32:33Z BPirkle (WMF) 12686 typo fix 2463840 wikitext text/x-wiki iswt80l295be41rsew9kwpqlf2xqkb7 0 460059 2463936 2450309 2026-10-06T08:15:03Z JMonton-WMF 47887 info about targets 2463936 wikitext text/x-wiki fj6ymeaz3ayhu99rkdxc3pi1bx4u1wd 0 460192 2463967 2457371 2026-10-06T09:27:47Z MLechvien-WMF 49197 Rephrase and simplify SCROLL to reference the newly published Service Catalog instead 2463967 wikitext text/x-wiki 4wpfy2j504dq2i4f23vb04taj051dte 2 460394 2463741 2445039 2026-10-05T15:26:45Z Milimetric 612 2463741 wikitext text/x-wiki 4vr9l8b7mz894355ig0yz5m1ijvjjnc 1 460409 2463730 2455170 2026-10-05T15:18:43Z Sannita (WMF) 42586 /* status of label service queries */ Reply 2463730 wikitext text/x-wiki sk4s6jmkqleirq13pegshgdh936m99i 2463773 2463730 2026-10-05T16:14:14Z Peter F. Patel-Schneider 42591 /* status of label service queries */ Reply 2463773 wikitext text/x-wiki gkauj03dc3jilozbsx9ikhvsy7otaml 0 460417 2463821 2459757 2026-10-05T18:02:59Z ISarantopoulos-WMF 34784 Add qwen3.8 model. Move gpt-oss to public section 2463821 wikitext text/x-wiki iy7yx5ea1gffmmjpxmv0t3lnlbjjsxi 0 460748 2463458 2462401 2026-10-05T12:13:30Z DPogorzelski-WMF 48508 2463458 wikitext text/x-wiki axnwq6rkod9q1xg6eq0ovu6ej56hebx 2463460 2463458 2026-10-05T12:16:18Z DPogorzelski-WMF 48508 2463460 wikitext text/x-wiki 6kfa0hntfhz7k6a0ru206z4eyidgoqg 2463495 2463460 2026-10-05T12:51:53Z DPogorzelski-WMF 48508 2463495 wikitext text/x-wiki e7883rk0xxxz4ialcazvvfo7zqxeovj 2463525 2463495 2026-10-05T13:11:57Z DPogorzelski-WMF 48508 2463525 wikitext text/x-wiki h84t8k7xto7flplmzjhoakt6po323r5 0 460771 2463459 2463449 2026-10-05T12:15:54Z JGiannelos (WMF) 17083 /* Architectural */ 2463459 wikitext text/x-wiki nncc7gi6sjz5ujudj4mocbmhb4ywj4k 2463464 2463459 2026-10-05T12:27:48Z JGiannelos (WMF) 17083 2463464 wikitext text/x-wiki jx5pl04kc7maqtbdrisbtugu37vo4p4 2463465 2463464 2026-10-05T12:30:47Z JGiannelos (WMF) 17083 /* Service dependencies */ 2463465 wikitext text/x-wiki rnz15skn7qlz4n7rwitqsfi3rhnraqe 2463468 2463465 2026-10-05T12:35:47Z JGiannelos (WMF) 17083 /* Request Classes */ 2463468 wikitext text/x-wiki r3v9e2xu4lmmsdus9ig1n1qs17te2zd 2463471 2463468 2026-10-05T12:38:02Z JGiannelos (WMF) 17083 /* Client-facing */ 2463471 wikitext text/x-wiki p6xxqju34n5chljj0g2l1tggtawk75y 2463473 2463471 2026-10-05T12:38:35Z JGiannelos (WMF) 17083 /* Request Classes */ 2463473 wikitext text/x-wiki ttg5bv1do2bdbavk4x84zxt3h9mcucr 2463496 2463473 2026-10-05T12:51:55Z JGiannelos (WMF) 17083 /* Client-facing */ 2463496 wikitext text/x-wiki mbp39fb6mu96923gowpkml0p1o2dxgh 2463500 2463496 2026-10-05T12:52:37Z JGiannelos (WMF) 17083 /* Request Classes */ 2463500 wikitext text/x-wiki 4fgjj2jmjddsa71tmdgwwe4q60cx8sh 2463606 2463500 2026-10-05T14:04:54Z JGiannelos (WMF) 17083 /* Deployment */ 2463606 wikitext text/x-wiki eqis7878gjje780qfql59m4upu56wlm 2463768 2463606 2026-10-05T15:54:40Z JGiannelos (WMF) 17083 /* Troubleshooting */ 2463768 wikitext text/x-wiki hh0vocakvek8wsbet6u188l2t7d4ek2 2463969 2463768 2026-10-06T09:45:22Z JGiannelos (WMF) 17083 /* Monitoring */ 2463969 wikitext text/x-wiki t7s4a3oslnlucyukdhxkqwwq9e6r4pu 2463971 2463969 2026-10-06T09:45:31Z JGiannelos (WMF) 17083 /* Operational */ 2463971 wikitext text/x-wiki 4drqv0uqbvhvhmlr9xe6u61fz3adl3o 0 460773 2463511 2463447 2026-10-05T12:59:08Z DPogorzelski-WMF 48508 2463511 wikitext text/x-wiki gb9qxrfa18qzk26eyxqjp7mo5pn0s4q 2463521 2463511 2026-10-05T13:08:53Z DPogorzelski-WMF 48508 2463521 wikitext text/x-wiki gn47kivpambhlhchi2rjy30bf4ftz00 498 460774 2463574 2026-10-05T13:40:40Z Labslogbot 55 Auto update of instance info. 2463574 wikitext text/x-wiki p00912hw5jgw3qrtch9cajsslcgj7wn 498 460775 2463575 2026-10-05T13:41:28Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for trove-only project mna-db in eqiad1 2463575 wikitext text/x-wiki 7az0w74dq54rnq3kk0tnuovcdh9zwek 3 460776 2463704 2026-10-05T15:02:28Z StrikerBot 8475 Welcome to Toolforge! 2463704 wikitext text/x-wiki kwx4ujk9r2m2cxfucopp3uv2tplolrh 1 460777 2463794 2026-10-05T17:16:58Z Sadads 7230 /* 27-B endpoint unstable? */ new section 2463794 wikitext text/x-wiki 2eg93xambwrd81953htwxwf4y7c2kyp 2463824 2463794 2026-10-05T18:07:35Z ISarantopoulos-WMF 34784 /* 27-B endpoint unstable? */ Reply 2463824 wikitext text/x-wiki 5sbynpbxg8vs4v9a0xujp8wtfnh0nj5 2463930 2463824 2026-10-06T08:14:15Z ISarantopoulos-WMF 34784 /* 27-B endpoint unstable? */ Reply 2463930 wikitext text/x-wiki cpawr4g9da9265g40t1ng9d969pkdas 498 460778 2463827 2026-10-05T18:15:48Z Stashbot 7414 andrewbogott: "openstack database instance rebuild eee0f5a4-1824-44b7-b781-e9846ca91efe 2e3ddba2-e36a-468d-a862-4a1ae0d4ed4d" for task T430503 2463827 wikitext text/x-wiki p48ygzikrw6vpz8wmv8ns0ia72g7n58 0 460779 2463838 2026-10-05T19:23:25Z EEvans (WMF) 5035 create page from copy of template 2463838 wikitext text/x-wiki c2v9e69iid8s89iqumybv8ad9dm6eay 2463844 2463838 2026-10-05T19:55:10Z EEvans (WMF) 5035 first-pass at "Organizational" 2463844 wikitext text/x-wiki 4xzfdnnz429vrnkc18kbew8i8uamkld 2463845 2463844 2026-10-05T19:59:54Z EEvans (WMF) 5035 first-pass at "Architectual" (much more is needed) 2463845 wikitext text/x-wiki c2skgib3fupl0968q2tpj8u2adxys58 2 460780 2463874 2026-10-05T21:54:36Z BrianJeffrey01 60265 Created page with "'''Professional Heating Installation in Nutley NJ''' A reliable heating system plays an important role in keeping homes and businesses comfortable during cold weather. '''AMC Plumbing Heating Cooling''' provides professional '''[https://amcnutleynjplumbinghvacservice.com heating installation Nutley NJ]''' services for residential and commercial properties. Our team understands that every property has different heating requirements. We focus on proper system selection,..." 2463874 wikitext text/x-wiki sg0xufg8nexxc7kblul4v2jqtfuy12k