Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.48.0-wmf.1 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2474655 2474205 2026-10-28T14:08:14Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice 2474655 wikitext text/x-wiki p63pjnkyavc8nminl0o9wk825o9n75c 2474656 2474655 2026-10-28T14:08:17Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice 2474656 wikitext text/x-wiki td5dp53qpetuce6sprujmip9cpywvmz 2474657 2474656 2026-10-28T14:08:26Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2474657 wikitext text/x-wiki 7o9mriupjqr6i1345dsg9gio8ltjux9 2474658 2474657 2026-10-28T14:09:14Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli 2474658 wikitext text/x-wiki 2a1naqo9jm7yoa4xvr2x2g4458cdk6v 2474678 2474658 2026-10-28T14:28:24Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2474678 wikitext text/x-wiki 9a7dt95ruryezu5r5znm9o5y2fx4b97 2474680 2474678 2026-10-28T14:29:13Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli 2474680 wikitext text/x-wiki h64i01q4sfrt807svwohe9kzhamfqj3 2474685 2474680 2026-10-28T14:31:04Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2474685 wikitext text/x-wiki 23xd58st0mc6pge9v8w7fp1z0xo8emg 2474687 2474685 2026-10-28T14:32:03Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2474687 wikitext text/x-wiki 5y9e19nxwqnsiui3b9sn2buiuhswa0s 2475028 2474687 2026-10-28T22:40:24Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2475028 wikitext text/x-wiki 9u5krtfksnz808nhcpeupltvdfi7co3 2475029 2475028 2026-10-28T22:41:22Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2475029 wikitext text/x-wiki pf3jr86hx1cluce8iqg00rgzjpaqpk0 2475110 2475029 2026-10-29T09:23:27Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T441425) 2475110 wikitext text/x-wiki lsujvpvszlt255fcgzp2sax8zavax5v 2475116 2475110 2026-10-29T09:29:05Z Stashbot 7414 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers (T441425) 2475116 wikitext text/x-wiki s6zvhepyl0sqg9izyeferdbh70oe5wa 2475122 2475116 2026-10-29T09:34:06Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T441425) 2475122 wikitext text/x-wiki 979xyjjhrwveacs9n9dhzskxv1qgho8 2475138 2475122 2026-10-29T09:52:25Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T441425) 2475138 wikitext text/x-wiki ouevelj6rytck7rawmo51b5b85b1h84 0 3818 2474964 2461434 2026-10-28T20:33:12Z LFrost-WMF 59983 /* More information */ puppetlabs.com => puppet.com 2474964 wikitext text/x-wiki 4l1muyy1yyyww18iwmi2l91xabd05e4 0 4108 2474575 2474559 2026-10-28T12:20:45Z ScheduleDeploymentBot 37566 Add [[gerrit:1359803]] to Wednesday, October 28 UTC afternoon backport window 2474575 wikitext text/x-wiki cguvmguoarp7t2iwi8yrjxcjqpxobse 2474724 2474575 2026-10-28T15:08:18Z Hashar 32 /* {{Deployment_day|date=2026-10-29}} */ 9:00 CEST Switch Gerrit from Java 17 to Java 21 {{phabT|T392465}} 2474724 wikitext text/x-wiki qobjhzd31i5ed3ipzwbt65c7tqlo5l8 2474726 2474724 2026-10-28T15:09:02Z Hashar 32 /* {{Deployment_day|date=2026-10-29}} */ 2474726 wikitext text/x-wiki ff4nxbydzu9h2002o0asr6ezim1dpvu 2474765 2474726 2026-10-28T15:39:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1359887]] to Wednesday, October 28 UTC late backport window 2474765 wikitext text/x-wiki o4hpbvsniv6vug9gnw2swrzwbk88lvs 2474863 2474765 2026-10-28T17:54:55Z ScheduleDeploymentBot 37566 Add [[gerrit:1358832]] to Wednesday, October 28 UTC late backport window 2474863 wikitext text/x-wiki phtj4ohk2wjz24hmqr51u266mltcbx1 2475165 2474863 2026-10-29T10:45:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1360367]] to Thursday, October 29 UTC afternoon backport window 2475165 wikitext text/x-wiki 1ypwolrza24xwipactv2h1vdqgghthe 0 4120 2474638 2432049 2026-10-28T13:28:39Z ASanford-WMF 52089 Link to the documentation for generating a hardware-backed SSH key 2474638 wikitext text/x-wiki o3qsxl2dc1ruuiq1u6iedan0kgkvr2y 0 4383 2474738 2428810 2026-10-28T15:19:32Z MVernon (WMF) 25893 /* Cluster Setup */ link to the envoyproxy bug 2474738 wikitext text/x-wiki jxs9l3ysy2mcgntyff6jrpz53amsrl2 0 4393 2475058 2339525 2026-10-29T06:28:33Z Hashar 32 Jenkins no more updates Beta Cluster {{phabT|T256168}} 2475058 wikitext text/x-wiki 09mdkrvsxhf21mk3jd1mtceabvy2fk9 2475059 2475058 2026-10-29T06:31:04Z Hashar 32 Small updates about the servers 2475059 wikitext text/x-wiki chk04vws94jhvrbzjq4bt821653iqxd 2475062 2475059 2026-10-29T06:36:43Z Hashar 32 Update Bullseye/Bookworm > Trixie 2475062 wikitext text/x-wiki qu1mcsvqr2qbtl7itq8nvy96hyeutus 498 6726 2474669 2473865 2026-10-28T14:21:07Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2474669 wikitext text/x-wiki 8936h09iofna4pu3rn1f3zr6qmwe40j 2474671 2474669 2026-10-28T14:22:30Z Stashbot 7414 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli 2474671 wikitext text/x-wiki ja6aap6najkf1gdwak5ziz4w2zsvio2 2474675 2474671 2026-10-28T14:27:15Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2474675 wikitext text/x-wiki aqo0gyobtembku2sdyrw2ge125jeq7b 2474677 2474675 2026-10-28T14:28:06Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2474677 wikitext text/x-wiki niclokx87f5asvu6remkommet2jdev1 2475026 2474677 2026-10-28T22:37:37Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2475026 wikitext text/x-wiki 7iphwvfyg2rik2o60giya68mop3y42g 2475027 2475026 2026-10-28T22:38:33Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2475027 wikitext text/x-wiki g5olm5y1vdoy23k7xoqba3hygb2ae1i 0 7919 2474568 2474567 2026-10-28T12:00:07Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync 2474568 wikitext text/x-wiki kdm0d6mxov3och9doebg5rqus51hyss 2474569 2474568 2026-10-28T12:00:14Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync 2474569 wikitext text/x-wiki j27jdk1gx5wz98ujdz9tw7vbb5wjdqn 2474570 2474569 2026-10-28T12:00:39Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica 2474570 wikitext text/x-wiki nbk1wpfztiil1sjwhhno3p4ijk7lbl6 2474571 2474570 2026-10-28T12:01:02Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica 2474571 wikitext text/x-wiki dlnfahcw57xkeb97edi8u1lkcygjxp9 2474572 2474571 2026-10-28T12:12:51Z Stashbot 7414 hnowlan@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase 2474572 wikitext text/x-wiki 3pvu8900z8c1xaedab66bwjsn9rzyjl 2474573 2474572 2026-10-28T12:15:53Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.dns.netbox 2474573 wikitext text/x-wiki lf5vc89qa8wa6e1puh4opfxqns9ucoj 2474574 2474573 2026-10-28T12:20:26Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for moved IPs on cr1-eqiad - cmooney@cumin1004" 2474574 wikitext text/x-wiki 7w9cg2kjfd24vudqx6ud3j402xut291 2474576 2474574 2026-10-28T12:20:46Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for moved IPs on cr1-eqiad - cmooney@cumin1004" 2474576 wikitext text/x-wiki 6fupbzdz8gtt4rehj8jjfjw9wxieccf 2474577 2474576 2026-10-28T12:20:51Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2474577 wikitext text/x-wiki 5w0wtenq0q5du6ynxe01mlzd65dxti2 2474578 2474577 2026-10-28T12:21:55Z Stashbot 7414 topranks: trunk all row A vlans from asw2-a-eqiad ae1 to ssw1-a1-eqiad lag1 T441014 2474578 wikitext text/x-wiki 63i28jpaxr92soq1optk0yi64kf48a5 2474579 2474578 2026-10-28T12:22:28Z Stashbot 7414 hnowlan@cumin1004: START - Cookbook sre.dns.wipe-cache statsd.eqiad.wmnet on all recursors 2474579 wikitext text/x-wiki 4lgtfj8bg4cdc8wps9za9pk2c6vmlf4 2474580 2474579 2026-10-28T12:22:33Z Stashbot 7414 hnowlan@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) statsd.eqiad.wmnet on all recursors 2474580 wikitext text/x-wiki pje7sevxpf9qh7cb79ek6vbzqs3268s 2474581 2474580 2026-10-28T12:23:07Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5004.*} and A:liberica 2474581 wikitext text/x-wiki s6b15cp98qeshiwzx6c4zmpohrb2kco 2474582 2474581 2026-10-28T12:23:31Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5004.*} and A:liberica 2474582 wikitext text/x-wiki lflt62wbtwd3f9yt1fp5qw639e2rag4 2474583 2474582 2026-10-28T12:25:53Z Stashbot 7414 topranks: re-enabling et-0/1/0 sub-interfaces for row A vlans on cr1-eqiad (they are now trunked on asw2-a-eqiad side to ssw1-a1-eqiad so no repeat of last time expected) T441014 2474583 wikitext text/x-wiki dgr0n8nhym3fvb88ipdbpmo8xihxjlu 2474584 2474583 2026-10-28T12:26:15Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica 2474584 wikitext text/x-wiki ifcwmfiecpdqoskmzbgcdm1jw5486kp 2474585 2474584 2026-10-28T12:26:38Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica 2474585 wikitext text/x-wiki nvapivm2p5lq9q6ehe6r6i3igf4ozez 2474586 2474585 2026-10-28T12:29:01Z Stashbot 7414 hnowlan@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase-codfw 2474586 wikitext text/x-wiki 48itr5mrw6w51g73prsmtdlpsb67aux 2474587 2474586 2026-10-28T12:29:08Z Stashbot 7414 mvolz@deploy1003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid 2474587 wikitext text/x-wiki mbkyzqmr00lv3i99vu86if1940dxc7z 2474588 2474587 2026-10-28T12:31:02Z Stashbot 7414 mvolz@deploy1003: Finished scap deploy-service citoid: update (duration: 50m 46s) 2474588 wikitext text/x-wiki b6938r6eas76z7hfo1p1vw2dfz3blhj 2474589 2474588 2026-10-28T12:32:02Z Stashbot 7414 topranks: revert VRRP changes on cr2-eqiad so cr1-eqiad takes over for some of eqiad row A subnets T441014 2474589 wikitext text/x-wiki rvfxxapkup8j9rycgm815914h9fh7my 2474590 2474589 2026-10-28T12:32:51Z Stashbot 7414 hnowlan@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase-eqiad 2474590 wikitext text/x-wiki hrjg8fopjbzemlhai8virgt0amzlzti 2474591 2474590 2026-10-28T12:34:55Z Stashbot 7414 topranks: make cr1-eqiad take over VRRP for the remaining row A vlans 2474591 wikitext text/x-wiki lrpgn67mfswaijs5cn5fiasecflxctj 2474592 2474591 2026-10-28T12:36:14Z Stashbot 7414 hnowlan@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase-codfw 2474592 wikitext text/x-wiki fjfwn3pjyk32m1lrw55x13c6dcbbcig 2474593 2474592 2026-10-28T12:40:01Z Stashbot 7414 hnowlan@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase-eqiad 2474593 wikitext text/x-wiki ahl7h6okra5xzwlbg5ao3p9dr0t5c1e 2474596 2474593 2026-10-28T12:48:03Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1359803|Add update maintenance script to drop watchlist rows for temp users (T439059)]] (1.48.0-wmf.1) 2474596 wikitext text/x-wiki pak0rg049pm7wnln79k4gg8jrlxeabi 2474598 2474596 2026-10-28T12:51:15Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply 2474598 wikitext text/x-wiki c349n948pnrc62mu4h5ao81ohw8pee6 2474599 2474598 2026-10-28T12:51:29Z Stashbot 7414 topranks: disable ae1 on cr2-eqiad connecting to asw2-a-eqiad (traffic is flowing via cr1-eqiad) T441014 2474599 wikitext text/x-wiki coje4yehoabvyjyzc01pvs1s683utn1 2474600 2474599 2026-10-28T12:51:34Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply 2474600 wikitext text/x-wiki rdvm1der49w85l8lh6v5qdx15h2s9n1 2474601 2474600 2026-10-28T12:52:15Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1359803|Add update maintenance script to drop watchlist rows for temp users (T439059)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474601 wikitext text/x-wiki 9h9dyzhtmj13ughdk0dmxbhopdydd1w 2474602 2474601 2026-10-28T12:53:51Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply 2474602 wikitext text/x-wiki tlagmuxpmqycqwtcdh1ylx3x7qbslu2 2474603 2474602 2026-10-28T12:54:17Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply 2474603 wikitext text/x-wiki 7d3q7fkn4gnddpvaqambpbtky4gmx45 2474604 2474603 2026-10-28T12:56:17Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2474604 wikitext text/x-wiki f62lmrxbnmk8gormjywxovujmgbj34u 2474605 2474604 2026-10-28T12:57:24Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply 2474605 wikitext text/x-wiki ttsli1v7zpk03p5oro9n2crmlfgjfy1 2474606 2474605 2026-10-28T12:57:49Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply 2474606 wikitext text/x-wiki fhljhrgm3zui567dqfm5wxj3knqmaim 2474607 2474606 2026-10-28T12:58:09Z Stashbot 7414 kart_: Updated cxserver to 2026-10-28-122929-production 2474607 wikitext text/x-wiki e5w3bt9f9m02s6n3553p58fmff3xwul 2474610 2474607 2026-10-28T13:02:01Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1359803|Add update maintenance script to drop watchlist rows for temp users (T439059)]] (1.48.0-wmf.1) (duration: 13m 59s) 2474610 wikitext text/x-wiki 7b8qwvr98k48gcm3ky80vnkfndztnsa 2474613 2474610 2026-10-28T13:07:00Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for Squashed diff to master (minus i18n) ([[gerrit:1359789|1.47.0-wmf.24]], [[gerrit:1359790|1.48.0-wmf.1]]) 2474613 wikitext text/x-wiki 3h8v1z5ons9y2oija5pig5egqbaewxg 2474614 2474613 2026-10-28T13:07:11Z Stashbot 7414 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2474614 wikitext text/x-wiki ob7ntmlwq1wasa9od965izcizzkubej 2474615 2474614 2026-10-28T13:08:56Z Stashbot 7414 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2474615 wikitext text/x-wiki 937idc09gdp1id4goy2fifagkm7gjlw 2474616 2474615 2026-10-28T13:09:18Z Stashbot 7414 Dreamy_Jazz: Running `foreachwikiindblist group0.dblist cleanupWatchlistTemporaryAccountRows.php --wiki=testwiki` for T439059 2474616 wikitext text/x-wiki cbed23ibte7ulr4fq21f9yo28ftqfcq 2474617 2474616 2026-10-28T13:09:33Z Stashbot 7414 Dreamy_Jazz: Running `foreachwikiindblist group0.dblist cleanupWatchlistTemporaryAccountRows.php` for T439059 2474617 wikitext text/x-wiki eqslnco9dyxp346hpu5zoax1k3qnoei 2474618 2474617 2026-10-28T13:11:31Z Stashbot 7414 mlitn@deploy1003: mlitn: Backport for Squashed diff to master (minus i18n) ([[gerrit:1359789|1.47.0-wmf.24]], [[gerrit:1359790|1.48.0-wmf.1]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474618 wikitext text/x-wiki l5q33trkm03hppspzgz6ar053p6a2db 2474619 2474618 2026-10-28T13:12:27Z Stashbot 7414 mlitn@deploy1003: mlitn: Continuing with deployment 2474619 wikitext text/x-wiki 00bmmpmqgtpt1kh7beb4gatfirtl23u 2474620 2474619 2026-10-28T13:12:51Z Stashbot 7414 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2337.codfw.wmnet 2474620 wikitext text/x-wiki 0ijestcd6qmr4d4rztj7tiqqmh5amys 2474621 2474620 2026-10-28T13:15:32Z Stashbot 7414 blake@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker2337.codfw.wmnet 2474621 wikitext text/x-wiki myt945grr7r6um3hxpm9pb3v6nia33r 2474622 2474621 2026-10-28T13:16:30Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.backup.offsite 2474622 wikitext text/x-wiki 5yqbioquanb0l35rr3742zxo9zbmzwm 2474623 2474622 2026-10-28T13:16:35Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0) 2474623 wikitext text/x-wiki 3jd7qhpjn8yr8xzr0g1rpi94xpjcf6g 2474624 2474623 2026-10-28T13:18:05Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for Squashed diff to master (minus i18n) ([[gerrit:1359789|1.47.0-wmf.24]], [[gerrit:1359790|1.48.0-wmf.1]]) (duration: 11m 05s) 2474624 wikitext text/x-wiki 0bx2yby33vs4x7mxfj9bgowflh4i008 2474625 2474624 2026-10-28T13:19:41Z Stashbot 7414 Dreamy_Jazz: Afternoon UTC backport window done 2474625 wikitext text/x-wiki 50e1cticlmd18jy0hf6y02b97pwbpdl 2474626 2474625 2026-10-28T13:20:52Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2474626 wikitext text/x-wiki dz9ua9ez9xaigo5bgr3pp4b3w5nemac 2474627 2474626 2026-10-28T13:21:07Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.backup.offsite 2474627 wikitext text/x-wiki kfzjm25xqgijv08qzgpp6nnju9r2bxl 2474628 2474627 2026-10-28T13:21:11Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0) 2474628 wikitext text/x-wiki hokyjrka5uxm3e5jir95hezzbtjz725 2474629 2474628 2026-10-28T13:21:15Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.backup.offsite 2474629 wikitext text/x-wiki ovxyvx91jyf89eyrw20cmvqxcwn6gto 2474630 2474629 2026-10-28T13:21:20Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0) 2474630 wikitext text/x-wiki fwpeu4dukx9w6b3hlorpud8utdvrvv8 2474631 2474630 2026-10-28T13:22:38Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2474631 wikitext text/x-wiki dhc6s87y2kvysq697jrjjkmymi0xp92 2474632 2474631 2026-10-28T13:22:48Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-lock-scap for datacenter switchover from eqiad to codfw 2474632 wikitext text/x-wiki sn7nzxqekqg59dgvdv1uio0prk3ctqa 2474633 2474632 2026-10-28T13:22:53Z Stashbot 7414 root@deploy1003: Locking from deployment [ALL REPOSITORIES]: Datacenter switchover from eqiad to codfw - 2474633 wikitext text/x-wiki o0aaalfh9jqushlxvi2nuqogg8a1x5c 2474634 2474633 2026-10-28T13:22:58Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.00-lock-scap (exit_code=0) for datacenter switchover from eqiad to codfw 2474634 wikitext text/x-wiki c1ly9258rdmxulm3foozm7dt4qbdtkb 2474635 2474634 2026-10-28T13:24:39Z Stashbot 7414 topranks: move asw2-a-eqiad ae2 ports into ae1 and enable the far-end on ssw1-a8-eqiad where the fibres have been moved T441014 2474635 wikitext text/x-wiki 7e2hnvx1fk9wxh7ln2jayw7bv0bkwzh 2474637 2474635 2026-10-28T13:28:29Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2474637 wikitext text/x-wiki erst3nlwqxhtpv0k54x14y2lxz7ebpm 2474639 2474637 2026-10-28T13:29:57Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2474639 wikitext text/x-wiki ozpmafl1vl90fg4qqksgdv9in3cd1c6 2474645 2474639 2026-10-28T14:00:51Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . 2474645 wikitext text/x-wiki g4bn4xx1wlelebavr8m9sx54oln6o7d 2474646 2474645 2026-10-28T14:04:32Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/mathoid: apply 2474646 wikitext text/x-wiki kq5frowpwwepdb743bjjhy4yugngyba 2474647 2474646 2026-10-28T14:04:42Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474647 wikitext text/x-wiki rtkk8rvpueuv24q0ilbobehhyafxg5o 2474648 2474647 2026-10-28T14:04:49Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/mathoid: apply 2474648 wikitext text/x-wiki acmfmwhggi51f6k2nswvxo0x420wq54 2474649 2474648 2026-10-28T14:04:57Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/mathoid: apply 2474649 wikitext text/x-wiki 1uyetlnnzupdfbwjvi6w6df0wotwlif 2474650 2474649 2026-10-28T14:05:12Z Stashbot 7414 tchin@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply 2474650 wikitext text/x-wiki pjwo7qxps0vqpb4ij00wa7k02erskkt 2474651 2474650 2026-10-28T14:05:29Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/mathoid: apply 2474651 wikitext text/x-wiki rpppuirnm9p9k9f0qb1mlgh3u99q6ed 2474652 2474651 2026-10-28T14:05:46Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/mathoid: apply 2474652 wikitext text/x-wiki aimuycrahda2vgatbwrrsps8fo0cjce 2474653 2474652 2026-10-28T14:06:09Z Stashbot 7414 bjensen: homer "lsw1-e3-codfw" commit "remove wikikube-worker2337 (T442714)" 2474653 wikitext text/x-wiki s3wlei1op4u1fb6xja9nz9kynket7ro 2474654 2474653 2026-10-28T14:06:15Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mathoid: apply 2474654 wikitext text/x-wiki 5gws9hp4dncbk6umpwgnv881mc0m410 2474659 2474654 2026-10-28T14:09:54Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.dns.netbox 2474659 wikitext text/x-wiki l8estl3gbrxo1m2ujiwn0kx3w6rywzs 2474661 2474659 2026-10-28T14:11:54Z Stashbot 7414 btullis@dns1004: START - running authdns-update 2474661 wikitext text/x-wiki 69m394tclw4ykzcg49473ei465j4tmx 2474662 2474661 2026-10-28T14:12:24Z Stashbot 7414 jgreen@dns2004: START - running authdns-update 2474662 wikitext text/x-wiki 4t3b8a5szmdpt9iyi1rfbnjox1l3g0o 2474663 2474662 2026-10-28T14:14:39Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row A on cr2-eqiad - cmooney@cumin1004" 2474663 wikitext text/x-wiki sn57399mpmf3a6s5dxx2qhvsvkyr8if 2474664 2474663 2026-10-28T14:15:02Z Stashbot 7414 jgreen@dns2004: START - running authdns-update 2474664 wikitext text/x-wiki cqr4eg1rf7vmwixm7xj6abuz97nnv5a 2474665 2474664 2026-10-28T14:15:10Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row A on cr2-eqiad - cmooney@cumin1004" 2474665 wikitext text/x-wiki np8o4fsruha2w15o8d4icygpj48vjm4 2474666 2474665 2026-10-28T14:15:14Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2474666 wikitext text/x-wiki n3pyb2u71vslty0c5v48td8s2buhn7g 2474667 2474666 2026-10-28T14:15:40Z Stashbot 7414 btullis@dns1004: START - running authdns-update 2474667 wikitext text/x-wiki 39b3pvd1n4tp5ehw0rx4t4wh71e5qa7 2474668 2474667 2026-10-28T14:18:14Z Stashbot 7414 btullis@dns1004: END - running authdns-update 2474668 wikitext text/x-wiki kqm1jy9yzjiy0cadndq0atsbmqhfwd6 2474670 2474668 2026-10-28T14:21:41Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5004.eqsin.wmnet with OS trixie 2474670 wikitext text/x-wiki 7kse9jh1eoumz4isi6l1keewtfejk6d 2474672 2474670 2026-10-28T14:22:43Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (push config only) 2474672 wikitext text/x-wiki rc7aqabz7d9pjxbhcstb5leo8ite9bo 2474673 2474672 2026-10-28T14:22:51Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (push config only) 2474673 wikitext text/x-wiki 62s2for6q3qk4mjadv7mr9drhr8ffoy 2474676 2474673 2026-10-28T14:27:59Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw 2474676 wikitext text/x-wiki qey71c2k9t4vzxmygn1vphei37plpjo 2474679 2474676 2026-10-28T14:29:05Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2474679 wikitext text/x-wiki 9mujx9em8luh4wih75oevhs83pj4wt3 2474681 2474679 2026-10-28T14:29:31Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2474681 wikitext text/x-wiki dj8d8mc03e6w5npzacq90x9tvk97uar 2474682 2474681 2026-10-28T14:30:43Z Stashbot 7414 ebysans@deploy1003: helmfile [staging] START helmfile.d/services/edit-analytics: apply 2474682 wikitext text/x-wiki cjv2phl7b1gkz5t43up4atqps68ll9z 2474683 2474682 2026-10-28T14:30:56Z Stashbot 7414 ebysans@deploy1003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply 2474683 wikitext text/x-wiki 9avur2bb88op4k64almauy75veac78q 2474684 2474683 2026-10-28T14:31:01Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2474684 wikitext text/x-wiki 4ne5bpnezmu0r5otb63lpfyckyhmecy 2474686 2474684 2026-10-28T14:31:35Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2474686 wikitext text/x-wiki mwqvrz8lpk10f83efh58igwl3rxsdxc 2474689 2474686 2026-10-28T14:33:43Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . 2474689 wikitext text/x-wiki 6p397fj9d90ahe5pc5xgwlnsr94z7je 2474690 2474689 2026-10-28T14:37:08Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . 2474690 wikitext text/x-wiki rp3xczaf9xd07gh3e7boi23acebn7gn 2474691 2474690 2026-10-28T14:38:02Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (remove config only) 2474691 wikitext text/x-wiki dn40llsq7fgyxsepjv65iyp2t74q9x7 2474692 2474691 2026-10-28T14:38:14Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (remove config only) 2474692 wikitext text/x-wiki d3gb8ls52s69m0kk5sb6f5b43hiaziq 2474693 2474692 2026-10-28T14:38:18Z Stashbot 7414 vgutierrez@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5004.eqsin.wmnet with OS trixie 2474693 wikitext text/x-wiki 9xfvjf4lr2hz6iisrswmfxcmzi9zw5q 2474694 2474693 2026-10-28T14:38:30Z Stashbot 7414 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 3:00:00 on asw2-b2-eqiad,cr[1-2]-eqiad,ssw1-a[1,8]-eqiad with reason: extending so down port alerts dont fire 2474694 wikitext text/x-wiki a4944j779pmjx23l2j03m1xlvt3u8nq 2474695 2474694 2026-10-28T14:39:56Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.reimage for host lvs5004.eqsin.wmnet with OS trixie 2474695 wikitext text/x-wiki qshsc9munl8gjyo469f1pbjozt1nk0k 2474696 2474695 2026-10-28T14:40:01Z Stashbot 7414 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on asw2-b-eqiad,cr[1-2]-eqiad,ssw1-a[1,8]-eqiad with reason: extending so down port alerts dont fire 2474696 wikitext text/x-wiki jydbf0rj2kyhune96bzr5c5fyjf9iwx 2474697 2474696 2026-10-28T14:40:07Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (push config only) 2474697 wikitext text/x-wiki qo79kucxiano7avcryw50aixx9rqigj 2474698 2474697 2026-10-28T14:40:15Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (push config only) 2474698 wikitext text/x-wiki t5rvngh2luikdachlqcivpmd64b151v 2474699 2474698 2026-10-28T14:41:06Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply 2474699 wikitext text/x-wiki jygdl3u8qysfknjzt04juc0udiupaci 2474700 2474699 2026-10-28T14:41:54Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply 2474700 wikitext text/x-wiki prnd4nhv0w7337d9207km1cyu47rvp4 2474701 2474700 2026-10-28T14:42:57Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . 2474701 wikitext text/x-wiki 0q6k6glxib4wruik28r4m97phjvqzx7 2474702 2474701 2026-10-28T14:43:48Z Stashbot 7414 slyngshede@cumin1004: END (ERROR) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=97) for datacenter switchover from eqiad to codfw 2474702 wikitext text/x-wiki fz8fei8dzsr1ist4a2e2sxo0ipfoe86 2474703 2474702 2026-10-28T14:43:57Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw 2474703 wikitext text/x-wiki acfice46h2kb2cdsqjeorrhvzz51otl 2474704 2474703 2026-10-28T14:49:28Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from eqiad to codfw 2474704 wikitext text/x-wiki k53zxyw5mbt9f8o4yjvzd1qmjlkl0js 2474705 2474704 2026-10-28T14:49:44Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqiad to codfw 2474705 wikitext text/x-wiki i1pjont4k8yek34gzp61alxekoofanm 2474706 2474705 2026-10-28T14:50:00Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from eqiad to codfw 2474706 wikitext text/x-wiki 70eidb9xt6n4gr87knub2htrdqlpejf 2474709 2474706 2026-10-28T14:52:31Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . 2474709 wikitext text/x-wiki 7gxc6yk5kd75d56kaakxf2jak2k42nk 2474710 2474709 2026-10-28T14:54:43Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw 2474710 wikitext text/x-wiki k47kjybvqnmmjjlfwje5yzb77tecbkw 2474711 2474710 2026-10-28T14:56:38Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw 2474711 wikitext text/x-wiki jplux6f2nqzp2ikhuuepnfkx0yqnnn4 2474713 2474711 2026-10-28T15:02:43Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw 2474713 wikitext text/x-wiki 4hjuqj903ydpml3l8zfbthlujxl9we3 2474714 2474713 2026-10-28T15:05:32Z Stashbot 7414 slyngshede@cumin1004: MediaWiki read-only period ends at: 2026-10-28 15:05:31.302917 2474714 wikitext text/x-wiki kii40yk9yxkpyjx8ei05o1bkt2zhhhp 2474715 2474714 2026-10-28T15:05:35Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw 2474715 wikitext text/x-wiki ncympsvv1rlyfx08vgefececezqx2iv 2474716 2474715 2026-10-28T15:06:14Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw 2474716 wikitext text/x-wiki 1sydhs24axnywvztlrli5xe8sltqvvw 2474717 2474716 2026-10-28T15:06:16Z Stashbot 7414 root@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: sync 2474717 wikitext text/x-wiki g7yz8v3nxbjvcojr3yk04ih0u0zrikt 2474718 2474717 2026-10-28T15:06:31Z Stashbot 7414 root@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: sync 2474718 wikitext text/x-wiki an2vchquxhm7qn91jem4uh7ms16c3qh 2474719 2474718 2026-10-28T15:06:34Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from eqiad to codfw 2474719 wikitext text/x-wiki gakrb7zc9vklhctpeszfkkgezd3nyye 2474721 2474719 2026-10-28T15:06:43Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw 2474721 wikitext text/x-wiki 9j965tbsjozf6lsol1umizmazwibwas 2474722 2474721 2026-10-28T15:06:46Z Stashbot 7414 root@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply 2474722 wikitext text/x-wiki 69i9o2pjx9qkwu060h65zzfvxnimhaq 2474723 2474722 2026-10-28T15:07:37Z Stashbot 7414 root@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply 2474723 wikitext text/x-wiki tseqncwc794ytwr43b5bccx8n1d93wy 2474725 2474723 2026-10-28T15:08:25Z Stashbot 7414 root@deploy1003: helmfile [codfw] START helmfile.d/services/mw-cron: apply 2474725 wikitext text/x-wiki ooufjoyfyhszb921oyk61r1rt5e23yn 2474727 2474725 2026-10-28T15:09:54Z Stashbot 7414 root@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-cron: apply 2474727 wikitext text/x-wiki 8gp0u0jz7eaypoegyvu48uqkrk2gx05 2474728 2474727 2026-10-28T15:10:14Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw 2474728 wikitext text/x-wiki 87fidcrzat830jdcdo8muzcr0eamxxl 2474729 2474728 2026-10-28T15:10:31Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw 2474729 wikitext text/x-wiki by5l800oge0o4qipevi4impln2x8yog 2474730 2474729 2026-10-28T15:10:59Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from eqiad to codfw 2474730 wikitext text/x-wiki p48pnpx5erl7ljvpedutlxdz0kgwbf5 2474731 2474730 2026-10-28T15:11:21Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2474731 wikitext text/x-wiki 3t2pnrhp07v4au5ktqtaiv5lvcxqmpi 2474732 2474731 2026-10-28T15:13:56Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2474732 wikitext text/x-wiki 04er0eteq7r16gmc6vxbgh9xe0huxl9 2474733 2474732 2026-10-28T15:16:05Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw 2474733 wikitext text/x-wiki gojzrujo7oaqnw4bgcfxd78e9n3e6wa 2474734 2474733 2026-10-28T15:16:48Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2474734 wikitext text/x-wiki 4pb38vc8g3qunrlp61ctq99fzq85gdp 2474735 2474734 2026-10-28T15:16:51Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2474735 wikitext text/x-wiki 9sa92xxiaf2grg3exrfpfih15sbjtvl 2474736 2474735 2026-10-28T15:17:03Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2474736 wikitext text/x-wiki sin6qr27io0kn3u7vejof7l3pyqsmw4 2474739 2474736 2026-10-28T15:19:40Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2474739 wikitext text/x-wiki 73xfi802v3rum2sp68rcqji4nq70fyi 2474740 2474739 2026-10-28T15:20:41Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2474740 wikitext text/x-wiki b0zqy9uh29arprk89ua744m0rpjr43b 2474741 2474740 2026-10-28T15:21:21Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2474741 wikitext text/x-wiki 74hjrqtgnnhugqa7bqv9akx10s1pdss 2474742 2474741 2026-10-28T15:21:31Z Stashbot 7414 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply 2474742 wikitext text/x-wiki nd3w5rd4rms89c8a4wv741h1f8noq44 2474743 2474742 2026-10-28T15:21:42Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2474743 wikitext text/x-wiki fjcj52hi0lfhbnode7snrcpyucoglej 2474744 2474743 2026-10-28T15:21:45Z Stashbot 7414 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply 2474744 wikitext text/x-wiki 86hfwv7uw3uftd49fp8fycy52cut81k 2474745 2474744 2026-10-28T15:22:09Z Stashbot 7414 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply 2474745 wikitext text/x-wiki eikj631zbge3v4x0lg5tt44sncanagp 2474746 2474745 2026-10-28T15:22:24Z Stashbot 7414 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply 2474746 wikitext text/x-wiki kc5y1t6y2nj07dnoxg8vaq2ywtnajxy 2474747 2474746 2026-10-28T15:22:36Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2474747 wikitext text/x-wiki sp5pqpfcun1983cvgktvtmqpzj4vttd 2474748 2474747 2026-10-28T15:22:47Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2474748 wikitext text/x-wiki j729lxmsbu9v56ic5xb1fcg2g58n6ql 2474749 2474748 2026-10-28T15:23:08Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2474749 wikitext text/x-wiki qlyrw6pmgkekh6ue471pepsgn22gv63 2474750 2474749 2026-10-28T15:27:18Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (remove config only) 2474750 wikitext text/x-wiki mjdjapxa1wniake3rn64fu86x5fb2ii 2474751 2474750 2026-10-28T15:27:30Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (remove config only) 2474751 wikitext text/x-wiki gr0g1patwfffj44j7vddydvdkk3z0p1 2474752 2474751 2026-10-28T15:28:33Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from eqiad to codfw 2474752 wikitext text/x-wiki lh5fclvbaf657ik1igxz8o9p7ybatl1 2474753 2474752 2026-10-28T15:28:56Z Stashbot 7414 slyngshede@cumin1004: START - Cookbook sre.switchdc.mediawiki.09-unlock-scap for datacenter switchover from eqiad to codfw 2474753 wikitext text/x-wiki pdpd9n8b8u0d4ubm75n3bmnpeweco0q 2474754 2474753 2026-10-28T15:28:59Z Stashbot 7414 root@deploy1003: Forcefully removed global lock: Datacenter switchover from eqiad to codfw - 2474754 wikitext text/x-wiki qcghm4mfwwyyp20k97595u6ofibeoe9 2474755 2474754 2026-10-28T15:29:03Z Stashbot 7414 root@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Datacenter switchover from eqiad to codfw - (duration: 126m 08s) 2474755 wikitext text/x-wiki 81s78d6ntqosi1yecc80oruejt99u1r 2474756 2474755 2026-10-28T15:29:06Z Stashbot 7414 slyngshede@cumin1004: END (PASS) - Cookbook sre.switchdc.mediawiki.09-unlock-scap (exit_code=0) for datacenter switchover from eqiad to codfw 2474756 wikitext text/x-wiki d3nugbrfpk20fif353g3hlp63d29xxs 2474757 2474756 2026-10-28T15:30:24Z Stashbot 7414 slyngshede@deploy1003: Started scap sync-world: Backport for [[gerrit:1319818|debug.json: order codfw (primary) DC backends first (T439674)]] 2474757 wikitext text/x-wiki 10yyhyxhkx6rmjai6h230sszygkb0d1 2474758 2474757 2026-10-28T15:31:09Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: sync 2474758 wikitext text/x-wiki fhcjw2c2b63855buler8l65p4z5wvml 2474759 2474758 2026-10-28T15:31:13Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: sync 2474759 wikitext text/x-wiki n3dvwjg0qsdygjtxjib7a1giv5gf7i7 2474761 2474759 2026-10-28T15:31:58Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage 2474761 wikitext text/x-wiki b6kdbsfvx4e7ynk2j0af82r3ko6mk19 2474762 2474761 2026-10-28T15:34:51Z Stashbot 7414 slyngshede@deploy1003: slyngshede: Backport for [[gerrit:1319818|debug.json: order codfw (primary) DC backends first (T439674)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474762 wikitext text/x-wiki cmts7txpqc6sr9r6o8iqrq20e8wgska 2474763 2474762 2026-10-28T15:35:06Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage 2474763 wikitext text/x-wiki c3shfp5a83ann7wyuisq6pena4spq8o 2474764 2474763 2026-10-28T15:36:00Z Stashbot 7414 slyngshede@deploy1003: slyngshede: Continuing with deployment 2474764 wikitext text/x-wiki cj3584pbrfpejyyvp7ecsqqbwalbmze 2474766 2474764 2026-10-28T15:41:44Z Stashbot 7414 slyngshede@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319818|debug.json: order codfw (primary) DC backends first (T439674)]] (duration: 11m 20s) 2474766 wikitext text/x-wiki 98t5fzbcs2r5259oe0rmqfx4bn1z48d 2474767 2474766 2026-10-28T15:53:01Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2474767 wikitext text/x-wiki n6jp69c2es1jeb06pmpxyxhln33kgws 2474768 2474767 2026-10-28T15:54:22Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2474768 wikitext text/x-wiki kjis6lv4t6z4trg7kc5givnzt0gz58a 2474769 2474768 2026-10-28T15:56:43Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5004.eqsin.wmnet with OS trixie 2474769 wikitext text/x-wiki qj7w9nxja2ctx3wyvfkurvrso78oysu 2474770 2474769 2026-10-28T16:06:50Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2474770 wikitext text/x-wiki 5nk6h0ackauk6pcooivf1spia3yfwsi 2474771 2474770 2026-10-28T16:07:37Z Stashbot 7414 moritzm: installing apache2 security updates 2474771 wikitext text/x-wiki 6n7k8q2ue3spe6wd6af8wuw71tqqf5u 2474772 2474771 2026-10-28T16:07:45Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2474772 wikitext text/x-wiki 24pckzjru02ywa7y18k60k7sc1157up 2474773 2474772 2026-10-28T16:11:06Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs2005.codfw.wmnet 2474773 wikitext text/x-wiki gwpx9eimanxnhzf0zeqrchwdqe90qig 2474779 2474773 2026-10-28T16:17:46Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs2005.codfw.wmnet 2474779 wikitext text/x-wiki f5vx3l6jizpd48ufjucv4zxhmwxwz97 2474782 2474779 2026-10-28T16:19:45Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs1004.eqiad.wmnet 2474782 wikitext text/x-wiki g3ckrwywlfgme15b8buvu1wd0iftmuf 2474783 2474782 2026-10-28T16:19:49Z Stashbot 7414 vgutierrez@cumin2003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5004.*} and A:liberica 2474783 wikitext text/x-wiki elkwv9njsjl06wf6rwl36zompkk5q0y 2474784 2474783 2026-10-28T16:20:12Z Stashbot 7414 vgutierrez@cumin2003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5004.*} and A:liberica 2474784 wikitext text/x-wiki d1we6nh31v33mgctnt06t3y1cqhigon 2474787 2474784 2026-10-28T16:21:17Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . 2474787 wikitext text/x-wiki 83hxc2dr9y79o8cei0y0pdm5h0yjqrv 2474790 2474787 2026-10-28T16:24:14Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-eqiad 2474790 wikitext text/x-wiki ami05wzbi4pmaomkgdfgt1abeijv9v4 2474791 2474790 2026-10-28T16:24:49Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet 2474791 wikitext text/x-wiki mvie18dde7ly64tzru6ty6wpmg1zxfd 2474792 2474791 2026-10-28T16:26:20Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs1004.eqiad.wmnet 2474792 wikitext text/x-wiki s3ade3wysexnnv3d3rkuahuuj93m17z 2474795 2474792 2026-10-28T16:33:29Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet 2474795 wikitext text/x-wiki adoe701bblp1rr6xnpgydv0kjmnodoq 2474796 2474795 2026-10-28T16:37:01Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . 2474796 wikitext text/x-wiki 9pyzxi0z9xjqe0zpukd4yy2vbyhe9vx 2474797 2474796 2026-10-28T16:39:33Z Stashbot 7414 kamila@dns7001: START - running authdns-update 2474797 wikitext text/x-wiki b27yo9pxvx68ycss8w9ehldqchhkmar 2474798 2474797 2026-10-28T16:41:16Z Stashbot 7414 kamila@cumin1004: START - Cookbook sre.hosts.decommission for hosts deploy2002.codfw.wmnet 2474798 wikitext text/x-wiki tt728svyq6frlyuilbh57gc6rig5klz 2474799 2474798 2026-10-28T16:42:01Z Stashbot 7414 kamila@dns7001: END - running authdns-update 2474799 wikitext text/x-wiki 47x5dv7ubqni1z4zgdpq8cxplig2db7 2474800 2474799 2026-10-28T16:42:20Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet 2474800 wikitext text/x-wiki qu5w248d1fndsmioeasssv1am8oof0o 2474801 2474800 2026-10-28T16:42:30Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet 2474801 wikitext text/x-wiki czol0kod4uj3mhkw7143i3gcgs9bwib 2474802 2474801 2026-10-28T16:43:02Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1044-1057,1064].eqiad.wmnet 2474802 wikitext text/x-wiki 25iutul10fg8a9j1ifvi3wzl0hz9wys 2474804 2474802 2026-10-28T16:46:20Z Stashbot 7414 kamila@cumin1004: START - Cookbook sre.dns.netbox 2474804 wikitext text/x-wiki rw5519hy19grn3ghobtq44q76gr1s6g 2474808 2474804 2026-10-28T16:50:17Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply 2474808 wikitext text/x-wiki gvgki8ukzue7iy0064tknchnnt5gf7d 2474809 2474808 2026-10-28T16:50:24Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply 2474809 wikitext text/x-wiki qbgk4y8wjbe1vse40s74u8kemsz24kz 2474810 2474809 2026-10-28T16:50:56Z Stashbot 7414 kamila@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: deploy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1004" 2474810 wikitext text/x-wiki nb4y51ny4lhgjv55aak3l7q1c8g00ob 2474811 2474810 2026-10-28T16:51:29Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply 2474811 wikitext text/x-wiki 7j7370jqr5ie0i395by7hls3gy25yop 2474812 2474811 2026-10-28T16:51:37Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply 2474812 wikitext text/x-wiki cmxyuc6mz694ata4rrj7alriewms8nc 2474813 2474812 2026-10-28T16:52:20Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply 2474813 wikitext text/x-wiki 9382thq7yibctl3vjggwl8m8tx2egwv 2474814 2474813 2026-10-28T16:52:31Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply 2474814 wikitext text/x-wiki l4xbeom7jlqr5f0otkaxuhohu730xos 2474815 2474814 2026-10-28T16:53:02Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-wdqs1004.eqiad.wmnet 2474815 wikitext text/x-wiki 68yqvnn8hv20tv2wxst6bml5f5x4um5 2474816 2474815 2026-10-28T16:53:11Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-wdqs1004.eqiad.wmnet 2474816 wikitext text/x-wiki djr961azzkfpna943o8qcdoai56r4do 2474817 2474816 2026-10-28T16:53:30Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-wdqs2005.codfw.wmnet 2474817 wikitext text/x-wiki 39wn9zkjj718daor83lagoj7k2om266 2474818 2474817 2026-10-28T16:53:37Z Stashbot 7414 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-wdqs2005.codfw.wmnet 2474818 wikitext text/x-wiki 8glhr8uvkqkrapifo0vm2xh8csc6z4z 2474819 2474818 2026-10-28T16:54:33Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2474819 wikitext text/x-wiki nrhbglf7tdj1jyyac9l076eijuvqgmb 2474820 2474819 2026-10-28T16:54:38Z Stashbot 7414 kamila@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: deploy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1004" 2474820 wikitext text/x-wiki 6ygm527u493lxq2pavhil4spawl5dgs 2474821 2474820 2026-10-28T16:54:41Z Stashbot 7414 kamila@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2474821 wikitext text/x-wiki j213i9e90fo0kzlemx4y9fp7998zsai 2474822 2474821 2026-10-28T16:54:44Z Stashbot 7414 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts deploy2002.codfw.wmnet 2474822 wikitext text/x-wiki mwzgrbgt246wqakeh8xvbmw4v0v6rij 2474823 2474822 2026-10-28T16:54:47Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2474823 wikitext text/x-wiki ox4o3hny3m1sclz11zaahpi0bptgufm 2474826 2474823 2026-10-28T16:56:03Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1044-1057,1064].eqiad.wmnet 2474826 wikitext text/x-wiki inz9afupda4xblspk6ugw105y8v5ijy 2474829 2474826 2026-10-28T17:04:33Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1044-1057,1064].eqiad.wmnet 2474829 wikitext text/x-wiki grxdbytd5xjv6zfd8n2d5jeoeddjzy6 2474830 2474829 2026-10-28T17:04:43Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1044-1057,1064].eqiad.wmnet 2474830 wikitext text/x-wiki ffb0cyhxz6wxviw6ckcu9c7d6r14jhq 2474831 2474830 2026-10-28T17:05:14Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1065-1079].eqiad.wmnet 2474831 wikitext text/x-wiki tqx6rsgaozlkzbjw57xlifdcrfw3l8d 2474832 2474831 2026-10-28T17:08:32Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply 2474832 wikitext text/x-wiki ca2hsgdsh4gwrtqc863497dimoh5m84 2474833 2474832 2026-10-28T17:10:56Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply 2474833 wikitext text/x-wiki 6s45vfv4k72r9whexg6i2k1n07jafvb 2474834 2474833 2026-10-28T17:11:30Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply 2474834 wikitext text/x-wiki fxyuu40ob8vybp7a8v855pt3cluar3d 2474835 2474834 2026-10-28T17:13:44Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply 2474835 wikitext text/x-wiki l52zsnjfnrpo6kl3ov0cs28q9xodykr 2474836 2474835 2026-10-28T17:13:50Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1065-1079].eqiad.wmnet 2474836 wikitext text/x-wiki hu7mhwgj3rzp0cy7jux0fvx9augd4ic 2474842 2474836 2026-10-28T17:22:34Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1065-1079].eqiad.wmnet 2474842 wikitext text/x-wiki 1w4mjgr4n4uzccufxyv1mv6p36xoeie 2474843 2474842 2026-10-28T17:22:45Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1065-1079].eqiad.wmnet 2474843 wikitext text/x-wiki rmqpc744aiqm8mijd1myhxf97oop2uv 2474844 2474843 2026-10-28T17:23:16Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet 2474844 wikitext text/x-wiki q8bqa6u0bx113vcztrtbx8inf0f0af0 2474848 2474844 2026-10-28T17:29:40Z Stashbot 7414 rzl@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2474848 wikitext text/x-wiki 1h1ryk5dnbofgdigefshh5thohog3im 2474849 2474848 2026-10-28T17:29:54Z Stashbot 7414 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2474849 wikitext text/x-wiki 4pxitx2z0o7hxyohq6kecvt3hegeimi 2474850 2474849 2026-10-28T17:32:58Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet 2474850 wikitext text/x-wiki iwmbn9q15y4kjwk2ck79qvw3l5fpo7w 2474851 2474850 2026-10-28T17:33:51Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply 2474851 wikitext text/x-wiki hbb2ln6llhzow1fwdo65cfnppppzp3q 2474852 2474851 2026-10-28T17:34:12Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2474852 wikitext text/x-wiki 5k1qkkepkody4fm57arqikgdv5b2t7k 2474853 2474852 2026-10-28T17:35:14Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2474853 wikitext text/x-wiki b5xwjzxxrn1h3uh1g5z19ehaz2olfd1 2474854 2474853 2026-10-28T17:35:37Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2474854 wikitext text/x-wiki 03fco4shrh6uq3g1a75vuivw7cjmtbt 2474856 2474854 2026-10-28T17:43:15Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet 2474856 wikitext text/x-wiki ttp0wbyk9fs6xhbbeadg777f0uq8nxq 2474857 2474856 2026-10-28T17:43:25Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet 2474857 wikitext text/x-wiki d31i2ts2kua5bjxv69iqit843u4mh7n 2474858 2474857 2026-10-28T17:43:56Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1119-1133].eqiad.wmnet 2474858 wikitext text/x-wiki f8rsaaqwjr37n0puof91nsfxxte7mi0 2474859 2474858 2026-10-28T17:45:16Z Stashbot 7414 aqu@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply 2474859 wikitext text/x-wiki 5n4wo9cnwr9kxb363irjovn63aqm7o7 2474860 2474859 2026-10-28T17:45:47Z Stashbot 7414 aqu@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply 2474860 wikitext text/x-wiki k1w0hq019imfbv7lh6f5jplvfu16u79 2474862 2474860 2026-10-28T17:52:59Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1119-1133].eqiad.wmnet 2474862 wikitext text/x-wiki j9c9cvdyyogbvkxkel52w2daoambj5d 2474865 2474862 2026-10-28T17:56:38Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2474865 wikitext text/x-wiki 02z9a02zckmgokjx6lqjt095fn7g2rv 2474867 2474865 2026-10-28T18:01:35Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1119-1133].eqiad.wmnet 2474867 wikitext text/x-wiki s78g96ft69ou4ovlw0fcr5vs9g321ad 2474868 2474867 2026-10-28T18:01:46Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1119-1133].eqiad.wmnet 2474868 wikitext text/x-wiki 5ktko9i66ardkad67kgwrzut0mawu28 2474869 2474868 2026-10-28T18:02:17Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1134-1148].eqiad.wmnet 2474869 wikitext text/x-wiki h5aabekar6tdtv31cs2cfhskmhbj9ck 2474872 2474869 2026-10-28T18:07:04Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2474872 wikitext text/x-wiki mkarig5gmctuxvfpalhiaf8mbadlpbo 2474874 2474872 2026-10-28T18:10:47Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1134-1148].eqiad.wmnet 2474874 wikitext text/x-wiki ezckl7ge1ouvnycxlwqr4m5nv5ann6h 2474875 2474874 2026-10-28T18:11:00Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2474875 wikitext text/x-wiki o9y42wods5rvpknucf79dmu00x3qp5x 2474876 2474875 2026-10-28T18:17:23Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2474876 wikitext text/x-wiki aempwr85xuxmnswyydbnarbou0nl6uo 2474877 2474876 2026-10-28T18:18:31Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1134-1148].eqiad.wmnet 2474877 wikitext text/x-wiki pdb96voe9jbta8qkeadu3wlmt6r60pa 2474878 2474877 2026-10-28T18:18:41Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1134-1148].eqiad.wmnet 2474878 wikitext text/x-wiki cxw59jafqievfrdtsbnysqc3uv57g3u 2474879 2474878 2026-10-28T18:19:13Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet 2474879 wikitext text/x-wiki 8n4g2uduuhihzwtm4dhy9z02ueqkquc 2474880 2474879 2026-10-28T18:19:17Z Stashbot 7414 rzl@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2474880 wikitext text/x-wiki 3z1bbu3wwmxvatpp29wjm1ibdf560bu 2474881 2474880 2026-10-28T18:19:28Z Stashbot 7414 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2474881 wikitext text/x-wiki nb4yvhj4jhg1pwzq3h0bszch3ssw8k2 2474882 2474881 2026-10-28T18:19:35Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2474882 wikitext text/x-wiki 1vgv2113ye0njl4dxwrs46w842aitdk 2474883 2474882 2026-10-28T18:25:38Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2474883 wikitext text/x-wiki rshrvaj10ap4ssca53nz82recx44y2r 2474884 2474883 2026-10-28T18:25:49Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2474884 wikitext text/x-wiki m46izsxmwyvtjpa3ycn5ns8ru408rmp 2474885 2474884 2026-10-28T18:25:54Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.1 refs T438223 2474885 wikitext text/x-wiki l08b9rlholgjuiw2rsx9d09rdyitt1e 2474886 2474885 2026-10-28T18:26:27Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2474886 wikitext text/x-wiki slff1qyea9ldn59jez8imu8zkz9qnf8 2474887 2474886 2026-10-28T18:30:59Z Stashbot 7414 rzl: rzl@apt1002:~$ sudo -i reprepro -C main includedeb bookworm-wikimedia /srv/wikimedia/pool/component/envoy-future/e/envoyproxy/envoyproxy_1.40.1-1_amd64.deb 2474887 wikitext text/x-wiki ony5pbzp41w5ta1rkc3e46ohjic2102 2474888 2474887 2026-10-28T18:31:35Z Stashbot 7414 rzl: rzl@apt1002:~$ sudo -i reprepro copy trixie-wikimedia bookworm-wikimedia envoyproxy 2474888 wikitext text/x-wiki c3jmxsc5u847d1ur3ece860puu4hkbr 2474889 2474888 2026-10-28T18:31:46Z Stashbot 7414 rzl: rzl@apt1002:~$ sudo -i reprepro copy bullseye-wikimedia bookworm-wikimedia envoyproxy # backwards, intended 2474889 wikitext text/x-wiki nb6nxvf0kw3916fzm6iwe29h136j4t9 2474890 2474889 2026-10-28T18:32:49Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet 2474890 wikitext text/x-wiki bfgow0z1pk4eevumrmpwj19cfbuyjky 2474891 2474890 2026-10-28T18:41:02Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet 2474891 wikitext text/x-wiki kheq6p7mfb627b3nlwj1m580evw5jey 2474892 2474891 2026-10-28T18:41:12Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet 2474892 wikitext text/x-wiki fv1gklkulo7l90ue5j9nqlk2673euex 2474893 2474892 2026-10-28T18:41:43Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1165,1240-1253].eqiad.wmnet 2474893 wikitext text/x-wiki qrzca8njixjr6ob16tw9vdnn1t8jftx 2474894 2474893 2026-10-28T18:45:39Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.307.0" for 155 host(s) 2474894 wikitext text/x-wiki 78276ke831kv71y51v58fbj9as1onba 2474896 2474894 2026-10-28T18:50:44Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1165,1240-1253].eqiad.wmnet 2474896 wikitext text/x-wiki hy87hiilm68i3kwqjj4ljq7woxtp63x 2474898 2474896 2026-10-28T18:57:59Z Stashbot 7414 Dreamy_Jazz: Running `foreachwikiindblist group1.dblist cleanupWatchlistTemporaryAccountRows.php` for T439059 2474898 wikitext text/x-wiki gvm6tpo3umz1kk7t3vcafmihtawl1oj 2474899 2474898 2026-10-28T18:58:22Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.307.0" for 2 host(s) 2474899 wikitext text/x-wiki pu70xl30au2j3ge2pen4ki4y1a669ie 2474900 2474899 2026-10-28T18:59:14Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1165,1240-1253].eqiad.wmnet 2474900 wikitext text/x-wiki r72m3kswsouhcq8f3q5ka1e45e0iq61 2474901 2474900 2026-10-28T18:59:19Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.307.0" completed for 2 hosts 2474901 wikitext text/x-wiki k4epyxa24poxk0ypadj6vkp7zagurhv 2474902 2474901 2026-10-28T18:59:24Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1165,1240-1253].eqiad.wmnet 2474902 wikitext text/x-wiki ch7ehxpap5878nglhlwd7zmml1x8r6s 2474903 2474902 2026-10-28T19:00:00Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1254-1268].eqiad.wmnet 2474903 wikitext text/x-wiki b2cp8719djq0qjruo8fxwwghhk2vbx1 2474906 2474903 2026-10-28T19:09:07Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1254-1268].eqiad.wmnet 2474906 wikitext text/x-wiki f2m657srk8sk7752f0r67p0fw0gsv0t 2474907 2474906 2026-10-28T19:18:08Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1254-1268].eqiad.wmnet 2474907 wikitext text/x-wiki qoj1tvg4gz9i8qitjcxotploys413ib 2474909 2474907 2026-10-28T19:18:18Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1254-1268].eqiad.wmnet 2474909 wikitext text/x-wiki 1e1oypcygl6izq7q9a3wjjsicsvsaqb 2474910 2474909 2026-10-28T19:18:50Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1269-1283].eqiad.wmnet 2474910 wikitext text/x-wiki t2twcq2gbhyx1j09h83pjfsms4ttz3a 2474911 2474910 2026-10-28T19:21:53Z Stashbot 7414 inflatador: [bking@deploy1003] `curl -XPOST -u $U -s https://opensearch-ttmserver.svc.eqiad.wmnet:30443/_cluster/reroute` Fix yellow status 2474911 wikitext text/x-wiki t660vklxs590c0gjybc6tw9td0ya714 2474912 2474911 2026-10-28T19:22:49Z Stashbot 7414 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply 2474912 wikitext text/x-wiki me1icdes9vai6v8igb161a6w1sd9zdr 2474913 2474912 2026-10-28T19:23:53Z Stashbot 7414 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply 2474913 wikitext text/x-wiki m7nuzh57a5kfoytxpi7zllk5tw3ybfp 2474914 2474913 2026-10-28T19:26:11Z Stashbot 7414 akhatun@deploy1003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply 2474914 wikitext text/x-wiki bvl50a5ush49fwg2ya4y870rmw02dib 2474915 2474914 2026-10-28T19:26:28Z Stashbot 7414 akhatun@deploy1003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply 2474915 wikitext text/x-wiki 68d1psk4y1hklq9xg2zutepwto22duo 2474918 2474915 2026-10-28T19:28:33Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1269-1283].eqiad.wmnet 2474918 wikitext text/x-wiki m7xfnrh4d39m74pza4nwi4kva9po881 2474919 2474918 2026-10-28T19:29:45Z Stashbot 7414 akhatun@deploy1003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply 2474919 wikitext text/x-wiki nqsicai6szljrwww0i3z7jexx6sacr0 2474920 2474919 2026-10-28T19:30:14Z Stashbot 7414 akhatun@deploy1003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply 2474920 wikitext text/x-wiki 17le0gwhzysft1n6qk0toom262ktqmd 2474921 2474920 2026-10-28T19:37:13Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1269-1283].eqiad.wmnet 2474921 wikitext text/x-wiki 0in7klqte3utesjn864ggt1p7ztyi1t 2474922 2474921 2026-10-28T19:37:24Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1269-1283].eqiad.wmnet 2474922 wikitext text/x-wiki aswuuexkvxwrcylwtxifjakivh9dghe 2474923 2474922 2026-10-28T19:37:53Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet 2474923 wikitext text/x-wiki 6i8m7xvntmbxcozhpid24qdprhtm9ix 2474925 2474923 2026-10-28T19:40:45Z Stashbot 7414 akhatun@deploy1003: Started scap sync-world: Backport for [[gerrit:1359949|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias ]] 2474925 wikitext text/x-wiki h91hmc7p3b2kr6o2lbaspuiehzxcueu 2474926 2474925 2026-10-28T19:45:06Z Stashbot 7414 akhatun@deploy1003: akhatun, otto: Backport for [[gerrit:1359949|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias ]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474926 wikitext text/x-wiki 8v0kq9wb463lf0dfzseqlp0tztoje9y 2474927 2474926 2026-10-28T19:45:53Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet 2474927 wikitext text/x-wiki dpd6saen4j5uodjjmfy21v7kxz627yq 2474928 2474927 2026-10-28T19:46:52Z Stashbot 7414 akhatun@deploy1003: akhatun, otto: Continuing with deployment 2474928 wikitext text/x-wiki ngik65dkvxeinayf39alcdma3r2zakf 2474929 2474928 2026-10-28T19:49:02Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.dns.netbox 2474929 wikitext text/x-wiki dbplaax9acptjc1h391doyfmn0k4opv 2474930 2474929 2026-10-28T19:52:15Z Stashbot 7414 pt1979@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (live debug) 2474930 wikitext text/x-wiki b1002p2z1bff5o71aowzzdb6vwa90vh 2474931 2474930 2026-10-28T19:52:33Z Stashbot 7414 akhatun@deploy1003: Finished scap sync-world: Backport for [[gerrit:1359949|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias ]] (duration: 11m 48s) 2474931 wikitext text/x-wiki j76aj6uyu0z0f8qlqvu19a2x7vy7igb 2474932 2474931 2026-10-28T19:54:15Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004" 2474932 wikitext text/x-wiki 09t1qq4uj68k12ya4ak68lzdauiuuh1 2474933 2474932 2026-10-28T19:54:20Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004" 2474933 wikitext text/x-wiki thvkljp4g3m703m7senjgmq1fv7fg4y 2474934 2474933 2026-10-28T19:54:23Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2474934 wikitext text/x-wiki 47i2lkwzuo8zdixoidiqozywabcvrx9 2474935 2474934 2026-10-28T19:54:26Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet 2474935 wikitext text/x-wiki eg7q71wia1r748ik9jbkwam67oc4203 2474936 2474935 2026-10-28T19:54:32Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet 2474936 wikitext text/x-wiki c93zulekfh3ix2m31zics2itzpw7nsl 2474937 2474936 2026-10-28T19:55:02Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1301-1314].eqiad.wmnet 2474937 wikitext text/x-wiki 81d1xv7c3fcoyen6a0inbxzfgd08xx7 2474938 2474937 2026-10-28T19:59:13Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.dns.netbox 2474938 wikitext text/x-wiki 7yhokuh9huog49voc4bi5mpv2uusi6a 2474939 2474938 2026-10-28T20:03:08Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004" 2474939 wikitext text/x-wiki p2tsyqrmg2r4bp5pd47yeh5rj5vana2 2474940 2474939 2026-10-28T20:03:11Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004" 2474940 wikitext text/x-wiki hviji5wezlcul5dikcm3v6ntzz5kemr 2474941 2474940 2026-10-28T20:03:14Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2474941 wikitext text/x-wiki ife880jmt9ezv3cxyatf6ft7gzpy640 2474942 2474941 2026-10-28T20:03:36Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1301-1314].eqiad.wmnet 2474942 wikitext text/x-wiki 12385drrud9ri6uw3hpw4jqvtc6knn9 2474943 2474942 2026-10-28T20:06:14Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1358832|Turn off "report a visual" bug on wikipedias (T430970)]] 2474943 wikitext text/x-wiki hirjm0tj84nsx1hqkvy83jt7aafxqrx 2474944 2474943 2026-10-28T20:10:19Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.dns.netbox 2474944 wikitext text/x-wiki 59285i5e43870wyf1b9ekoz2438lmjd 2474945 2474944 2026-10-28T20:10:37Z Stashbot 7414 arlolra@deploy1003: arlolra: Backport for [[gerrit:1358832|Turn off "report a visual" bug on wikipedias (T430970)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2474945 wikitext text/x-wiki k0n9bcq7ealwex064ymt2bu82egt78i 2474946 2474945 2026-10-28T20:11:58Z Stashbot 7414 arlolra@deploy1003: arlolra: Continuing with deployment 2474946 wikitext text/x-wiki e315mvhx7gypnp1xvvx8fj63jq1muub 2474947 2474946 2026-10-28T20:12:06Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1301-1314].eqiad.wmnet 2474947 wikitext text/x-wiki qsqvovkrscxuaix3t99z4pxc4zjz0ip 2474948 2474947 2026-10-28T20:12:15Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1301-1314].eqiad.wmnet 2474948 wikitext text/x-wiki 311a73rv9g4m6l85nyvbtac920i8qjz 2474949 2474948 2026-10-28T20:12:42Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1315-1328].eqiad.wmnet 2474949 wikitext text/x-wiki ik8am18awfz8bwen6jb6tybnu9r99lt 2474950 2474949 2026-10-28T20:14:13Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update last few IPs that got awawy - cmooney@cumin1004" 2474950 wikitext text/x-wiki 7ukxc1vw125mjdzju8y33nax8mo3b7g 2474951 2474950 2026-10-28T20:14:17Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update last few IPs that got awawy - cmooney@cumin1004" 2474951 wikitext text/x-wiki tt209mmvvu1aehgzv1ejb9zx3j6t7wj 2474952 2474951 2026-10-28T20:14:19Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2474952 wikitext text/x-wiki etmsvzls5k12e1jmc1uj8ts6brsbqwp 2474955 2474952 2026-10-28T20:17:39Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1358832|Turn off "report a visual" bug on wikipedias (T430970)]] (duration: 11m 26s) 2474955 wikitext text/x-wiki hgpuelckbum8j2q2jzrfgvm33ciuavo 2474957 2474955 2026-10-28T20:19:38Z Stashbot 7414 mutante: etherpad1004 - apache httpd disabled by puppet - etherpad running normally 2474957 wikitext text/x-wiki qi6zp260m6qw1qjb7cwtwkds8onkans 2474958 2474957 2026-10-28T20:20:34Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1315-1328].eqiad.wmnet 2474958 wikitext text/x-wiki a007w9w4qjmo18p9w321vlsgj8xfb9n 2474960 2474958 2026-10-28T20:29:06Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1315-1328].eqiad.wmnet 2474960 wikitext text/x-wiki mjzcsrq7zdnz67jyrzi9lb8hzyesx09 2474961 2474960 2026-10-28T20:29:16Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1315-1328].eqiad.wmnet 2474961 wikitext text/x-wiki jpbfqlw6iih3nfmmsesdanaa9862bax 2474962 2474961 2026-10-28T20:29:44Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1329-1342].eqiad.wmnet 2474962 wikitext text/x-wiki c8wd3sar80bugzz8kz82ff8t1lyyb4i 2474967 2474962 2026-10-28T20:40:47Z Stashbot 7414 sbassett: Deployed security mitigations for T435455, T439844, T442538 2474967 wikitext text/x-wiki sk2n5udxeqxiklofu9kyigyhvc8gueo 2474969 2474967 2026-10-28T20:42:34Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1329-1342].eqiad.wmnet 2474969 wikitext text/x-wiki 3ju0923lta7mjdv9l0jx6t781a45m3e 2474970 2474969 2026-10-28T20:50:21Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1329-1342].eqiad.wmnet 2474970 wikitext text/x-wiki jbafptohnpi1wpshl52avnzsbqs55j2 2474971 2474970 2026-10-28T20:50:30Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1329-1342].eqiad.wmnet 2474971 wikitext text/x-wiki swcwnjtrip23f1xb4m9l4vj1df6enj4 2474972 2474971 2026-10-28T20:50:58Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1343-1356].eqiad.wmnet 2474972 wikitext text/x-wiki j2pqnel6980a8d07xsf9814t2a63fzx 2474973 2474972 2026-10-28T20:59:23Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1343-1356].eqiad.wmnet 2474973 wikitext text/x-wiki 0qtzthaxzql5gk3o9lyfp0euwiw49xb 2474974 2474973 2026-10-28T21:00:48Z Stashbot 7414 jforrester@deploy1003: Started scap deploy-service wikifunctions: Weekly function-evaluator service deploy 2474974 wikitext text/x-wiki 4ej5r5jlydk3jt02v7lsow2bkx9j3nk 2474975 2474974 2026-10-28T21:00:52Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions 2474975 wikitext text/x-wiki nnpv1x5ktmxyjfzylwwy27g3n9oxb6y 2474976 2474975 2026-10-28T21:01:25Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions 2474976 wikitext text/x-wiki 1wjiy79bf79upmhavufv9f8gh26wy7m 2474977 2474976 2026-10-28T21:01:54Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions 2474977 wikitext text/x-wiki ajh1o76lu2ht3ynpunh21juyuo4qhbd 2474978 2474977 2026-10-28T21:02:17Z Stashbot 7414 jforrester@deploy1003: Finished scap deploy-service wikifunctions: Weekly function-evaluator service deploy (duration: 01m 29s) 2474978 wikitext text/x-wiki jyl5snthqin1286z0s9gb2huql4uoxw 2474979 2474978 2026-10-28T21:06:17Z Stashbot 7414 jforrester@deploy1003: Started scap deploy-service wikifunctions: Weekly f-e service deploy, this time with a bump commit 2474979 wikitext text/x-wiki ia0bcwuaug1a1hxm94w5zzv3bii0vbz 2474980 2474979 2026-10-28T21:06:20Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions 2474980 wikitext text/x-wiki 5h91ads0hhrajelgigb39fwuz1m0vpv 2474981 2474980 2026-10-28T21:06:42Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1343-1356].eqiad.wmnet 2474981 wikitext text/x-wiki 435jmm7599foootpzxnq3w2o78eodnp 2474982 2474981 2026-10-28T21:06:51Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1343-1356].eqiad.wmnet 2474982 wikitext text/x-wiki 20fhwnhj12s2pxngvu3enbhroru4wf0 2474983 2474982 2026-10-28T21:07:18Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions 2474983 wikitext text/x-wiki s4lh0vw972d0ugg402h95oq9ajnrjal 2474984 2474983 2026-10-28T21:07:21Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1357-1370].eqiad.wmnet 2474984 wikitext text/x-wiki qkly3y0vmq4eekrvv8ybw470p3hp8gq 2474985 2474984 2026-10-28T21:07:43Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions 2474985 wikitext text/x-wiki biejdmwokvj108ct7lexjvheqqyc3b6 2474986 2474985 2026-10-28T21:08:28Z Stashbot 7414 jforrester@deploy1003: Finished scap deploy-service wikifunctions: Weekly f-e service deploy, this time with a bump commit (duration: 02m 11s) 2474986 wikitext text/x-wiki rg2j61jdf3e1p6pc1vh8xatomn8cwiz 2474987 2474986 2026-10-28T21:12:55Z Stashbot 7414 jforrester@deploy1003: Started scap deploy-service wikifunctions: Weekly f-o service deploy 2474987 wikitext text/x-wiki teaxnayqv2vb7ccgsppwqvkg5lvxbuf 2474988 2474987 2026-10-28T21:12:58Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions 2474988 wikitext text/x-wiki h06agmskxegi5z5f06berrvo0n9ade2 2474989 2474988 2026-10-28T21:13:20Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions 2474989 wikitext text/x-wiki mwhm92ppntk2eeafz5saey9figbsayl 2474990 2474989 2026-10-28T21:14:01Z Stashbot 7414 jforrester@deploy1003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions 2474990 wikitext text/x-wiki ibjgct4mzkxasi1d9in7lrats463cga 2474991 2474990 2026-10-28T21:14:42Z Stashbot 7414 jforrester@deploy1003: Finished scap deploy-service wikifunctions: Weekly f-o service deploy (duration: 01m 47s) 2474991 wikitext text/x-wiki otgj07lcvownfi9x3s9znr8904fopkv 2474992 2474991 2026-10-28T21:17:00Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1357-1370].eqiad.wmnet 2474992 wikitext text/x-wiki h2cnc0ol576vxon94iqfezsra8iy8e9 2474993 2474992 2026-10-28T21:18:55Z Stashbot 7414 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply 2474993 wikitext text/x-wiki mb7gyfq94rl229guyyinhczsl7zk8ez 2474994 2474993 2026-10-28T21:19:02Z Stashbot 7414 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply 2474994 wikitext text/x-wiki 2q2akovf8tsgga1opflvkd3nkzwdv0i 2474995 2474994 2026-10-28T21:20:38Z Stashbot 7414 akhatun@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply 2474995 wikitext text/x-wiki 9846mov4h9bs4ni7se0hc0th64nhvy2 2474996 2474995 2026-10-28T21:20:54Z Stashbot 7414 akhatun@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply 2474996 wikitext text/x-wiki g46ibuhi0oldzofi4eyvct7lzqynntd 2474997 2474996 2026-10-28T21:21:16Z Stashbot 7414 akhatun@deploy1003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply 2474997 wikitext text/x-wiki g85urt5oyfnzzkaakno0f5wgov4idhi 2474998 2474997 2026-10-28T21:21:36Z Stashbot 7414 akhatun@deploy1003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply 2474998 wikitext text/x-wiki bxjfq1jplrpn01hdmjt9hz66c0trr9s 2474999 2474998 2026-10-28T21:21:46Z Stashbot 7414 akhatun@deploy1003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply 2474999 wikitext text/x-wiki skx98di9ek2hgi10nrgpp2g8xkmydws 2475000 2474999 2026-10-28T21:22:06Z Stashbot 7414 akhatun@deploy1003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply 2475000 wikitext text/x-wiki 26u2n1tvo5ududu0djc0nkpoipguyqv 2475001 2475000 2026-10-28T21:24:54Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1357-1370].eqiad.wmnet 2475001 wikitext text/x-wiki k87i7iepyqruy9cq3xnwq7pnevnqdl4 2475002 2475001 2026-10-28T21:25:03Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1357-1370].eqiad.wmnet 2475002 wikitext text/x-wiki tjidgrvy8kdik8k968su0wtv1sts6yk 2475003 2475002 2026-10-28T21:25:32Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1371-1384].eqiad.wmnet 2475003 wikitext text/x-wiki nddf9gsn38booubquodr63xxcdbitxh 2475004 2475003 2026-10-28T21:26:07Z Stashbot 7414 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (live debug) 2475004 wikitext text/x-wiki k8sxqi7rsqv5wnnjmls49cad743c95b 2475005 2475004 2026-10-28T21:33:54Z Stashbot 7414 mutante: deploy1003 - sudo find . -path '*/.git/DEPLOY_HEAD' -exec sed -i -E '/^git_(server|scheme|deploy_dir):/d' {} + (backup in /root/DEPLOY_HEAD-backup.tgz) - T197470 2475005 wikitext text/x-wiki hph5o5hg7liw83y6szddr637lycpx7q 2475006 2475005 2026-10-28T21:34:21Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1371-1384].eqiad.wmnet 2475006 wikitext text/x-wiki b0n47pvjmcrltmvcsxsgguwxp7731iq 2475009 2475006 2026-10-28T21:42:05Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1371-1384].eqiad.wmnet 2475009 wikitext text/x-wiki 1y3zq9v14qi0ymvww041o58s3vyshpv 2475010 2475009 2026-10-28T21:42:14Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1371-1384].eqiad.wmnet 2475010 wikitext text/x-wiki c5ys4n8jpaxrokxepqvc531uk82w8vo 2475011 2475010 2026-10-28T21:42:18Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp1001.eqiad.wmnet 2475011 wikitext text/x-wiki sfdkza5llgdksdxu7ow4ct4w0r0hmu8 2475012 2475011 2026-10-28T21:42:51Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp1001.eqiad.wmnet 2475012 wikitext text/x-wiki 6e166rrd3ju98ru3w0xdzewf4yq89x6 2475013 2475012 2026-10-28T21:47:00Z Stashbot 7414 jasmine@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp1001.eqiad.wmnet 2475013 wikitext text/x-wiki 4gxt8jm8mvpkp57kh6b5hyrwntugj87 2475014 2475013 2026-10-28T21:47:03Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp1001.eqiad.wmnet 2475014 wikitext text/x-wiki nka5cu8vid7ckvyogsfxmzwyaav13qz 2475015 2475014 2026-10-28T21:47:06Z Stashbot 7414 jasmine@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-eqiad 2475015 wikitext text/x-wiki ji89p2dlv5xgpftg7qcevflp57nmtrt 2475016 2475015 2026-10-28T21:51:38Z Stashbot 7414 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikimania2007wiki --logwiki=metawiki MogosCZ Mogoskii # T442713 2475016 wikitext text/x-wiki qcjuupn0dxcy69ltyiuct4jnr4et70s 2475017 2475016 2026-10-28T22:05:11Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1359380|Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705)]] 2475017 wikitext text/x-wiki sebrb3uq4hm1nmqiga0q4xixh65b5sf 2475018 2475017 2026-10-28T22:09:27Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1359380|Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2475018 wikitext text/x-wiki pc1zpftekvxx0jifritqyzn6dm3d4fp 2475019 2475018 2026-10-28T22:11:40Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2475019 wikitext text/x-wiki 1zqisbmqf36180b2hrtfodnpyh1d4td 2475020 2475019 2026-10-28T22:17:24Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1359380|Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705)]] (duration: 12m 13s) 2475020 wikitext text/x-wiki ovbg0bqi6oisp4vntsfwysenw5mqspe 2475021 2475020 2026-10-28T22:20:10Z Stashbot 7414 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1357787|Disable Discord QuickSurvey]] 2475021 wikitext text/x-wiki 4kl7fg35tw0p9mvflbyoyjrzg4qpo3z 2475022 2475021 2026-10-28T22:24:28Z Stashbot 7414 egardner@deploy1003: derenrich, egardner: Backport for [[gerrit:1357787|Disable Discord QuickSurvey]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2475022 wikitext text/x-wiki ha9qhtnegrblo50x6njii0t8e0170jk 2475023 2475022 2026-10-28T22:25:12Z Stashbot 7414 egardner@deploy1003: derenrich, egardner: Continuing with deployment 2475023 wikitext text/x-wiki dq9dyf9cdbdy6g3rijhhhgwsgbiqq9t 2475024 2475023 2026-10-28T22:30:51Z Stashbot 7414 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1357787|Disable Discord QuickSurvey]] (duration: 10m 41s) 2475024 wikitext text/x-wiki br6zwhddb7jut1gldpe4sajr6asomk5 2475025 2475024 2026-10-28T22:35:04Z Stashbot 7414 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1359934|ShareCard: Show page views as a monthly rate (T439164)]] (1.48.0-wmf.1), [[gerrit:1359937|ShareCard: Hide images on the bad image list (T439161)]] (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) ([[gerrit:1359972|1.47.0-wmf.24]], [[gerrit:1359974|1.48.0-wmf.1]]) 2475025 wikitext text/x-wiki 43ewwtnp3ln0ap8uudxnq5izj4l1127 2475030 2475025 2026-10-28T22:56:54Z Stashbot 7414 egardner@deploy1003: sbassett, egardner: Backport for [[gerrit:1359934|ShareCard: Show page views as a monthly rate (T439164)]] (1.48.0-wmf.1), [[gerrit:1359937|ShareCard: Hide images on the bad image list (T439161)]] (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) ([[gerrit:1359972|1.47.0-wmf.24]], [[gerrit:1359974|1.48.0-wmf.1]]) synced to the testservers (see https://wikitech.wikimedia.org/ 2475030 wikitext text/x-wiki 7hwhvi5m6j1s09ymdqc8p2pratd0qeq 2475031 2475030 2026-10-28T23:00:02Z Stashbot 7414 egardner@deploy1003: sbassett, egardner: Continuing with deployment 2475031 wikitext text/x-wiki oy3u5jwzdn3iajqkenmop04g5co48dj 2475032 2475031 2026-10-28T23:12:02Z Stashbot 7414 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1359934|ShareCard: Show page views as a monthly rate (T439164)]] (1.48.0-wmf.1), [[gerrit:1359937|ShareCard: Hide images on the bad image list (T439161)]] (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) ([[gerrit:1359972|1.47.0-wmf.24]], [[gerrit:1359974|1.48.0-wmf.1]]) (duration: 36m 58s) 2475032 wikitext text/x-wiki ra9cinrbbugmwul2w5u20obk1j8pdlv 2475035 2475032 2026-10-29T00:00:05Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync 2475035 wikitext text/x-wiki 8jthidf5mj3m4vwptekk1huf4xity8f 2475036 2475035 2026-10-29T00:00:09Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync 2475036 wikitext text/x-wiki 7q6j6g1v6n7y0wxv1gm7nsfpzonqdc3 2475042 2475036 2026-10-29T04:11:19Z Stashbot 7414 herron@cumin1004: START - Cookbook sre.mysql.depool depool db1262: depooling replica after host flap 2475042 wikitext text/x-wiki 75nn140ta8wtjaw23nkokwicncso54l 2475043 2475042 2026-10-29T04:14:10Z Stashbot 7414 herron@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: depooling replica after host flap 2475043 wikitext text/x-wiki 8izob6aegyiagc9vtgu9pf0eh2c08th 2475044 2475043 2026-10-29T05:20:37Z Stashbot 7414 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2475044 wikitext text/x-wiki 23azdyxn0wi2huaowo6qq78dh5emx1j 2475045 2475044 2026-10-29T05:23:50Z Stashbot 7414 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2475045 wikitext text/x-wiki fldygy8kg6fbqaqtb8ik9yuqjzvczrb 2475046 2475045 2026-10-29T05:27:21Z Stashbot 7414 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . 2475046 wikitext text/x-wiki 32b37dyd6ti7nyhzac83lu5nkf915vv 2475047 2475046 2026-10-29T05:45:26Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply 2475047 wikitext text/x-wiki 51xf9ls92aotun8mvwosmm6xmyduyuh 2475049 2475047 2026-10-29T05:48:21Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2475049 wikitext text/x-wiki pxnrkslwidqsc9aw1nsud3mecj2qn3f 2475050 2475049 2026-10-29T05:48:26Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . 2475050 wikitext text/x-wiki s2jfyocibpm6gswbklm51u0n5saxxly 2475051 2475050 2026-10-29T05:48:39Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply 2475051 wikitext text/x-wiki rrpnb0fm7lcqqtjofbnl6o2xddcwtx6 2475052 2475051 2026-10-29T05:49:57Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply 2475052 wikitext text/x-wiki 8wy5yhtw39a4727t5auwll7wcynvamy 2475053 2475052 2026-10-29T05:54:49Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply 2475053 wikitext text/x-wiki 3jsw2x5pxljwbagz08fxmzvw61woyfe 2475054 2475053 2026-10-29T06:00:58Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2475054 wikitext text/x-wiki 7pu9omoux5h4kwbn5pwhud90pehkrpr 2475055 2475054 2026-10-29T06:03:06Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply 2475055 wikitext text/x-wiki f96g1pdbh5n7m9ohlszqomzn484ekw4 2475056 2475055 2026-10-29T06:10:23Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply 2475056 wikitext text/x-wiki d7gx8kkcndjpfa4cbjrdli1hjh8rion 2475060 2475056 2026-10-29T06:35:49Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x1 2475060 wikitext text/x-wiki 33095qtfhoelkq8sn3dwqxd4xo5lmw0 2475061 2475060 2026-10-29T06:36:18Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x1 2475061 wikitext text/x-wiki nt1sfketv5444mj0m33jlqjwsqubei7 2475063 2475061 2026-10-29T06:37:58Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x3 2475063 wikitext text/x-wiki mfomzqhmppzogy42kp6nl9a50xbjuen 2475064 2475063 2026-10-29T06:38:15Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x3 2475064 wikitext text/x-wiki 2ys6uvtpkcv1k7wmbzucaz321j3a7ji 2475065 2475064 2026-10-29T06:40:25Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x4 2475065 wikitext text/x-wiki ffzpdlbekl5y0mrt9bk7op9uaxhhzcr 2475066 2475065 2026-10-29T06:40:57Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x4 2475066 wikitext text/x-wiki no1cwb4ohf2vkkrlfdl1hyaib078gnp 2475067 2475066 2026-10-29T06:43:29Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section es6 2475067 wikitext text/x-wiki bti4l30inosdsam69mpxwc2x9uj8brc 2475068 2475067 2026-10-29T06:44:00Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section es6 2475068 wikitext text/x-wiki kx7zwemact6igp1ogwng2k54vf6sysz 2475069 2475068 2026-10-29T06:46:02Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section es7 2475069 wikitext text/x-wiki i2bez5alu07mng1ppapv0m5y039gi7k 2475070 2475069 2026-10-29T06:46:44Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section es7 2475070 wikitext text/x-wiki 0yc24jjjd1yclchufzb221o0tojua3w 2475071 2475070 2026-10-29T06:47:19Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s6 2475071 wikitext text/x-wiki gbtmbi0u8kr75gukt808x6pfl1zvt05 2475072 2475071 2026-10-29T06:47:37Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s6 2475072 wikitext text/x-wiki 5t9nig6rk2jmba3gsyg246rpess0kuq 2475073 2475072 2026-10-29T06:49:23Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s5 2475073 wikitext text/x-wiki 6why96jcx5xonolpg4ktlcyecxhx80d 2475074 2475073 2026-10-29T06:49:51Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s5 2475074 wikitext text/x-wiki cphvmiq8bea92nat2txuvxj097dpphc 2475075 2475074 2026-10-29T06:50:48Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s2 2475075 wikitext text/x-wiki q3sntgvudwg37ry1ddpyczkd10ctyi3 2475076 2475075 2026-10-29T06:51:10Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s2 2475076 wikitext text/x-wiki 3m0glwo12aefmt7j0b8lo30ikgzj8tl 2475077 2475076 2026-10-29T06:51:47Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s3 2475077 wikitext text/x-wiki oxwu76mb3qe6qwwirbh53cw7dn4guop 2475078 2475077 2026-10-29T06:52:03Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s3 2475078 wikitext text/x-wiki ghomp530xijeix3ck8nsh1t7kjfbej9 2475079 2475078 2026-10-29T06:52:26Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s7 2475079 wikitext text/x-wiki 3yx4av6ij32h5b5vaeiq24mh6onz1x6 2475080 2475079 2026-10-29T06:52:43Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s7 2475080 wikitext text/x-wiki pi9rhc24nzh21mh33nyo06yn6wd39t6 2475081 2475080 2026-10-29T06:53:08Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s8 2475081 wikitext text/x-wiki 3fdh6rvrjorpyj5mb0era0bdr1ls1ov 2475082 2475081 2026-10-29T06:53:22Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s8 2475082 wikitext text/x-wiki floj4hfdycnbu2fchkatxuvd28glka7 2475083 2475082 2026-10-29T06:53:43Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s4 2475083 wikitext text/x-wiki p3dvtawmwnj4ttv2xyqb80k3s2vqyvc 2475084 2475083 2026-10-29T06:53:59Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s4 2475084 wikitext text/x-wiki fwbanphu8e8540c6z1yswa7j6ibouht 2475085 2475084 2026-10-29T06:55:38Z Stashbot 7414 marostegui@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s1 2475085 wikitext text/x-wiki lj7fjjx1mfbou6ml3ve36q2mk5a9jqf 2475086 2475085 2026-10-29T06:56:15Z Stashbot 7414 marostegui@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s1 2475086 wikitext text/x-wiki 2bxqbhvhaslqws1ecmz91op7wflsnam 2475087 2475086 2026-10-29T06:56:25Z Stashbot 7414 marostegui: DB circular replication has been disabled everywhere T436501 2475087 wikitext text/x-wiki 8iqjuef3tirx0nietltcalf1m44wwcy 2475088 2475087 2026-10-29T07:43:36Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2475088 wikitext text/x-wiki rhzx4w2tcxj973tynxk5w0yxidn4mms 2475089 2475088 2026-10-29T07:43:40Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . 2475089 wikitext text/x-wiki t3os81sjlo9fm1xh3y3ro9sandjvr4x 2475093 2475089 2026-10-29T08:30:10Z Stashbot 7414 moritzm: imported Jenkins 2.580.2 to thirdparty/jenkins for trixie-wikimedia T442030 2475093 wikitext text/x-wiki mahgtwufpvgsbto46nc4ag5gekkg018 2475094 2475093 2026-10-29T08:30:38Z Stashbot 7414 hashar: Restarted Gerrit on gerrit2002/gerrit2003 (spare/replica) to switch to Java 21 | T392465 2475094 wikitext text/x-wiki f0fa27al8bl2h8l0i2w2g0n9cljwp0q 2475095 2475094 2026-10-29T08:34:14Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs6003.drmrs.wmnet with OS trixie 2475095 wikitext text/x-wiki 306mukns4p1k4xbvurglbdv7dzeths8 2475097 2475095 2026-10-29T08:35:11Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6003.drmrs.wmnet (push config only) 2475097 wikitext text/x-wiki 213ragaxnh91m207z6pvs5yg0ffcp1q 2475098 2475097 2026-10-29T08:35:16Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6003.drmrs.wmnet (push config only) 2475098 wikitext text/x-wiki 1g9ti0kb7l2gi2vtvtdeymdcp2kyhi5 2475101 2475098 2026-10-29T08:38:26Z Stashbot 7414 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on gerrit1003.wikimedia.org with reason: Java upgrade 2475101 wikitext text/x-wiki jm1msva3uva8t2rtebx3nxvrz5xg9f3 2475102 2475101 2026-10-29T08:38:59Z Stashbot 7414 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 14 hosts with reason: Java upgrade 2475102 wikitext text/x-wiki 7abkscp5v388cjpyqc917ljgwpyo5z0 2475103 2475102 2026-10-29T08:42:38Z Stashbot 7414 hashar: restarting Gerrit 2475103 wikitext text/x-wiki f5mghoikm3p9th5iwcauy9j59zonu6f 2475104 2475103 2026-10-29T08:48:57Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6003.drmrs.wmnet (remove config only) 2475104 wikitext text/x-wiki cefo19m7wpusqv55nm1dj0sdnp6y9zx 2475105 2475104 2026-10-29T08:49:04Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6003.drmrs.wmnet (remove config only) 2475105 wikitext text/x-wiki aqnoqomztm411zbu0dbv7c3yprj5be6 2475106 2475105 2026-10-29T08:51:58Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6003.drmrs.wmnet with reason: host reimage 2475106 wikitext text/x-wiki iezxqwc3ef0hiqukpf4fjztgvdx8lvc 2475107 2475106 2026-10-29T08:56:59Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6003.drmrs.wmnet with reason: host reimage 2475107 wikitext text/x-wiki g2e23r5cartoi776quskhkklakn63lx 2475108 2475107 2026-10-29T09:16:27Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6003.drmrs.wmnet with OS trixie 2475108 wikitext text/x-wiki 3if927cp3vjjm68orbx0bvr4l1qb0iw 2475109 2475108 2026-10-29T09:17:39Z Stashbot 7414 moritzm: installing ca-certificates updates on Bookworm 2475109 wikitext text/x-wiki d7m52x7gc3561rd15u6fog4jfpyyepd 2475111 2475109 2026-10-29T09:23:52Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2475111 wikitext text/x-wiki 63me2s1vpzarizo9vgto43hb67kcby8 2475112 2475111 2026-10-29T09:25:12Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2475112 wikitext text/x-wiki 6jxg0jubhweib6pgfls66k9530a6rh7 2475113 2475112 2026-10-29T09:27:59Z Stashbot 7414 jiji@cumin2003: START - Cookbook sre.mysql.depool depool db1228: Depooling replica - T442815 2475113 wikitext text/x-wiki dsrfro34j9hnbfxdacs5u2ac0at6d14 2475114 2475113 2026-10-29T09:28:12Z Stashbot 7414 jiji@cumin2003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1228: Depooling replica - T442815 2475114 wikitext text/x-wiki 05x8hwc5medpxvw77e7t29fmtntnn4r 2475115 2475114 2026-10-29T09:29:00Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . 2475115 wikitext text/x-wiki bv03cijp2f3zd9hzwc4ptb0deayhnvn 2475117 2475115 2026-10-29T09:32:36Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2475117 wikitext text/x-wiki eexskario9itbe30wu07aa869piyaw3 2475118 2475117 2026-10-29T09:32:59Z Stashbot 7414 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad 2475118 wikitext text/x-wiki enlrni67i55p3otwqdisbewjw7wqie3 2475119 2475118 2026-10-29T09:33:07Z Stashbot 7414 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=eqiad 2475119 wikitext text/x-wiki rspdwxx2biw3d67mco17gqx02tfgivs 2475120 2475119 2026-10-29T09:33:17Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2475120 wikitext text/x-wiki s1gafp2d4ndev56p1t0vivrlryo8zcs 2475123 2475120 2026-10-29T09:34:27Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . 2475123 wikitext text/x-wiki 603t8klf2b7u8cgerea08m3i3ct88v0 2475125 2475123 2026-10-29T09:36:10Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2475125 wikitext text/x-wiki g8r44yd50fif36y1hpoe441v1dahht4 2475126 2475125 2026-10-29T09:37:25Z Stashbot 7414 moritzm: installing Bind security updates 2475126 wikitext text/x-wiki 5gvur0b07spqtl00onxukyevos910jm 2475127 2475126 2026-10-29T09:38:25Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2475127 wikitext text/x-wiki qykfmqehj7pgo3zogqe507w7vynvot9 2475128 2475127 2026-10-29T09:40:43Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6002.*} and A:liberica 2475128 wikitext text/x-wiki dhkb4h810ipp6b4rgi53vtwk1u1skq9 2475129 2475128 2026-10-29T09:41:01Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6002.*} and A:liberica 2475129 wikitext text/x-wiki er8in47nksnbeu8zdbgxd86rqa94y2y 2475130 2475129 2026-10-29T09:41:34Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . 2475130 wikitext text/x-wiki dev0e7wkl6tih7qz5za94gxly61tol9 2475131 2475130 2026-10-29T09:45:37Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2475131 wikitext text/x-wiki bn9jg1p50e1y3k134gmwa3feqlmta8y 2475132 2475131 2026-10-29T09:46:53Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs6002.drmrs.wmnet with OS trixie 2475132 wikitext text/x-wiki tfbn2miftq9t6prv9hy44nklk86oeuk 2475133 2475132 2026-10-29T09:47:50Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6002.drmrs.wmnet (push config only) 2475133 wikitext text/x-wiki 58prcko10egsjao042xq40p2a9padun 2475134 2475133 2026-10-29T09:47:56Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6002.drmrs.wmnet (push config only) 2475134 wikitext text/x-wiki efi945tx0lmgmvdiuqmv6ftqxqq9zpx 2475135 2475134 2026-10-29T09:48:45Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2475135 wikitext text/x-wiki daxg9butntxn0i8oql58v06d4f2kh7q 2475136 2475135 2026-10-29T09:49:26Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . 2475136 wikitext text/x-wiki 5qcmuaxl22724yt8s0820gzzo784exz 2475137 2475136 2026-10-29T09:52:17Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2475137 wikitext text/x-wiki 8cpjif1tbqvjlhawsa8xxf4k6a86ojm 2475139 2475137 2026-10-29T09:53:57Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2475139 wikitext text/x-wiki snxhfrqkit21m7ivkr2yx2qsx4odkqr 2475141 2475139 2026-10-29T09:59:31Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . 2475141 wikitext text/x-wiki 6276c44mzv7p5vv2ymondv3hnqekip8 2475142 2475141 2026-10-29T10:01:15Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6002.drmrs.wmnet (remove config only) 2475142 wikitext text/x-wiki hse92mnaei5zdz0fl19373xgojtygod 2475143 2475142 2026-10-29T10:01:22Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6002.drmrs.wmnet (remove config only) 2475143 wikitext text/x-wiki 25sj3xew2ppabbl5quujp2p9hroizba 2475145 2475143 2026-10-29T10:04:15Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6002.drmrs.wmnet with reason: host reimage 2475145 wikitext text/x-wiki 38dv8oaysvsqzzmo9jocydzfcvgqlkb 2475146 2475145 2026-10-29T10:04:40Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2475146 wikitext text/x-wiki 9ud78yfqldomirxzoze5jb0wyffkrti 2475147 2475146 2026-10-29T10:05:42Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2475147 wikitext text/x-wiki 26o4fnnud8yv15g7mgt3zrd5m873wph 2475148 2475147 2026-10-29T10:07:16Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6002.drmrs.wmnet with reason: host reimage 2475148 wikitext text/x-wiki t1sn21ltke7jt1sgrb6ignyozjowt8w 2475149 2475148 2026-10-29T10:08:34Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . 2475149 wikitext text/x-wiki 08p2tbyuebmvgt59a1mkzulcugqwbql 2475150 2475149 2026-10-29T10:11:39Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2475150 wikitext text/x-wiki 88muiatv561tsjmtx7y2e17csqn6iph 2475151 2475150 2026-10-29T10:12:40Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2475151 wikitext text/x-wiki 1xapliu3701djxadyktlxeoozje9gwd 2475152 2475151 2026-10-29T10:13:58Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . 2475152 wikitext text/x-wiki jtev9t1hg3fdge513eruw23n868a1yd 2475153 2475152 2026-10-29T10:24:23Z Stashbot 7414 moritzm: installing Unbound security updates 2475153 wikitext text/x-wiki 3h6hy6t7nee5flfvdad3svmc0ujoxcu 2475154 2475153 2026-10-29T10:26:05Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6002.drmrs.wmnet with OS trixie 2475154 wikitext text/x-wiki ekli5m52ym59eyernnnjxk2yv4jeam7 2475155 2475154 2026-10-29T10:27:02Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Enable non-batch k8s native sidecars (T417800) 2475155 wikitext text/x-wiki i5rducs9cqbrxp0xm88hhlbxeyiyl6z 2475158 2475155 2026-10-29T10:34:22Z Stashbot 7414 blake@deploy1003: Rolling back deployment 2475158 wikitext text/x-wiki 7x2v80zifect28fss26sqzzebalbqsu 2475159 2475158 2026-10-29T10:34:31Z Stashbot 7414 moritzm: installing Apache httpd security updates 2475159 wikitext text/x-wiki b2z2m939gp5atx9wx3r1gm1dc9k9h07 2475161 2475159 2026-10-29T10:36:57Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet 2475161 wikitext text/x-wiki nd0p7a9oiutb705kabcirmub42obsbn 2475162 2475161 2026-10-29T10:40:27Z Stashbot 7414 blake@deploy1003: Failed scap sync-world: Enable non-batch k8s native sidecars (T417800) (duration: 14m 32s) 2475162 wikitext text/x-wiki i2ql2xt6vmrwt7wm97yk3pth4si0yfm 2475163 2475162 2026-10-29T10:40:45Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet 2475163 wikitext text/x-wiki buc6xlyygann4uy6pf3z012vfpcr43b 2475164 2475163 2026-10-29T10:41:18Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Enable non-batch k8s native sidecars (T417800) 2475164 wikitext text/x-wiki nr098rsubp2ggo6aib35mh9ncoax9ke 2475166 2475164 2026-10-29T10:48:53Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Enable non-batch k8s native sidecars (T417800) (duration: 07m 55s) 2475166 wikitext text/x-wiki eitlj81nrevf6ggjgdnlo1qkwlb09sm 2475167 2475166 2026-10-29T10:49:11Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6002.*} and A:liberica 2475167 wikitext text/x-wiki pj0gl7azlvzdpf62zd1shjvthztjs9v 2475168 2475167 2026-10-29T10:49:17Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6002.*} and A:liberica 2475168 wikitext text/x-wiki j37aqw12k7n6cvz0739p1ud0quey00n 2475172 2475168 2026-10-29T10:59:16Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6001.*} and A:liberica 2475172 wikitext text/x-wiki k916q0hd7mt41si74olz3ny8dryid4z 2475173 2475172 2026-10-29T10:59:35Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6001.*} and A:liberica 2475173 wikitext text/x-wiki 44fvyior6vtr6e2s966a4bwnno0jz16 2475174 2475173 2026-10-29T11:06:12Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs6001.drmrs.wmnet with OS trixie 2475174 wikitext text/x-wiki hgtjia05z7fwklsm1n08oa3qot7u92r 2475175 2475174 2026-10-29T11:07:09Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6001.drmrs.wmnet (push config only) 2475175 wikitext text/x-wiki 8o66y9ue82x8senut6zz6bngsmr76ec 2475176 2475175 2026-10-29T11:07:14Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6001.drmrs.wmnet (push config only) 2475176 wikitext text/x-wiki 4noqld22rwiiddsm2t1nae6iytjnmi7 2475181 2475176 2026-10-29T11:18:33Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1360375|Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768)]] (1.48.0-wmf.1), [[gerrit:1360000|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.48.0-wmf.1) 2475181 wikitext text/x-wiki pgoujlq4rtlgjhanj3q8kw9cmdz3tsx 2475182 2475181 2026-10-29T11:20:17Z Stashbot 7414 vgutierrez: upload purged 0.26 to trixie-wikimedia (apt.wm.o ) - T442736 2475182 wikitext text/x-wiki 6wr77pinx6g22uh2gwpvstydbp69s68 2475183 2475182 2026-10-29T11:20:21Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.dhcp for host lvs6001.drmrs.wmnet (remove config only) 2475183 wikitext text/x-wiki rogods8exfgweha1wnxh3imuth3z9ym 2475184 2475183 2026-10-29T11:20:25Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6001.drmrs.wmnet (remove config only) 2475184 wikitext text/x-wiki 94oxn5hqyllu1vbf3hk20bgnxg7bd28 2475187 2475184 2026-10-29T11:22:42Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.backup.offsite 2475187 wikitext text/x-wiki 60t6et9ezuma562mc9r089pxq64cyj2 2475188 2475187 2026-10-29T11:22:46Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.backup.offsite (exit_code=0) 2475188 wikitext text/x-wiki jiajcuzpxcnppdjf11mj03a7imombab 2475189 2475188 2026-10-29T11:23:12Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6001.drmrs.wmnet with reason: host reimage 2475189 wikitext text/x-wiki s6h4hnk0jehvbjlybt21f8ob23az0gv 2475190 2475189 2026-10-29T11:23:17Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1360375|Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768)]] (1.48.0-wmf.1), [[gerrit:1360000|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2475190 wikitext text/x-wiki emonjnjo3anigusr0e2ffobc7eg6v64 2475191 2475190 2026-10-29T11:27:20Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2475191 wikitext text/x-wiki q0n6fgjrjrzyk4zn3j7vlpsp3hjml5j 2475193 2475191 2026-10-29T11:29:07Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6001.drmrs.wmnet with reason: host reimage 2475193 wikitext text/x-wiki awhdb2sxu3haja0mvrxn8mup4q1wsrh 2475194 2475193 2026-10-29T11:33:08Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1360375|Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768)]] (1.48.0-wmf.1), [[gerrit:1360000|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.48.0-wmf.1) (duration: 14m 35s) 2475194 wikitext text/x-wiki rp2corcxr3q69mh9e5w0pb072vqx7nc 2475199 2475194 2026-10-29T11:47:46Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6001.drmrs.wmnet with OS trixie 2475199 wikitext text/x-wiki k56cic6w3t2s2qlt3g4hm4numvcu7ae 2475200 2475199 2026-10-29T11:53:40Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2475200 wikitext text/x-wiki a3pmwelottgsnvpnuxsxpl4w37slu09 2475201 2475200 2026-10-29T11:57:01Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2475201 wikitext text/x-wiki p1s14qxseu3xhxdvd5j8ezv02mhdv7w 2475202 2475201 2026-10-29T11:57:09Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2475202 wikitext text/x-wiki qs3yfmlekfawyj4tm3sp50iz23qudga 0 17290 2474712 2474376 2026-10-28T15:01:49Z Stashbot 7414 dancy: `SET GLOBAL read_only = 0;` on deployment-db15.deployment-prep.eqiad1.wikimedia.cloud 2474712 wikitext text/x-wiki 59x49zmi4jhbax7upn0d2f8otkxfcnm 2474737 2474712 2026-10-28T15:19:22Z Stashbot 7414 dancy: wmf-beta-update-all operation restored. 2474737 wikitext text/x-wiki qcqoudvngeealyql619k2zx4g9jop0v 2474866 2474737 2026-10-28T17:57:46Z Stashbot 7414 hashar: integration: reenabled in Jenkins the jobs relaying on scm polling or on a timer # https://gerrit.wikimedia.org/r/c/integration/config/+/1357483/ 2474866 wikitext text/x-wiki khobtyjp9d0bhxgnoswufieqyvmeutq 12 18681 2475169 2462279 2026-10-29T10:50:27Z Taavi-WMF 41365 remove php7.4 2475169 wikitext text/x-wiki mq9qa3h1hrpvcyif6s3r578onugbjt4 2475170 2475169 2026-10-29T10:51:22Z Taavi-WMF 41365 /* Other / generic web servers */ not nginx anymore 2475170 wikitext text/x-wiki acyiqcagwkm2f22svbqcjcn72fn8pwa 2475171 2475170 2026-10-29T10:51:47Z Taavi-WMF 41365 /* Response buffering */ not relevant anymore 2475171 wikitext text/x-wiki hn5as2d3crsgp6w5m6rcqzi26vhhv6m 498 30942 2474775 2474168 2026-10-28T16:15:20Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/369 2474775 wikitext text/x-wiki d639p2j44pc41c6976oegt56s0y07rz 2474776 2474775 2026-10-28T16:15:49Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/369 2474776 wikitext text/x-wiki e8qir7ksn0v4mkhv3x3x2psjxnbmivs 2474777 2474776 2026-10-28T16:16:20Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2474777 wikitext text/x-wiki fzj7qkia8vu9pwnfi403q5pmyquy68k 2474778 2474777 2026-10-28T16:16:59Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2474778 wikitext text/x-wiki ab0o258sbu57oqyvng6yshqwlip6yn8 2474780 2474778 2026-10-28T16:18:30Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2474780 wikitext text/x-wiki t47l9z0yzo02jyzvi6jhlkkgy6vve5x 2474781 2474780 2026-10-28T16:18:58Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2474781 wikitext text/x-wiki 3bme2eil1139wrl4bubhloftxm5cxt2 2474785 2474781 2026-10-28T16:20:28Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2474785 wikitext text/x-wiki 81b8s70hb15rvla68zu8ys83vw5irl0 2474786 2474785 2026-10-28T16:20:55Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2474786 wikitext text/x-wiki h2kvjj7pl7prjj0t0hvcdqg2bw53hyr 2474788 2474786 2026-10-28T16:22:10Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2474788 wikitext text/x-wiki 3l2ilq2j8ux89jviuerqi5q5o8qh1rx 2474789 2474788 2026-10-28T16:22:36Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2474789 wikitext text/x-wiki fwo9ccfi7umoxlg8j58nsbplp1p7ynt 2474803 2474789 2026-10-28T16:45:53Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2474803 wikitext text/x-wiki iiqsupymm21shpayhp64333u2shqyvo 2474805 2474803 2026-10-28T16:46:25Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2474805 wikitext text/x-wiki kityoxhtvanxgv5c1leh8t0oih9gc33 2474806 2474805 2026-10-28T16:49:16Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 2474806 wikitext text/x-wiki bkyy4ccsev6r36mw3m2x2pnbm5sh23k 2474807 2474806 2026-10-28T16:49:32Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 2474807 wikitext text/x-wiki 5t49wc6a1ro52r421yhpygky6kb5o2t 2474824 2474807 2026-10-28T16:55:25Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 2474824 wikitext text/x-wiki bm9aw8649movfjyni0qcpr8ol5ya3qe 2474825 2474824 2026-10-28T16:55:40Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 2474825 wikitext text/x-wiki d2o9b1xn6t22a1ktiqiy4m0gsuy56l3 2474827 2474825 2026-10-28T16:56:52Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2474827 wikitext text/x-wiki rwa1r918uo642pastvynvpeb59b65ta 2474828 2474827 2026-10-28T16:57:57Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2474828 wikitext text/x-wiki 1cvxcssanc9so3iwnaivndzje3857xs 498 55577 2474840 2474522 2026-10-28T17:21:34Z Stashbot 7414 taavi: remove default security group rule for ssh access from 10.244.0.0/16 2474840 wikitext text/x-wiki jmt15fyvvrbdh6r52lof6oo92h12t5x 498 213752 2475185 2196502 2026-10-29T11:20:49Z Stashbot 7414 taavi: delete striker-docker-02 (bookworm) which has a too old docker version to run modern striker, and would need an upgrade to get rid of old PHP versions T329687 2475185 wikitext text/x-wiki lk76jaw0qpdwy21c29flvkl7r83etls 498 443628 2474871 2474455 2026-10-28T18:07:01Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507911 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60) 2474871 wikitext text/x-wiki ibq0aedmj9nv7qvgcbnabxxocweq4jh 2474897 2474871 2026-10-28T18:50:52Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217792 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe) 2474897 wikitext text/x-wiki tnenzierw2v4m82i9u3me5d52ejq79p 0 443686 2474608 2474542 2026-10-28T13:01:14Z Mvolz (WMF) 2813 2474608 wikitext text/x-wiki 29lbot5ymx3bmc7duzsaqx92fmenrxs 2474609 2474608 2026-10-28T13:01:57Z Mvolz (WMF) 2813 /* Manually */ 2474609 wikitext text/x-wiki kku7d6w4pb7z8pfe5xpf2bjevh5f542 2474611 2474609 2026-10-28T13:02:15Z Mvolz (WMF) 2813 /* Deprecated API */ don't really need this anymore. 2474611 wikitext text/x-wiki 5iaph9f99891zo3egzio5qtehstptse 2474612 2474611 2026-10-28T13:03:39Z Mvolz (WMF) 2813 /* Config change */ 2474612 wikitext text/x-wiki bryw2jbjznr98cdf16ekril02xk4wmh 498 443949 2475090 2465974 2026-10-29T08:02:11Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 44f14cc (Fix crash when creating a missing grouping page) for T438301 2475090 wikitext text/x-wiki oi6pxnqorncnjrt6fwijlei8m39bfq1 2475091 2475090 2026-10-29T08:02:15Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy be8878d (Match SSE section keys on the last underscore) for T425477 2475091 wikitext text/x-wiki 8t3x8pusp4u10prmw0ylwwf0ayzoj4m 2475092 2475091 2026-10-29T08:02:18Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 9ecbeff (Stream grouping page creation progress over SSE) for T425477 2475092 wikitext text/x-wiki euk0mszyocn6eakbg2xxpzcb9ezlo6c 2475144 2475092 2026-10-29T10:03:40Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 3cfcf64 (Deduplicate Listeria columns differing only by reference check) for T368001 2475144 wikitext text/x-wiki 3lceiv0lworgaekndjgis9oa339quxu 2475156 2475144 2026-10-29T10:32:45Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 415f8be (Mark grouping-page creation complete and fix count wording) for T425477 2475156 wikitext text/x-wiki p75wi1lx3upu498hwccs5iidnzhub66 2475157 2475156 2026-10-29T10:32:48Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 0bd0104 (Hyperlink created grouping pages in live update progress) for T425477 2475157 wikitext text/x-wiki ox7jwli24po7ue29ga8mduv3any481n 0 446037 2475160 2467637 2026-10-29T10:36:52Z LSobanski (WMF) 18783 /* To become the Incident Coordinator (IC) */ Added manual instruction for creating the status document 2475160 wikitext text/x-wiki puk8wb737gr5h58gw8v0oyf2fkirho8 0 447718 2475121 2467819 2026-10-29T09:33:40Z Hashar 32 /* Create new branches */ use explicit refs when creating wmf/stable-XX branch 2475121 wikitext text/x-wiki knqgv8iicw0hhs80deuarg5qdtk22wu 2475124 2475121 2026-10-29T09:35:18Z Hashar 32 /* Create new branches */ 2475124 wikitext text/x-wiki 05xbv8u5hwapvja6vynhmf41x6munus 2475140 2475124 2026-10-29T09:57:00Z Hashar 32 /* Fetch our additional plugins */ 2475140 wikitext text/x-wiki dfivmzwy2anlhcbthjxh544pgb10bss 0 449160 2475037 2474389 2026-10-29T00:01:37Z Dexbot 30554 Bot: Updating the report 2475037 wikitext text/x-wiki 8mgbbvj570v74fnho4egqi0z2s71hji 0 451180 2474846 2456535 2026-10-28T17:26:14Z Krinkle 44 /* Retention */ 2474846 wikitext text/x-wiki mte0h9scj5mtep1dckglshib03zz13a 2474847 2474846 2026-10-28T17:28:03Z Krinkle 44 /* Retention */ 2474847 wikitext text/x-wiki ig75vw3crt897qz2xfdvowibwq1r12b 498 452890 2474873 2474457 2026-10-28T18:07:58Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507977 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60) 2474873 wikitext text/x-wiki ed13ufx92o5x23hbah7940jp6ltn808 0 454546 2474597 2463777 2026-10-28T12:51:00Z KReid-WMF 49588 /* Instrumentation and events data */ Update Metrics Platform to Test Kitchen 2474597 wikitext text/x-wiki f7wbpjl7tkdvik57ag2bxtpnga58vhm 0 457955 2474636 2474267 2026-10-28T13:24:47Z ASanford-WMF 52089 Update instructions to include generation and upload of key attestation files. 2474636 wikitext text/x-wiki 3qjk7fecd44y7uz4sfgo2hi4taptj79 2474707 2474636 2026-10-28T14:50:48Z ASanford-WMF 52089 /* 4. Add the public keys to puppet repo for production access */ Add form link to step 4 2474707 wikitext text/x-wiki 7klijpkqxg5hue6v1lawy7l9b9gsg4s 2474708 2474707 2026-10-28T14:51:43Z ASanford-WMF 52089 /* 3. Generate new SSH key pair(s) */ 2474708 wikitext text/x-wiki nxr7c0lzvs7gjvrqaevwutlcgto01ps 2474774 2474708 2026-10-28T16:13:04Z Matma Rex 562 /* 3. Generate new SSH key pair(s) */ Seems easier to just give one command for the more common case 2474774 wikitext text/x-wiki kks1tbbkxmtd8sg1s38h3k9gsnebe6i 2474793 2474774 2026-10-28T16:27:32Z Matma Rex 562 /* 3. Generate new SSH key pair(s) */ Update Windows instructions 2474793 wikitext text/x-wiki kc1oatbhbx5dewndl7dzxhqpwztc2lw 2474794 2474793 2026-10-28T16:29:36Z Matma Rex 562 /* On Windows */ Clarify 2474794 wikitext text/x-wiki oythz3zrvghpo9v13ivaiqptr8opato 498 458787 2474870 2474451 2026-10-28T18:06:55Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741508078 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60) 2474870 wikitext text/x-wiki 7xl4l86hcm6bzw1helwvs3b95gb45oa 2474895 2474870 2026-10-28T18:50:19Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217679 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe) 2474895 wikitext text/x-wiki 3nw8xopf2sad0yjpfbxa8titrq6v1c5 498 459243 2474640 2474456 2026-10-28T13:34:54Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40697046893 (https://github.com/cluebotng/component-configs/commits/65c2635e0eca77c926d6ecfdd3bc4243290cb2de) 2474640 wikitext text/x-wiki 3xg5b1se0iqbbdx3i1kau7q4j2j31l9 2474644 2474640 2026-10-28T13:45:07Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40698633627 (https://github.com/cluebotng/component-configs/commits/29d35958a8230b19c67327fae4cc1a2c55f8f4f0) 2474644 wikitext text/x-wiki tpqwrlgzrly1negjj1zcf2h15eb4bre 2474908 2474644 2026-10-28T19:18:16Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40752366984 (https://github.com/cluebotng/component-configs/commits/b792020c142bd09c08d70ae0f40684c9c3f81079) 2474908 wikitext text/x-wiki h4zxet1b6bitdv826q8rm4ngg8pg9sh 2474924 2474908 2026-10-28T19:40:09Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40755839963 (https://github.com/cluebotng/component-configs/commits/43e1fcf7f4599d40cc9b9bd87abfeba5baede79b) 2474924 wikitext text/x-wiki dpbcgukbl0vns15nqjs5vl12liomeq4 0 459342 2474916 2461741 2026-10-28T19:26:57Z BKing (WMF) 29017 /* Platform owner */ 2474916 wikitext text/x-wiki ie8lf62cw2uvkqt5q04fv1ukhyf1gru 2474917 2474916 2026-10-28T19:27:23Z BKing (WMF) 29017 /* Platform owner */ 2474917 wikitext text/x-wiki 5i85opesrgat3npo7s70fk7h8q7xlz6 0 459699 2474904 2474423 2026-10-28T19:05:30Z Stashbot 7414 ejegg: SmashPig upgraded from 7b7586f2 to d446404f 2474904 wikitext text/x-wiki mxg965xs8dx2y9b1j61fnysnc9mhvzj 2475007 2474904 2026-10-28T21:36:24Z Stashbot 7414 ejegg: payments-wiki upgraded from ce87dea1 to 9bfda667 2475007 wikitext text/x-wiki 5n6lxmg30os46g684tebmyl404c4nnk 2475008 2475007 2026-10-28T21:38:24Z Stashbot 7414 ejegg: donorwiki upgraded from c32d227e to 9bfda667 2475008 wikitext text/x-wiki 1as3awssu57mukji03e4vg0glgqqsql 2475038 2475008 2026-10-29T00:01:37Z Stashbot 7414 eileen: * civicrm upgraded from 7f775db4 to 2302bf06 2475038 wikitext text/x-wiki ml5v3qzqb05xj6js8o6qn5ngrftzchy 2475040 2475038 2026-10-29T01:35:57Z Stashbot 7414 eileen: civicrm upgraded from 2302bf06 to d58bdb5c 2475040 wikitext text/x-wiki ewcd61cwhcrsh2hrhnv2xmke4s8milc 2475048 2475040 2026-10-29T05:46:43Z Stashbot 7414 eileen: civicrm upgraded from d58bdb5c to d1dc8124 2475048 wikitext text/x-wiki ffxwq7qosml3eqazs3ve67gz5bisahu 0 459743 2474963 2473252 2026-10-28T20:31:43Z LErickson-WMF 51715 /* Components */ 2474963 wikitext text/x-wiki eyc8pff2iw604z3e3z5pxb4nqmrxtqo 2474965 2474963 2026-10-28T20:34:12Z LErickson-WMF 51715 /* Monitoring, SLO */ 2474965 wikitext text/x-wiki ivfr06adt6ks7t7ok9973vx2pamh22e 2474966 2474965 2026-10-28T20:38:59Z LErickson-WMF 51715 2474966 wikitext text/x-wiki m6zualyir1k9tz2t5zxw5kccl10asm9 2474968 2474966 2026-10-28T20:41:24Z LErickson-WMF 51715 2474968 wikitext text/x-wiki r1wlboj0rt8ov85mozk934qswjwyqo7 2475057 2474968 2026-10-29T06:27:14Z STrug-WMF 51028 2475057 wikitext text/x-wiki e2w4zcncgn9n0g3wqp786vlho9qd3vu 0 459745 2474954 2404365 2026-10-28T20:15:37Z LErickson-WMF 51715 /* Monitoring */ 2474954 wikitext text/x-wiki g4d47s68hoy472qmk3i8yav6fwowedh 498 459970 2474959 2473494 2026-10-28T20:21:13Z Stashbot 7414 wmbot~difronzo@tools-bastion-15: deployed v0.12.0 (by DiFronzo on GitHub) 2474959 wikitext text/x-wiki 1n0zt2p6y76ot4dqxa9jvnon7urshp0 2475034 2474959 2026-10-28T23:37:52Z Stashbot 7414 wmbot~difronzo@tools-bastion-15: deployed v0.12.1 (by DiFronzo on GitHub) 2475034 wikitext text/x-wiki 93benfaf7gs6u7mam1ouh5irz1rsqwv 0 460627 2474953 2474308 2026-10-28T20:15:27Z AKhatun (WMF) 50461 Add dashboard links 2474953 wikitext text/x-wiki bymvwmlrib7k6gkd5glhjlznsz606ea 498 460727 2474956 2473493 2026-10-28T20:19:32Z Stashbot 7414 wmbot~difronzo@tools-bastion-15: deployed face-service-v0.12.0 (by DiFronzo on GitHub) 2474956 wikitext text/x-wiki p3dxqdspi24id4s456tkpzryy8uwt7y 2475033 2474956 2026-10-28T23:36:18Z Stashbot 7414 wmbot~difronzo@tools-bastion-15: deployed face-service-v0.12.1 (by DiFronzo on GitHub) 2475033 wikitext text/x-wiki iw36745x5pdz2whxc7xc3kxhsw7jp9x 0 460748 2474595 2466441 2026-10-28T12:43:48Z DPogorzelski-WMF 48508 2474595 wikitext text/x-wiki 7ea8uskc85e62bealcj1a6un870pt59 2474641 2474595 2026-10-28T13:35:29Z DPogorzelski-WMF 48508 2474641 wikitext text/x-wiki 3bb4r71wgfx6z53cbs44tyt486c03b0 2474642 2474641 2026-10-28T13:40:31Z DPogorzelski-WMF 48508 2474642 wikitext text/x-wiki 7tkbjxa2q6md9jwbgi0xt4kuy59cke6 2474660 2474642 2026-10-28T14:11:29Z DPogorzelski-WMF 48508 2474660 wikitext text/x-wiki lr1p0o21mpjrz2qct6i8avtecjg4s2x 0 460773 2474594 2465710 2026-10-28T12:43:14Z DPogorzelski-WMF 48508 2474594 wikitext text/x-wiki nygtqio1c2six6p95l0xpqv48nnsi1w 2474643 2474594 2026-10-28T13:42:07Z DPogorzelski-WMF 48508 2474643 wikitext text/x-wiki cuzefdjrzpumbchlqcip1jd3y6cgjha 0 460830 2475196 2470853 2026-10-29T11:35:38Z Taavi-WMF 41365 Taavi-WMF moved page [[News/2026 Rotating Credentials]] to [[News/2026 Toolforge credential rotation]]: follow precedent from other News pages 2470853 wikitext text/x-wiki t4p38c6nc6bbqy1072bpeuj6ms0pye7 0 460831 2474720 2471110 2026-10-28T15:06:39Z DPogorzelski-WMF 48508 2474720 wikitext text/x-wiki k3bw5c6gyp0qe8ci41dph5b9justlum 2474760 2474720 2026-10-28T15:31:19Z JMeybohm (WMF) 16709 2474760 wikitext text/x-wiki pbbf49zixh7fqqikmxqd4m2kzo9vshc 0 460840 2474841 2473896 2026-10-28T17:22:21Z BOplaid 59120 /* Updates and guidance */ 2474841 wikitext text/x-wiki 8ydg0z8m1eipsznnnydqbu5tpi0cieb 2475192 2474841 2026-10-29T11:28:45Z Taavi-WMF 41365 link to toolforge tool list 2475192 wikitext text/x-wiki ro8fcild6ceq04s25hurrzy7ghjtq0o 1 460842 2474837 2474176 2026-10-28T17:17:23Z DamianZaremba 54 /* php 7.4 still set as default */ Reply 2474837 wikitext text/x-wiki 7j3c2nlpik8ow87x6ja0z8r9cg9b6to 2474838 2474837 2026-10-28T17:17:56Z DamianZaremba 54 Fix editor applying nowiki 2474838 wikitext text/x-wiki bo2nkzuxavi4zrdbmrwo6tvd8l56pkb 2474839 2474838 2026-10-28T17:19:14Z DamianZaremba 54 /* Other WMCS (no PHP) */ Reply 2474839 wikitext text/x-wiki s6tt9mt9g37z5qihx3dtdfyxjrd0dmp 2474861 2474839 2026-10-28T17:47:49Z Ahecht 5894 /* php 7.4 still set as default */ Reply 2474861 wikitext text/x-wiki tl9rzicocx0j1boymk8vo7dgzse54b1 2474864 2474861 2026-10-28T17:56:17Z Taavi-WMF 41365 /* php 7.4 still set as default */ Reply 2474864 wikitext text/x-wiki 943l5w9p6uucmc6auwmxmmhhu4mg8j1 2475039 2474864 2026-10-29T00:17:40Z BLiviero-WMF 53603 /* Exceptions */ added a link to toolforge tools that are offline as of Oct-27 2475039 wikitext text/x-wiki kp9u6xamr4f7v2onj7l0m05ziefy4g2 2475195 2475039 2026-10-29T11:34:42Z Ziv 52835 /* Commons Delinker log */ new section 2475195 wikitext text/x-wiki oiaido0dhig4hjce89pun9gkyhh4tbb 2475198 2475195 2026-10-29T11:37:46Z Ziv 52835 /* Commons Delinker log */ Reply 2475198 wikitext text/x-wiki r5etlsrxxz984mb5k2bvj982c1dnsff 0 460881 2474674 2474429 2026-10-28T14:24:44Z RCoccioli (WMF) 7416 Opened wlm-it-visual 2474674 wikitext text/x-wiki 2v88iw461163gemsf8reruxiw3bpii2 2474688 2474674 2026-10-28T14:32:30Z Taavi-WMF 41365 toolsbeta, paws 2474688 wikitext text/x-wiki rjabo5zo9pp3wclh7s7kb8eajqb69p5 2475096 2474688 2026-10-29T08:34:36Z Taavi-WMF 41365 wikiapiary back 2475096 wikitext text/x-wiki ia6qb11d766y1orlga44bo75q11twe4 2475099 2475096 2026-10-29T08:35:54Z Taavi-WMF 41365 pixel 2475099 wikitext text/x-wiki sgkgqbiod9hhss39x0psxw0x6t4p7ix 2475100 2475099 2026-10-29T08:38:01Z Taavi-WMF 41365 wikidata-dev 2475100 wikitext text/x-wiki pmgve1nribl8wmdr4t9wamrkipps4j3 2475177 2475100 2026-10-29T11:08:05Z Taavi-WMF 41365 mobileappsperformance 2475177 wikitext text/x-wiki gadkly36914emt1obaa5bebk3lurbfl 2475178 2475177 2026-10-29T11:09:24Z Taavi-WMF 41365 civicrm-prototype already done 2475178 wikitext text/x-wiki pop9sngso08mopja7tzxnfef1teeqqz 2475179 2475178 2026-10-29T11:10:37Z Taavi-WMF 41365 rm duplicate mediawiki-quickstart 2475179 wikitext text/x-wiki n4hzbylmr85dym6ud7uk8s0hrz0daxj 2475180 2475179 2026-10-29T11:12:37Z Taavi-WMF 41365 wikicommunityhealth ok 2475180 wikitext text/x-wiki gfo1q2170c15fgef8frjb6d3sh1a8j9 2475186 2475180 2026-10-29T11:21:51Z Taavi-WMF 41365 striker 2475186 wikitext text/x-wiki 1s4ctq1i9wji5r45rg1vv9c1gq0bzjn 0 460894 2474905 2474305 2026-10-28T19:05:38Z BLiviero-WMF 53603 injected initial paragraph with call-to-action 2474905 wikitext text/x-wiki 4ra4f5xjvbmbl24nha4ybrt5bom6gpl 0 460899 2474845 2026-10-28T17:23:43Z STrug-WMF 51028 Created page with "=== Source Code Version Control === ==== Where does code live? ==== In general all source code for the WDQSv2 project is kept in [[GitLab|Gitlab]], more specifically in the [[gitlab:repos/wikidata-platform|wikidata-platform Gitlab group]]. All components that make up the main WDQSv2 deployment are maintained in the [[gitlab:repos/wikidata-platform/wdqs/|Gitlab wdqs sub-group]]. One major exception of this rule is the helm/helmfile description of the WDQSv2 DSE Kuber..." 2474845 wikitext text/x-wiki tnd0x95meyati6mysbnoqpmx2zjesd2 6 460900 2474855 2026-10-28T17:35:43Z APizzata-WMF 47941 {{File information |Description= |Source= |Date= |Author= |other_versions= }} 2474855 wikitext text/x-wiki s03jfytztn8v9r14jjn87mb6rqrtgs4 0 460901 2475041 2026-10-29T01:58:06Z LErickson-WMF 51715 Created page with "=WDQS bulk data reload= WDQS serves query results from an index that is built from a Wikidata snapshot. Streaming updates from Kafka keep the WDQS index current with the constant updates that Wikidata receives, but occasionally events do get dropped, so we supplement the streaming updates by periodically snapshotting Wikidata and building a new index from scratch. An Airflow DAG does the work of snapshotting Wikidata, building the index, and pushing it to a storage loc..." 2475041 wikitext text/x-wiki ourop8k2r2ub33u48x68gcb55jlf12g 0 460902 2475197 2026-10-29T11:35:38Z Taavi-WMF 41365 Taavi-WMF moved page [[News/2026 Rotating Credentials]] to [[News/2026 Toolforge credential rotation]]: follow precedent from other News pages 2475197 wikitext text/x-wiki 4s08kn2k5xehvb3mo1nhzz4hlf4n83g