Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.16 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2450033 2449999 2026-08-20T12:05:02Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2450033 wikitext text/x-wiki kbpx9ka12sk4jzuqvkiwbazxfuv5pj4 2450044 2450033 2026-08-20T12:18:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2450044 wikitext text/x-wiki 67oc5sunho54hy8biztnqnh6foe87kg 2450114 2450044 2026-08-20T13:29:37Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2450114 wikitext text/x-wiki 7evjgayl7m94totte4n4jxlglkf01xg 2450117 2450114 2026-08-20T13:34:41Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2450117 wikitext text/x-wiki cvrsx3wr4154alg1xnat0seskee0iaw 2450154 2450117 2026-08-20T14:21:33Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli 2450154 wikitext text/x-wiki 5wcxo1zqdsn3qhuynu6j5pd8ayv8grm 2450157 2450154 2026-08-20T14:24:47Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli 2450157 wikitext text/x-wiki ifqrunqteblh34syhqo83ead2931xn0 2450195 2450157 2026-08-20T15:33:32Z Stashbot 7414 dcaro: restart tools-puppetserver-01, unresponsive 2450195 wikitext text/x-wiki 8tl82votmwprp90m5bhv17shpziw839 2450199 2450195 2026-08-20T15:41:23Z Stashbot 7414 dcaro: tools-puppetserver-01 was out of memory because of java and killed sssd backend 2450199 wikitext text/x-wiki cp3wtffyg40c4y6g7x58yy0jobpqza0 2450204 2450199 2026-08-20T15:56:25Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2450204 wikitext text/x-wiki 45q4xgaadtk770os7ico8cuwq2txjio 2450219 2450204 2026-08-20T16:11:22Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2450219 wikitext text/x-wiki 6usp4k85nv8yztq97s6izzlshjkusp7 0 3195 2450312 2358661 2026-08-20T22:13:09Z BryanDavis 1604 /* See Also */ Link to new [[Developer account]] page 2450312 wikitext text/x-wiki 3n87zqxxd9e9nmemsbpjcrx62ya8ny7 2450313 2450312 2026-08-20T22:14:09Z BryanDavis 1604 added [[Category:LDAP]] using [[w:Wikipedia:HotCat|HotCat]] 2450313 wikitext text/x-wiki pp368jgskfwajzjd82ifs9sizxwdo63 0 4108 2450256 2450004 2026-08-20T17:52:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1324752]] to Thursday, August 20 UTC late backport window 2450256 wikitext text/x-wiki tdlrhpp7kx1vr12c6u6rzysgwoq3unn 2450266 2450256 2026-08-20T18:19:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1325920]] to Thursday, August 20 UTC late backport window 2450266 wikitext text/x-wiki avpts14lflyrqwozwdmvz7lfsxx5ze9 2450330 2450266 2026-08-20T23:51:31Z BryanDavis 1604 /* {{Deployment_day|date=2026-08-24}} */ schedule a shellbox deployment during MediaWiki infrastructure (UTC late) window 2450330 wikitext text/x-wiki r2uaz8m1eyx50qgtpl3j4nlt65gspfo 2450380 2450330 2026-08-21T08:04:15Z ScheduleDeploymentBot 37566 Add [[gerrit:1326848]] to Monday, August 24 UTC afternoon backport window 2450380 wikitext text/x-wiki t48wjv0t1sr0njmdw49zs1bq16yjq5j 498 6726 2450109 2450027 2026-08-20T13:24:42Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2450109 wikitext text/x-wiki kzi7de14ytn01hu3p16ujjm2cs5qwkd 2450112 2450109 2026-08-20T13:28:53Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2450112 wikitext text/x-wiki blglgt6ia029cr50zuo3870qq8224e7 2450149 2450112 2026-08-20T14:13:53Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli 2450149 wikitext text/x-wiki 3r5n49hmyygt8ozjjnhuyo2qywo1r2v 2450152 2450149 2026-08-20T14:18:03Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli 2450152 wikitext text/x-wiki m9btkg5hm1baypk77e3vaax459wrxaq 2450200 2450152 2026-08-20T15:43:18Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2450200 wikitext text/x-wiki n1ua5fyeqj6x38v3p9jf0uejdxl3m9p 2450203 2450200 2026-08-20T15:56:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2450203 wikitext text/x-wiki emg1rvbla9uyken7ewv0zrh73nicszx 0 7919 2450028 2450026 2026-08-20T12:00:22Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host stat1010.eqiad.wmnet with OS bookworm 2450028 wikitext text/x-wiki cb0yz07om8anhmavne5edjxhum2hzpo 2450029 2450028 2026-08-20T12:01:26Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.ganeti.makevm for new host db1902.eqiad.wmnet 2450029 wikitext text/x-wiki 2qjsmi6it75tf4wbr8mmux4htfwcmwd 2450030 2450029 2026-08-20T12:01:29Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2450030 wikitext text/x-wiki g82vntgxb0n317f91e1qcudvqoinfbt 2450031 2450030 2026-08-20T12:04:10Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host stat1010.eqiad.wmnet with OS bookworm 2450031 wikitext text/x-wiki 2g5gy5d7kti59bf4hzjwffoj52ze9or 2450032 2450031 2026-08-20T12:04:20Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host stat1009.eqiad.wmnet with OS bookworm 2450032 wikitext text/x-wiki 0rpn7hf7ruw4hys2gjo0st02ipoirw4 2450034 2450032 2026-08-20T12:08:16Z Stashbot 7414 tgr_: T413390 running CentralAuth:FixRenamedUserGlobalEditCount --wiki=metawiki --since=20250901000000 --until=20260301000000 --fix 2450034 wikitext text/x-wiki 5eip6194b4by7aqusbmvb0ne9rj27fq 2450035 2450034 2026-08-20T12:08:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1902.eqiad.wmnet - fceratto@cumin1003" 2450035 wikitext text/x-wiki 5jmq9iaj22iu2f5l89it7qt8slkoc42 2450036 2450035 2026-08-20T12:08:58Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1902.eqiad.wmnet - fceratto@cumin1003" 2450036 wikitext text/x-wiki 9bgjmxh2pytaz4u0e6cle6dyziqel7r 2450037 2450036 2026-08-20T12:09:02Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2450037 wikitext text/x-wiki 2fw9mdy1xac0k0hvstuetc32hnvfgwc 2450038 2450037 2026-08-20T12:09:05Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1902.eqiad.wmnet on all recursors 2450038 wikitext text/x-wiki oi7jrx7frqpupe6qckrcbdmiq3slutt 2450039 2450038 2026-08-20T12:09:07Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1902.eqiad.wmnet on all recursors 2450039 wikitext text/x-wiki 2ko44i2qrezsdn12dwrb96444fl5gsj 2450040 2450039 2026-08-20T12:09:34Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1902.eqiad.wmnet - fceratto@cumin1003" 2450040 wikitext text/x-wiki 3gspb3bpyc1po51bsxpo981xgpjvg2x 2450041 2450040 2026-08-20T12:09:38Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1902.eqiad.wmnet - fceratto@cumin1003" 2450041 wikitext text/x-wiki cunzuclvifocwrcr5bsj4611dvbby2o 2450042 2450041 2026-08-20T12:12:11Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1902.eqiad.wmnet with OS trixie 2450042 wikitext text/x-wiki 7nblm44vberwhq51stoa4yulz83wsv7 2450043 2450042 2026-08-20T12:14:06Z Stashbot 7414 elukey: move the Docker Registry's /v2/dev/.* prefix to its dedicated S3 backend - T432829 2450043 wikitext text/x-wiki 70zjc98f2bxvhn3w8cm55b9p03tjdxn 2450045 2450043 2026-08-20T12:21:09Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on stat1010.eqiad.wmnet with reason: host reimage 2450045 wikitext text/x-wiki fkqez1b4s9ztju867q3nasto21l7ltf 2450046 2450045 2026-08-20T12:22:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1902.eqiad.wmnet with reason: host reimage 2450046 wikitext text/x-wiki g2pvhdnw8x338sb22m5hobgww7b9i3k 2450047 2450046 2026-08-20T12:24:05Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat1010.eqiad.wmnet with reason: host reimage 2450047 wikitext text/x-wiki le26vabxrlhhyprysno53ryalpj3s49 2450048 2450047 2026-08-20T12:25:11Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2450048 wikitext text/x-wiki 1ohvu88i8vacz9cegxyv1hxy465yw0t 2450049 2450048 2026-08-20T12:25:36Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on stat1009.eqiad.wmnet with reason: host reimage 2450049 wikitext text/x-wiki 6gj0990y5zx3quuekq1liwatclij8hl 2450050 2450049 2026-08-20T12:28:25Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1902.eqiad.wmnet with reason: host reimage 2450050 wikitext text/x-wiki lfdydhtorjvu2mg1hce7i3651vtg4n1 2450051 2450050 2026-08-20T12:28:54Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad cct - cmooney@cumin1003" 2450051 wikitext text/x-wiki n4g0nk86ovzmwwuanq6xxz7k5h0i8dh 2450052 2450051 2026-08-20T12:31:37Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad cct - cmooney@cumin1003" 2450052 wikitext text/x-wiki n0zjsi45r38zfkiveqr4o5b67e5zaer 2450053 2450052 2026-08-20T12:31:40Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2450053 wikitext text/x-wiki 3usmpb1lumwdfdyws557ms4by3dzeaz 2450054 2450053 2026-08-20T12:31:49Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2450054 wikitext text/x-wiki q0bs75fl7o085bbrb2xxp4o61u5weev 2450055 2450054 2026-08-20T12:32:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat1009.eqiad.wmnet with reason: host reimage 2450055 wikitext text/x-wiki fk1uaps8zzb1qlf1nptcwp6oi40c2kz 2450056 2450055 2026-08-20T12:33:12Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1001.eqiad.wmnet 2450056 wikitext text/x-wiki 9tldtbxvow994n4necxkiz8q1a7ktev 2450057 2450056 2026-08-20T12:34:11Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2450057 wikitext text/x-wiki dxn0ix0ojptulkfpf97t0956zkvgilm 2450058 2450057 2026-08-20T12:35:33Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1001.eqiad.wmnet 2450058 wikitext text/x-wiki dta547mvnsrpgy7lznmc9lcc3o0n28l 2450059 2450058 2026-08-20T12:35:39Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1002.eqiad.wmnet 2450059 wikitext text/x-wiki fx1qb7f8wr3081efdxpfr4we63wtvf7 2450060 2450059 2026-08-20T12:37:59Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1002.eqiad.wmnet 2450060 wikitext text/x-wiki 7xb20olcpdwfc3wxknfpneh4z52hj0u 2450061 2450060 2026-08-20T12:38:05Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1003.eqiad.wmnet 2450061 wikitext text/x-wiki 3of0yga0jil34v3xq06hh8bj80t4dhp 2450063 2450061 2026-08-20T12:40:26Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1003.eqiad.wmnet 2450063 wikitext text/x-wiki n726dsmo4jy39on94p9al950e97nqyi 2450064 2450063 2026-08-20T12:41:24Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2003.codfw.wmnet 2450064 wikitext text/x-wiki dam4h0upy86f9gfb3khf8hvujuk0sis 2450065 2450064 2026-08-20T12:42:30Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host db1902.eqiad.wmnet with OS trixie 2450065 wikitext text/x-wiki kbrad3pkgrjztvy19y9nocepfqaud5n 2450066 2450065 2026-08-20T12:42:32Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=1) for new host db1902.eqiad.wmnet 2450066 wikitext text/x-wiki c7cvifk3g2d5jhnm6m4fe00lfclwjvs 2450067 2450066 2026-08-20T12:43:48Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2003.codfw.wmnet 2450067 wikitext text/x-wiki sgeqzgadljeluk4se8imchg893j0ouo 2450068 2450067 2026-08-20T12:44:00Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2002.codfw.wmnet 2450068 wikitext text/x-wiki ersfyzpbjm764t0vw3uoqqp45h7mest 2450069 2450068 2026-08-20T12:46:24Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2002.codfw.wmnet 2450069 wikitext text/x-wiki hk8i18n86uqc4dcrobhuufzmdz2soop 2450070 2450069 2026-08-20T12:46:34Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2001.codfw.wmnet 2450070 wikitext text/x-wiki 748bf5g767hysr0f2fmt7k8wovn4xru 2450071 2450070 2026-08-20T12:48:58Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2001.codfw.wmnet 2450071 wikitext text/x-wiki g2l54i5ym3o6oofk8f7dbtrnp4wo7xy 2450072 2450071 2026-08-20T12:49:20Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2450072 wikitext text/x-wiki aehj1v1fs2zxianympphu0pe0e301rh 2450073 2450072 2026-08-20T12:50:16Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2003.codfw.wmnet 2450073 wikitext text/x-wiki aop7j5gpeextj2byv0qal063etnmip7 2450074 2450073 2026-08-20T12:54:03Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2003.codfw.wmnet 2450074 wikitext text/x-wiki 8ymubjanm3bitjssue9arw1rpwy6o2l 2450075 2450074 2026-08-20T12:54:13Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2002.codfw.wmnet 2450075 wikitext text/x-wiki nh8jmf155gp1a0wzsbv6ezcqfvj6o1h 2450076 2450075 2026-08-20T12:54:48Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad GTT vpls - cmooney@cumin1003" 2450076 wikitext text/x-wiki 13lvskbmau1c030gr7zlztuc8t62wmt 2450077 2450076 2026-08-20T12:55:49Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad GTT vpls - cmooney@cumin1003" 2450077 wikitext text/x-wiki 98jvri20pcrbsbx870zw3td2mys5pmh 2450078 2450077 2026-08-20T12:55:51Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2450078 wikitext text/x-wiki dj3s8oyuw3ev1a4hu3f4kkerlspfgqp 2450079 2450078 2026-08-20T12:56:38Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2002.codfw.wmnet 2450079 wikitext text/x-wiki 0scnmmcxs3iusk9fvjz3gpadopq2kg0 2450080 2450079 2026-08-20T12:57:09Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2001.codfw.wmnet 2450080 wikitext text/x-wiki f3sz1lq9ritz825ftzrp1d9flowdr83 2450082 2450080 2026-08-20T12:59:34Z Stashbot 7414 cmooney@dns3003: START - running authdns-update 2450082 wikitext text/x-wiki 430tndammuq3onwf08fmhni3nqdrg05 2450083 2450082 2026-08-20T12:59:37Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2001.codfw.wmnet 2450083 wikitext text/x-wiki h4uo17v1yj94mk8vyf1lagfnue54rip 2450084 2450083 2026-08-20T13:00:18Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2001.codfw.wmnet 2450084 wikitext text/x-wiki q4t2mlzejfp2d1uo75mkx7tyut0a522 2450085 2450084 2026-08-20T13:02:00Z Stashbot 7414 cmooney@dns3003: END - running authdns-update 2450085 wikitext text/x-wiki 5g5iq5j9r47v4l7hd4wbkh7rnelc7ff 2450086 2450085 2026-08-20T13:02:09Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2450086 wikitext text/x-wiki o1ap0w2ombs1w0uh11lhwmwalmih54i 2450088 2450086 2026-08-20T13:03:43Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2001.codfw.wmnet 2450088 wikitext text/x-wiki gjmzau3ia37xd2anx2cruvnjsbztsoy 2450089 2450088 2026-08-20T13:03:51Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2002.codfw.wmnet 2450089 wikitext text/x-wiki mpbkripq42gj2bqgi8f4amre4pazxcc 2450091 2450089 2026-08-20T13:04:23Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2450091 wikitext text/x-wiki 993eo1y4pkp1z2gfvuxxifv4914mdx4 2450092 2450091 2026-08-20T13:06:52Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1327511|UIC: Fix page:page instead of page:other in instrumentation]] 2450092 wikitext text/x-wiki ogk9qa0vwwc284jw073apunf210xmm1 2450093 2450092 2026-08-20T13:07:16Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2002.codfw.wmnet 2450093 wikitext text/x-wiki sbcpg8t633gj2c9e3gqoxn8thue8z7a 2450094 2450093 2026-08-20T13:08:41Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2002.codfw.wmnet 2450094 wikitext text/x-wiki fo33svsswsbud3ip2cbw4ufsuajlenz 2450095 2450094 2026-08-20T13:08:55Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1327511|UIC: Fix page:page instead of page:other in instrumentation]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450095 wikitext text/x-wiki 1p3crs1886xpauq4w6ydokadhvys9di 2450096 2450095 2026-08-20T13:09:34Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2450096 wikitext text/x-wiki 2louwiks6k7ca70vpd8z3ein1qe60aa 2450098 2450096 2026-08-20T13:12:06Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2002.codfw.wmnet 2450098 wikitext text/x-wiki hqngyy62ykr7inxl7gk1r8rhm806str 2450099 2450098 2026-08-20T13:13:22Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet 2450099 wikitext text/x-wiki fb9inirpk54utwa6vbcx6b8me4nt2op 2450100 2450099 2026-08-20T13:13:52Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327511|UIC: Fix page:page instead of page:other in instrumentation]] (duration: 07m 00s) 2450100 wikitext text/x-wiki 7uyc3hob2wpiuovb9yuc8wdx8vio2m4 2450101 2450100 2026-08-20T13:16:48Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2001.codfw.wmnet 2450101 wikitext text/x-wiki gcy5ged7jevdyeq0umkxica3zl60v3m 2450102 2450101 2026-08-20T13:17:09Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1001.eqiad.wmnet 2450102 wikitext text/x-wiki 2myohqpsl41wyb65fn727jh1he36hnv 2450103 2450102 2026-08-20T13:20:33Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1001.eqiad.wmnet 2450103 wikitext text/x-wiki fz5wnku5baluyi0ccfd50swml3y7jl1 2450104 2450103 2026-08-20T13:20:40Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1002.eqiad.wmnet 2450104 wikitext text/x-wiki 9curkp4u32hrpigw0wimpbptzjb1ibu 2450105 2450104 2026-08-20T13:21:30Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host stat1010.eqiad.wmnet with OS bookworm 2450105 wikitext text/x-wiki l6tu033r3xmdcjezs40tycm3gn8xd0a 2450106 2450105 2026-08-20T13:24:00Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1002.eqiad.wmnet 2450106 wikitext text/x-wiki lfsx3gsj4o7cn7rsckn1mu5nqwt3oo3 2450107 2450106 2026-08-20T13:24:26Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker 2450107 wikitext text/x-wiki 3d5p7ftw4apqmafnf51oi0due6x7wvp 2450108 2450107 2026-08-20T13:24:29Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet 2450108 wikitext text/x-wiki rb1700xbjkkd7t21b25us4jgrbthalu 2450110 2450108 2026-08-20T13:26:30Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host stat1009.eqiad.wmnet with OS bookworm 2450110 wikitext text/x-wiki ebwnlbmnp4knp4ud3etuap7o3pooubq 2450111 2450110 2026-08-20T13:28:34Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1025.eqiad.wmnet 2450111 wikitext text/x-wiki pia2fpxkefatypfrk89q3twqcqtho0i 2450113 2450111 2026-08-20T13:29:15Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2450113 wikitext text/x-wiki a49lffbe7w7p19mu20yqw0xaoan14s4 2450115 2450113 2026-08-20T13:31:16Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2450115 wikitext text/x-wiki sojeezokd7hrodo9c6hnwcigokf2vno 2450116 2450115 2026-08-20T13:34:36Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet 2450116 wikitext text/x-wiki s616ljg9dd5252s7tspe8j3amgrevt4 2450119 2450116 2026-08-20T13:38:45Z Stashbot 7414 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1025.eqiad.wmnet with reason: Removing s6 from clouddb1025 2450119 wikitext text/x-wiki msbi9vcejked7qpoeuagf8owoa9bngy 2450120 2450119 2026-08-20T13:39:01Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2450120 wikitext text/x-wiki 7h9oie8pwyz942pu5qw2udobgyftsz0 2450122 2450120 2026-08-20T13:41:02Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2450122 wikitext text/x-wiki hkn9bpwh27smropwojmo9zo3dv1tllj 2450123 2450122 2026-08-20T13:41:19Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet 2450123 wikitext text/x-wiki axfj3hhyof91ibzfq29r3x18qeukw61 2450124 2450123 2026-08-20T13:41:22Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet 2450124 wikitext text/x-wiki 6cennbfvhm0slg313c5d8vugkytngyj 2450125 2450124 2026-08-20T13:41:28Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet 2450125 wikitext text/x-wiki 6t7jd3ennntuplxihj7t9nlj6qlczbl 2450128 2450125 2026-08-20T13:49:49Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1025.eqiad.wmnet 2450128 wikitext text/x-wiki 9htw6bzs6ekmcs11k4r2phm0o23s8hr 2450129 2450128 2026-08-20T13:50:10Z Stashbot 7414 fnegri@cumin1003: conftool action : set/weight=100; selector: name=clouddb1025.eqiad.wmnet 2450129 wikitext text/x-wiki a84klx60rmkd92zkm6sd3qk913bdcmw 2450130 2450129 2026-08-20T13:51:32Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet 2450130 wikitext text/x-wiki cle23o51prcilryf15imzbmxasagpou 2450131 2450130 2026-08-20T13:53:24Z Stashbot 7414 moritzm: installing apr-util security updates 2450131 wikitext text/x-wiki 5lh298x4qt4qa257zpx1pbgpeyaiph4 2450132 2450131 2026-08-20T13:56:06Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2450132 wikitext text/x-wiki 9kfv8jjx99z2nf5oqpdz5bz5noez0vg 2450133 2450132 2026-08-20T13:56:20Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1025.eqiad.wmnet 2450133 wikitext text/x-wiki ocx1reyjd4g4pf6xhxas28egdhboglp 2450134 2450133 2026-08-20T13:56:23Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1025.eqiad.wmnet 2450134 wikitext text/x-wiki 87dx7lp5u7qclkd1zhi1nrpn92eg9jv 2450135 2450134 2026-08-20T13:56:47Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2450135 wikitext text/x-wiki 7m2wslskgyuofz4dkssjbd1zlsswk2n 2450138 2450135 2026-08-20T13:58:18Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet 2450138 wikitext text/x-wiki ialv08yyv77zv9uqv958bbol7xex4ut 2450139 2450138 2026-08-20T13:58:21Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet 2450139 wikitext text/x-wiki nudj5mj4jcxeh8y0n4igkdu9qmppv8m 2450140 2450139 2026-08-20T13:58:24Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet 2450140 wikitext text/x-wiki hjk653wfogm12dqtqdjh9c4s4ucgwsf 2450143 2450140 2026-08-20T14:06:11Z Stashbot 7414 moritzm: installing libheif security updates 2450143 wikitext text/x-wiki 9tb728ppijd5gq5mjj2bqehur957pn5 2450144 2450143 2026-08-20T14:08:29Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet 2450144 wikitext text/x-wiki 4q53qwv7o9a3r6od5pdcx5mj62aj1sf 2450145 2450144 2026-08-20T14:13:01Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet 2450145 wikitext text/x-wiki kmippw1fur0a8qez9a8yofhzdbtbqgj 2450146 2450145 2026-08-20T14:13:05Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet 2450146 wikitext text/x-wiki puyk7vffe1uavfz4xrkica96zl1rs75 2450147 2450146 2026-08-20T14:13:08Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker 2450147 wikitext text/x-wiki lg4ygm3h4im0qrg2wg3ytizesb9xvj3 2450148 2450147 2026-08-20T14:13:32Z Stashbot 7414 moritzm: installing util-linux security updates 2450148 wikitext text/x-wiki 7hrkn1h0l3hi1e0qd1jrvdmqvwsmfdq 2450150 2450148 2026-08-20T14:14:32Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2450150 wikitext text/x-wiki 2jxq7ekqfeys4wzfeyhvexrit5w26wl 2450151 2450150 2026-08-20T14:14:37Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2450151 wikitext text/x-wiki enuucc4uygf9ulaf8s0cz8aic55kbzq 2450153 2450151 2026-08-20T14:19:33Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries used on new transport backup eqiad codfw - cmooney@cumin1003" 2450153 wikitext text/x-wiki 5eln4is9vfrddu4izceuxh5n43oz99g 2450155 2450153 2026-08-20T14:22:50Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries used on new transport backup eqiad codfw - cmooney@cumin1003" 2450155 wikitext text/x-wiki de3tsb4u04fs45jp968jq4kfyplp107 2450156 2450155 2026-08-20T14:22:52Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2450156 wikitext text/x-wiki 8vl0c1ddbtgrxbzlty5db2i1vp1jxsl 2450159 2450156 2026-08-20T14:27:18Z Stashbot 7414 topranks: reconfigure eqiad<->codfw bgp settings 2450159 wikitext text/x-wiki 5quqdqdzb6rsdh5rkhjh5uxw4g2k2ds 2450160 2450159 2026-08-20T14:33:22Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=yes; selector: name=cp1100.* 2450160 wikitext text/x-wiki shvf9itdve54ny9a4xlcoulfiwy9i75 2450163 2450160 2026-08-20T14:49:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.ganeti.makevm for new host db1903.eqiad.wmnet 2450163 wikitext text/x-wiki 9yyfpsrulgbkq6pc1qn1k84q16r7yc1 2450164 2450163 2026-08-20T14:49:13Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2450164 wikitext text/x-wiki lrj6nciuz8pylokn7s4lasap6f8bbdo 2450165 2450164 2026-08-20T14:53:14Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1903.eqiad.wmnet - fceratto@cumin1003" 2450165 wikitext text/x-wiki g63au52quernimz8yhitllbq9xfdqn7 2450166 2450165 2026-08-20T14:53:18Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1903.eqiad.wmnet - fceratto@cumin1003" 2450166 wikitext text/x-wiki 8rz16mbbkb95jzdl2i59z3jgqirprbz 2450167 2450166 2026-08-20T14:53:20Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2450167 wikitext text/x-wiki irdowgvt1vn7e46c4yn6kaxyi0hjbnf 2450168 2450167 2026-08-20T14:53:23Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1903.eqiad.wmnet on all recursors 2450168 wikitext text/x-wiki apkaqa560wvuuxcy0pgd9ltd1s4xq5c 2450169 2450168 2026-08-20T14:53:25Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1903.eqiad.wmnet on all recursors 2450169 wikitext text/x-wiki c2lou2tcn6y5cuw3puhm87awk2fc8fl 2450170 2450169 2026-08-20T14:53:56Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1903.eqiad.wmnet - fceratto@cumin1003" 2450170 wikitext text/x-wiki 8vgkwbrt95xud4h7ee8i3ns0aovtz97 2450171 2450170 2026-08-20T14:54:00Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1903.eqiad.wmnet - fceratto@cumin1003" 2450171 wikitext text/x-wiki fxrp4l7e94phsh0f2g04lyrlolmu9mp 2450172 2450171 2026-08-20T14:54:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1903.eqiad.wmnet with OS trixie 2450172 wikitext text/x-wiki 8j3djm6i04749jl1q7d6v54pkd0mzo7 2450175 2450172 2026-08-20T15:07:40Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1903.eqiad.wmnet with reason: host reimage 2450175 wikitext text/x-wiki kcs577z92d8qlj96u9313yjr8j495jt 2450176 2450175 2026-08-20T15:14:29Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1903.eqiad.wmnet with reason: host reimage 2450176 wikitext text/x-wiki lwb3ptsbakjpwoamychfsg47o6eyqt2 2450177 2450176 2026-08-20T15:17:15Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2450177 wikitext text/x-wiki ju12vu9w7uyomvhf0ebh0hbhmq6v2d3 2450178 2450177 2026-08-20T15:19:32Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2450178 wikitext text/x-wiki bri8gz3n8txzlnx0mni58rhk4q3yhl3 2450180 2450178 2026-08-20T15:19:40Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2450180 wikitext text/x-wiki 6vbhzrmntdo1co8fjl4rsg25t7d920k 2450181 2450180 2026-08-20T15:21:50Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2450181 wikitext text/x-wiki pfw6jq00ju2b1xdj4pf1qw8py4bnb6h 2450182 2450181 2026-08-20T15:21:58Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2450182 wikitext text/x-wiki ew1x59kq6tafliegj15gmlwplqhx04t 2450183 2450182 2026-08-20T15:24:09Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2450183 wikitext text/x-wiki qcpgba44j60et3aiypuwemgiuv55ud5 2450184 2450183 2026-08-20T15:24:18Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2450184 wikitext text/x-wiki 21awq8rwtngol73utk11u8mupl6adqz 2450185 2450184 2026-08-20T15:26:33Z Stashbot 7414 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2450185 wikitext text/x-wiki tbv2eeciqbpkn6g13psty0xnfxtay7r 2450186 2450185 2026-08-20T15:26:42Z Stashbot 7414 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. 2450186 wikitext text/x-wiki o9s93ygxckiplc8hty5w7vil3uv5r8z 2450187 2450186 2026-08-20T15:28:27Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1903.eqiad.wmnet with OS trixie 2450187 wikitext text/x-wiki 2bbfemzpyy67t7vh8r3gy2xgj0wqbws 2450188 2450187 2026-08-20T15:28:29Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host db1903.eqiad.wmnet 2450188 wikitext text/x-wiki awj7sgjkhewn5ud1ukacczeoplus1ru 2450189 2450188 2026-08-20T15:28:45Z Stashbot 7414 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. 2450189 wikitext text/x-wiki kkgxce7e75ye37ukqzy0pf6yscjgxbv 2450190 2450189 2026-08-20T15:28:54Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2450190 wikitext text/x-wiki m9iq54vbewl917ne9t12ttqz706s6zx 2450191 2450190 2026-08-20T15:30:31Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2450191 wikitext text/x-wiki h0kspqk79ajnyf91uvx62ltmf35k4r2 2450192 2450191 2026-08-20T15:30:38Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2450192 wikitext text/x-wiki laizelt8etlntcj35gvq9y3gcipww6o 2450193 2450192 2026-08-20T15:32:11Z Stashbot 7414 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2450193 wikitext text/x-wiki eiamypvu6azw4085hkl52zc2481kas0 2450194 2450193 2026-08-20T15:32:18Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2450194 wikitext text/x-wiki o1krqaw72hg7h7wy0mtic98ronc6z29 2450196 2450194 2026-08-20T15:35:20Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2450196 wikitext text/x-wiki cwptklif8p1mq0g1efr4ig0ewean87y 2450197 2450196 2026-08-20T15:35:27Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2450197 wikitext text/x-wiki 2jxj8bkhriv1s0o5cg9etdnm9h9ipx0 2450198 2450197 2026-08-20T15:37:51Z Stashbot 7414 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2450198 wikitext text/x-wiki 5ezo5gp295qu23oqnh1yleai3d2gl24 2450202 2450198 2026-08-20T15:55:28Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2450202 wikitext text/x-wiki eiodsz8qudis2wzlqmiozhv2q6dymno 2450205 2450202 2026-08-20T15:56:29Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2450205 wikitext text/x-wiki gaqra5hfs124bgenp8mkrp675knhmhf 2450206 2450205 2026-08-20T15:59:22Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2450206 wikitext text/x-wiki fm9np0ua2gjuet9gchi8pnqv4t1eitq 2450207 2450206 2026-08-20T16:00:34Z Stashbot 7414 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2450207 wikitext text/x-wiki 5l70lrvqqiwlvlich8s2t52x46k6rbm 2450208 2450207 2026-08-20T16:05:08Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reimage for host stat1011.eqiad.wmnet with OS bookworm 2450208 wikitext text/x-wiki q5qwdhywijk9xmijzulzncuv8xa12os 2450209 2450208 2026-08-20T16:05:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host stat1011 2450209 wikitext text/x-wiki f74vwex3o8ydp5qc34x879wnxrbgeiw 2450210 2450209 2026-08-20T16:05:41Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.netbox 2450210 wikitext text/x-wiki 9gdaagxgtxtt2l073y15pya59a8x00w 2450211 2450210 2026-08-20T16:09:29Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host stat1011 - bking@cumin2003" 2450211 wikitext text/x-wiki 9mk9i1ax3kw6oz4gbsrcow9fzjsnp6i 2450212 2450211 2026-08-20T16:09:44Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host stat1011 - bking@cumin2003" 2450212 wikitext text/x-wiki rx9idmocwdu50t2gpposekcqudyqjwx 2450213 2450212 2026-08-20T16:09:46Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2450213 wikitext text/x-wiki 35jh7nx719tkeiv58zsvboeolk1gjs3 2450214 2450213 2026-08-20T16:09:49Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.dns.wipe-cache stat1011.eqiad.wmnet 14.36.64.10.in-addr.arpa 4.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2450214 wikitext text/x-wiki l7qleewu8bodgdjow6h2shmwvq1y1mm 2450215 2450214 2026-08-20T16:09:51Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) stat1011.eqiad.wmnet 14.36.64.10.in-addr.arpa 4.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2450215 wikitext text/x-wiki h1x6rymi2rmrw1scnzx4v40al3r2z7m 2450216 2450215 2026-08-20T16:09:54Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host stat1011 2450216 wikitext text/x-wiki o9uvn6afqnchg26q5noxvtme3bdlwhp 2450217 2450216 2026-08-20T16:10:21Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host stat1011 2450217 wikitext text/x-wiki j7mxx7paj18dxcp711fmkg40ny0se1m 2450218 2450217 2026-08-20T16:10:23Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host stat1011 2450218 wikitext text/x-wiki c4jg0vh93e40r2hu1lwdnjkrm71s6eb 2450220 2450218 2026-08-20T16:14:52Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=no; selector: name=dns6002.* [reason: depooling for trixie upgrade] 2450220 wikitext text/x-wiki sk5961gnbcl2yk7s83dr1k8qt5vl2g7 2450221 2450220 2026-08-20T16:15:47Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns6002.wikimedia.org with OS trixie 2450221 wikitext text/x-wiki 4j44j5w8zjltbh42wfpu0ynws6s1qoi 2450222 2450221 2026-08-20T16:34:09Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns6002.wikimedia.org with reason: host reimage 2450222 wikitext text/x-wiki otwod133cdk0qcluhtq4udps5p47p3o 2450223 2450222 2026-08-20T16:36:26Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db2901.codfw.wmnet 2450223 wikitext text/x-wiki hvj9uhvidlm9jh7d1lfraaeg86uodr1 2450224 2450223 2026-08-20T16:36:44Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on stat1011.eqiad.wmnet with reason: host reimage 2450224 wikitext text/x-wiki 6075cgzyo1z82ts2r41ckb865sdh0gx 2450225 2450224 2026-08-20T16:39:49Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns6002.wikimedia.org with reason: host reimage 2450225 wikitext text/x-wiki moemkaozju20lznipm29rw44naczz0l 2450226 2450225 2026-08-20T16:43:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat1011.eqiad.wmnet with reason: host reimage 2450226 wikitext text/x-wiki tgom0ipyi5mdfld5mo78c4lchbl3hku 2450227 2450226 2026-08-20T16:43:55Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.dns.netbox 2450227 wikitext text/x-wiki dgf7cgr32yrg34cx1c3f77x19p7vnlp 2450228 2450227 2026-08-20T16:51:46Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2901.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" 2450228 wikitext text/x-wiki enavoafo5t8r7e78jsgoby06t5eelqc 2450229 2450228 2026-08-20T16:51:57Z Stashbot 7414 swfrench-wmf: disable-puppet on A:cp for ATS Lua change - T427666 2450229 wikitext text/x-wiki s13fcszo68gjnka56goty86zij88gk8 2450233 2450229 2026-08-20T17:15:44Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1327590|Make sure fpsmax is an int value (T318419)]] 2450233 wikitext text/x-wiki loaiitt7fj4id3cm48wu8r43eqrsmsp 2450234 2450233 2026-08-20T17:16:05Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2450234 wikitext text/x-wiki 1ceiggg32m4dnv9wvnid13k48wz7m2i 2450235 2450234 2026-08-20T17:17:48Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1327590|Make sure fpsmax is an int value (T318419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450235 wikitext text/x-wiki jsx207m224ztfy330a5xacpvqnbldj5 2450237 2450235 2026-08-20T17:20:10Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2450237 wikitext text/x-wiki itf4hqv9n8v83l65ae4mihdymw4deeh 2450238 2450237 2026-08-20T17:24:22Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327590|Make sure fpsmax is an int value (T318419)]] (duration: 08m 37s) 2450238 wikitext text/x-wiki sx02mepwzmabx8eslx3pvismjqvflib 2450239 2450238 2026-08-20T17:28:01Z Stashbot 7414 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply 2450239 wikitext text/x-wiki mwwq52123c839u4z54wlikc0l2n0eyz 2450240 2450239 2026-08-20T17:29:09Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply 2450240 wikitext text/x-wiki stcv489m562d0d463f79n7xw2um99cv 2450241 2450240 2026-08-20T17:29:24Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply 2450241 wikitext text/x-wiki 0rpum8dvcuswpzpectdbsc3l40gfi8k 2450242 2450241 2026-08-20T17:29:26Z Stashbot 7414 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply 2450242 wikitext text/x-wiki tu2h4dob9qsfinlqxi65uq9sm49c613 2450243 2450242 2026-08-20T17:29:31Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply 2450243 wikitext text/x-wiki hthlz5c1z5atd0vxm4bf6lwn67t6yvz 2450244 2450243 2026-08-20T17:29:50Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply 2450244 wikitext text/x-wiki 1ib7471dc07dilfl6mnm00vwmtggqqg 2450245 2450244 2026-08-20T17:30:26Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply 2450245 wikitext text/x-wiki l360vd3ie8d9go280h5oq743ck800j4 2450246 2450245 2026-08-20T17:30:41Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply 2450246 wikitext text/x-wiki gz403xpzr4tyhge3udin7huyorn6jsj 2450247 2450246 2026-08-20T17:30:47Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply 2450247 wikitext text/x-wiki qu00bnoy5j4fgh433q4himodg5jqj1k 2450248 2450247 2026-08-20T17:30:49Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply 2450248 wikitext text/x-wiki kqved8rwbp006ka0ppuw07ll7e0byz8 2450251 2450248 2026-08-20T17:31:39Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6002.wikimedia.org with OS trixie 2450251 wikitext text/x-wiki 4gf4jyyrcv7wziaeubnpdmimqzepf1y 2450252 2450251 2026-08-20T17:34:28Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host stat1011.eqiad.wmnet with OS bookworm 2450252 wikitext text/x-wiki 4ivavpkx3f078sk9wwgztlluq1oaanb 2450254 2450252 2026-08-20T17:51:15Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2450254 wikitext text/x-wiki qszbpei6hqbn7iu6d7wjv7rpl50yagk 2450255 2450254 2026-08-20T17:51:25Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2450255 wikitext text/x-wiki kt32girc53rjejag60gh5clsx185j1x 2450257 2450255 2026-08-20T17:56:47Z Stashbot 7414 cdobbins@cumin1003: START - Cookbook sre.hosts.remove-downtime for dns6002.wikimedia.org 2450257 wikitext text/x-wiki 2qhriiwd0w4encp9h5yvp7bfbfv4jxc 2450258 2450257 2026-08-20T17:56:49Z Stashbot 7414 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns6002.wikimedia.org 2450258 wikitext text/x-wiki qnzb0dc2y3ddtdxagkyixme3kkmcjjq 2450259 2450258 2026-08-20T17:57:24Z Stashbot 7414 cdobbins@cumin1003: conftool action : set/pooled=yes; selector: name=dns6002.* [reason: depooling for trixie upgrade] 2450259 wikitext text/x-wiki 1ls8a058g5zn2q2ta0c50rxjca7wapf 2450260 2450259 2026-08-20T17:58:33Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2450260 wikitext text/x-wiki 8wg373n4248zd902hb596k3e6v1xukz 2450261 2450260 2026-08-20T17:59:06Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2450261 wikitext text/x-wiki op6zsjd1uyjygyzszlhpjutol0kkjhd 2450262 2450261 2026-08-20T18:01:15Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2450262 wikitext text/x-wiki 8kot37zjeanysj9dbge9w22vhtvaii7 2450263 2450262 2026-08-20T18:01:59Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2450263 wikitext text/x-wiki jl448eb4btn1pzxrg4wnebyrmebytat 2450264 2450263 2026-08-20T18:05:25Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2450264 wikitext text/x-wiki jrlp2bed5rwhn8e0siyk0se7opwkmz3 2450265 2450264 2026-08-20T18:07:31Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2450265 wikitext text/x-wiki o5ararjz0yngw5ps3807sq8dspdi5iw 2450268 2450265 2026-08-20T18:36:59Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2450268 wikitext text/x-wiki 0gfdio11srgk20ik4stw2y1s9ew9el8 2450269 2450268 2026-08-20T18:37:11Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2450269 wikitext text/x-wiki tl0d49miz17v0wuds3rwsz7pkfznjng 2450270 2450269 2026-08-20T18:37:46Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2450270 wikitext text/x-wiki 45p44rpd7fc5gxpxfm0qq7b6qfbdek0 2450271 2450270 2026-08-20T18:37:55Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2450271 wikitext text/x-wiki gqq7csxiqo1w3ldc7p8iizaqhrlf3ig 2450272 2450271 2026-08-20T18:43:34Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1327614|Avoid casting fpxmax to string (T318419)]] 2450272 wikitext text/x-wiki 3x7s92581yo9c7d5tsxxaoaf0cjednx 2450273 2450272 2026-08-20T18:45:40Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1327614|Avoid casting fpxmax to string (T318419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450273 wikitext text/x-wiki b8m0vnoyi4x0ernjhevlknlaf8q3sp4 2450274 2450273 2026-08-20T18:46:19Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2450274 wikitext text/x-wiki om5wnpup3fam06vj2wt1cnk7dxwprg8 2450275 2450274 2026-08-20T18:46:59Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2450275 wikitext text/x-wiki 5wgp9kuxb4bgozp91t0yuxa8vnspiyc 2450276 2450275 2026-08-20T18:50:27Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2450276 wikitext text/x-wiki eihnk2b5kr4ccye5tjog0i2dm24m06z 2450277 2450276 2026-08-20T18:51:02Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327614|Avoid casting fpxmax to string (T318419)]] (duration: 07m 28s) 2450277 wikitext text/x-wiki n1ve30zlle4polde6ar9b1n9au5zcoq 2450278 2450277 2026-08-20T19:05:36Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1327598|Make \Omicron non upright (like \Chi) (T434428)]], [[gerrit:1327596|Render overline of \bar with stretchy=false (T435456)]] 2450278 wikitext text/x-wiki 7zeuvveqijof09vlv14t5q9hnggdf3x 2450279 2450278 2026-08-20T19:07:38Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1327598|Make \Omicron non upright (like \Chi) (T434428)]], [[gerrit:1327596|Render overline of \bar with stretchy=false (T435456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450279 wikitext text/x-wiki d0ht8qqy0xnt2hvi1lu4kknhx27o0z0 2450280 2450279 2026-08-20T19:20:08Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2450280 wikitext text/x-wiki lgqhyzl9vaxy7016e97n8n0wmkq0qh2 2450281 2450280 2026-08-20T19:24:29Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327598|Make \Omicron non upright (like \Chi) (T434428)]], [[gerrit:1327596|Render overline of \bar with stretchy=false (T435456)]] (duration: 18m 54s) 2450281 wikitext text/x-wiki 1b2fd0tvv6e7jq6ed87m12zun0go1ms 2450283 2450281 2026-08-20T20:08:16Z Stashbot 7414 cdanis@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "bug fixes & UX fixes - cdanis@cumin1003" 2450283 wikitext text/x-wiki omv0nmslsaidyhk3pqrkc1iu4ja2q8k 2450284 2450283 2026-08-20T20:08:20Z Stashbot 7414 cdanis@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: bug fixes & UX fixes - cdanis@cumin1003 2450284 wikitext text/x-wiki 3q8y48vekr78ba87yvvvwoisx6q57d9 2450285 2450284 2026-08-20T20:09:09Z Stashbot 7414 cdanis@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: bug fixes & UX fixes - cdanis@cumin1003 2450285 wikitext text/x-wiki o65mop44jbewr3t554xmb1jm3wy5u7z 2450286 2450285 2026-08-20T20:09:12Z Stashbot 7414 cdanis@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "bug fixes & UX fixes - cdanis@cumin1003" 2450286 wikitext text/x-wiki 35h16ga9a4lh2pmc942zfp2u8d3wf2e 2450287 2450286 2026-08-20T20:10:35Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2450287 wikitext text/x-wiki dxeacquegjwb3n64ztguesyvd9qpsc4 2450288 2450287 2026-08-20T20:14:13Z Stashbot 7414 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2450288 wikitext text/x-wiki goz9q9m2rg4kljenbj7q96es8r9od6e 2450289 2450288 2026-08-20T20:23:33Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1324752|InitialiseSettings: Enable 2FA banners on remaining private wikis (T428103)]], [[gerrit:1325920|Remove sending email to legal team about rejected requests (T374053)]] 2450289 wikitext text/x-wiki mjat4xd5lqnjrl4zgiox9xrpbqhqpok 2450290 2450289 2026-08-20T20:24:19Z Stashbot 7414 reedy@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.15,1.47.0-wmf.16,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/med 2450290 wikitext text/x-wiki 84qbhzo7kdtyuboqbimfkvsk8rminbc 2450291 2450290 2026-08-20T20:51:52Z Stashbot 7414 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1324752|InitialiseSettings: Enable 2FA banners on remaining private wikis (T428103)]], [[gerrit:1325920|Remove sending email to legal team about rejected requests (T374053)]] 2450291 wikitext text/x-wiki 324kj8yl24v6ssluzru52f2s2rl5tct 2450292 2450291 2026-08-20T20:54:03Z Stashbot 7414 reedy@deploy1003: neriah, reedy: Backport for [[gerrit:1324752|InitialiseSettings: Enable 2FA banners on remaining private wikis (T428103)]], [[gerrit:1325920|Remove sending email to legal team about rejected requests (T374053)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450292 wikitext text/x-wiki 6vi0mgptmmtpbxr8yze31lpw23z14qb 2450293 2450292 2026-08-20T20:54:44Z Stashbot 7414 reedy@deploy1003: neriah, reedy: Continuing with deployment 2450293 wikitext text/x-wiki 1ipkhw21qbkcgz4fdpaorvdeq5bqqch 2450294 2450293 2026-08-20T20:59:11Z Stashbot 7414 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324752|InitialiseSettings: Enable 2FA banners on remaining private wikis (T428103)]], [[gerrit:1325920|Remove sending email to legal team about rejected requests (T374053)]] (duration: 07m 18s) 2450294 wikitext text/x-wiki 2pnyzsorktwuqqlyn0gdnzimntmwusa 2450295 2450294 2026-08-20T21:17:40Z Stashbot 7414 maryum: Deployed security fix for T433020 2450295 wikitext text/x-wiki kwlfgx8ng73d1jqcfaavli1n175c4ya 2450296 2450295 2026-08-20T21:27:23Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1327219|RunSingleJob: Define MW_ENTRY_POINT for flamegraph sample attribution (T435422)]] 2450296 wikitext text/x-wiki 79kh5d3yyu26g174s9act0ieaomk7x6 2450297 2450296 2026-08-20T21:29:28Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1327219|RunSingleJob: Define MW_ENTRY_POINT for flamegraph sample attribution (T435422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450297 wikitext text/x-wiki 8d4mvoezagn4gynjbo50stzff9r0oep 2450298 2450297 2026-08-20T21:31:24Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1054 2450298 wikitext text/x-wiki 9y66zx7qunlfdmpa3bv13quqpnzs2t3 2450299 2450298 2026-08-20T21:31:36Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1054 2450299 wikitext text/x-wiki a6au2fes5reemw7l37sfwso91p2xjwd 2450300 2450299 2026-08-20T21:31:40Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2450300 wikitext text/x-wiki 29s8c02y2p0aje68b591az87lm18gpa 2450301 2450300 2026-08-20T21:31:55Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2450301 wikitext text/x-wiki gflyd1xpsjx67poy2o87jhhsf3kqydq 2450302 2450301 2026-08-20T21:35:53Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327219|RunSingleJob: Define MW_ENTRY_POINT for flamegraph sample attribution (T435422)]] (duration: 08m 30s) 2450302 wikitext text/x-wiki nsrssy3crpelzn71vfpgp4ja4gv8gma 2450303 2450302 2026-08-20T21:36:16Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2450303 wikitext text/x-wiki p13imk68b4644fpy3ha80de8j85iv8y 2450304 2450303 2026-08-20T21:36:37Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1054.eqiad.wmnet with OS trixie 2450304 wikitext text/x-wiki l1tar3xxyy0tpubd02uinlq459fek8n 2450305 2450304 2026-08-20T21:51:44Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1054.eqiad.wmnet with reason: host reimage 2450305 wikitext text/x-wiki 78wtxj00edb2dot0db98thhszqmhl3u 2450307 2450305 2026-08-20T21:58:24Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1054.eqiad.wmnet with reason: host reimage 2450307 wikitext text/x-wiki 9yvxc94yig7uu61sy6svf04xw7dix8p 2450315 2450307 2026-08-20T22:14:36Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2450315 wikitext text/x-wiki 5e3hcrt3uujl5hjpsw494mfh23jc0ec 2450318 2450315 2026-08-20T22:14:59Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2450318 wikitext text/x-wiki 70hyqq7g1nakuhvoh5tfon8eb10swvl 2450319 2450318 2026-08-20T22:15:02Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1054.eqiad.wmnet with OS trixie 2450319 wikitext text/x-wiki bmqg9o6klw2ike1hlxbyuq8anoueeda 2450325 2450319 2026-08-20T23:31:02Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1327652|RunSingleJob: Add ProfilingContext::init() (T435422)]] 2450325 wikitext text/x-wiki hnobigbptf5il3lkw0ae2hhiywteeq2 2450326 2450325 2026-08-20T23:33:08Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1327652|RunSingleJob: Add ProfilingContext::init() (T435422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450326 wikitext text/x-wiki 33wirdhn0n4pc5s0j8zaomykpglgnda 2450327 2450326 2026-08-20T23:38:49Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2450327 wikitext text/x-wiki 0pp3fxhx2uil3cnvum7ru9q3kp4wuvb 2450328 2450327 2026-08-20T23:43:07Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327652|RunSingleJob: Add ProfilingContext::init() (T435422)]] (duration: 12m 06s) 2450328 wikitext text/x-wiki fzosckl6don4r2l2a9fx91sb2bfpg0m 2450336 2450328 2026-08-21T02:00:40Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2450336 wikitext text/x-wiki t9wxmw4xqsmai5owc6qer3aakm48oir 2450337 2450336 2026-08-21T02:08:22Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 41s) 2450337 wikitext text/x-wiki j6i12l96ur6rs94tv6q8hodyzqhhi8i 2450338 2450337 2026-08-21T04:20:06Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2450338 wikitext text/x-wiki ic769jxssumn0wea2wgr30j6lg3k0ly 2450339 2450338 2026-08-21T04:20:45Z Stashbot 7414 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2450339 wikitext text/x-wiki 736p1cf6a61t6nikwwarrk6tyz0nyet 2450340 2450339 2026-08-21T04:20:47Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2450340 wikitext text/x-wiki 176qaswefh0zo574uxr7xu5tcg4bjgr 2450341 2450340 2026-08-21T04:21:29Z Stashbot 7414 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2450341 wikitext text/x-wiki 8iu8w5nzkqeghadbkhfo8lg6g82dn9k 2450342 2450341 2026-08-21T04:24:38Z Stashbot 7414 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1324963|Add Produnto to extension-list (T421436)]], [[gerrit:1324964|Enable Produnto on Beta (T421436)]] 2450342 wikitext text/x-wiki 512r6gbq9f687xndb7f1fu1xsma1bsk 2450343 2450342 2026-08-21T04:44:30Z Stashbot 7414 tstarling@deploy1003: tstarling: Backport for [[gerrit:1324963|Add Produnto to extension-list (T421436)]], [[gerrit:1324964|Enable Produnto on Beta (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450343 wikitext text/x-wiki gm7k0pbss7ihrf055ajid4jy44bgnf2 2450344 2450343 2026-08-21T04:45:36Z Stashbot 7414 tstarling@deploy1003: tstarling: Continuing with deployment 2450344 wikitext text/x-wiki 7l5e4pzyh65v2cu8iah81uinxuqjjzy 2450345 2450344 2026-08-21T04:59:27Z Stashbot 7414 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1324963|Add Produnto to extension-list (T421436)]], [[gerrit:1324964|Enable Produnto on Beta (T421436)]] (duration: 34m 48s) 2450345 wikitext text/x-wiki 6yo4ezbn4fous00cju2lt6x3qaw9q6w 2450347 2450345 2026-08-21T06:17:47Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1283 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96213 and previous config saved to /var/cache/conftool/dbconfig/20260821-061743-marostegui.json 2450347 wikitext text/x-wiki 730zjk3xzh8ul3syaav79192e0zi336 2450348 2450347 2026-08-21T06:18:06Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1283: Pool back 2450348 wikitext text/x-wiki lavvy3dhx5371qxpme9c51w6j8k3x2r 2450350 2450348 2026-08-21T06:35:09Z Stashbot 7414 moritzm: powercycling netmon2002 2450350 wikitext text/x-wiki iv40b3ncs4ztm87loxewnb6qsnc7qn8 2450351 2450350 2026-08-21T06:42:42Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts netmon2002.wikimedia.org 2450351 wikitext text/x-wiki jn86eb4pxnti085dwdzsd07fvqrg1gc 2450354 2450351 2026-08-21T07:03:10Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1283: Pool back 2450354 wikitext text/x-wiki ezahlvyhz928jqowvvln9wcgxerlwkt 2450355 2450354 2026-08-21T07:13:19Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2450355 wikitext text/x-wiki mot268nq3sg432lfh3ee82exou58zyb 2450356 2450355 2026-08-21T07:14:11Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2450356 wikitext text/x-wiki d0cu2yosjidaohhnsr4jx53fg3k2l8v 2450357 2450356 2026-08-21T07:14:19Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2450357 wikitext text/x-wiki egbijggmz63j722a3417eawud6trtpo 2450358 2450357 2026-08-21T07:14:59Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2450358 wikitext text/x-wiki gsq522za4u35hzz94u0qa7a3zfhtrb7 2450359 2450358 2026-08-21T07:15:08Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2450359 wikitext text/x-wiki px4efrfrmw3bh3c4crxw8o30z73m2lx 2450360 2450359 2026-08-21T07:18:25Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2450360 wikitext text/x-wiki sbq5iyytms2hu0qog8166y046yoiulw 2450361 2450360 2026-08-21T07:18:34Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2450361 wikitext text/x-wiki cs91dw5cyyg7sjupk27lh1awvfm0e8f 2450362 2450361 2026-08-21T07:24:54Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2450362 wikitext text/x-wiki mcyeyoidn6ldbybocl0872t8y428inz 2450363 2450362 2026-08-21T07:25:22Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2450363 wikitext text/x-wiki rpyojx1l9jck0e129fmfa3z46td4ccn 2450364 2450363 2026-08-21T07:25:25Z Stashbot 7414 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2450364 wikitext text/x-wiki dz1kkta4cluxammmartkz49bzfg8zfa 2450365 2450364 2026-08-21T07:25:33Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2450365 wikitext text/x-wiki hjxdss7uqyugp9a9jno0aejfj1i2ybf 2450366 2450365 2026-08-21T07:25:37Z Stashbot 7414 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2450366 wikitext text/x-wiki jbpl308qbtq0pebjibwvpwbnyasbftr 2450367 2450366 2026-08-21T07:25:44Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2450367 wikitext text/x-wiki 5i43mtcz6bw6jprjf5c5xlf5gmy5ofp 2450368 2450367 2026-08-21T07:25:47Z Stashbot 7414 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2450368 wikitext text/x-wiki i28um0jj0vadh4h8tgqaqsqo33be9ju 2450369 2450368 2026-08-21T07:25:55Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2450369 wikitext text/x-wiki cswzo48zhi792m8w27ya1nm9t3l671l 2450370 2450369 2026-08-21T07:25:58Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2450370 wikitext text/x-wiki pm7umzlxhrol1wqr1ovwfnsxb825uo7 2450373 2450370 2026-08-21T07:33:06Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1327679|Block: Disable flaky API test (T435272 T389028)]], [[gerrit:1327678|API: wfDebugLog for thumberror]] 2450373 wikitext text/x-wiki f8pwwtx6p2ifnzxduzsk8o7fodmxri6 2450374 2450373 2026-08-21T07:37:21Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1327679|Block: Disable flaky API test (T435272 T389028)]], [[gerrit:1327678|API: wfDebugLog for thumberror]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2450374 wikitext text/x-wiki qxem7bjjynznmlfcs553j086coec0mf 2450377 2450374 2026-08-21T07:41:26Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2450377 wikitext text/x-wiki pgu7a90hy8cxm8ggsjbb0czq5vgrq42 2450378 2450377 2026-08-21T07:48:40Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1327679|Block: Disable flaky API test (T435272 T389028)]], [[gerrit:1327678|API: wfDebugLog for thumberror]] (duration: 15m 34s) 2450378 wikitext text/x-wiki jcmuujpon7kptcu5xfuntb8zyd7rn7a 2450388 2450378 2026-08-21T08:31:57Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2450388 wikitext text/x-wiki jcz1imx8gsv39tnd1ddfl9lnaovumwd 2450390 2450388 2026-08-21T08:33:40Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-codfw 2450390 wikitext text/x-wiki im91ex0lnyj0wn0aw1y4hf1f5b96i9n 2450391 2450390 2026-08-21T08:33:44Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2001.codfw.wmnet 2450391 wikitext text/x-wiki f4lsrcycgq6c806tm9v7cr3rjojgdk3 2450392 2450391 2026-08-21T08:34:12Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2450392 wikitext text/x-wiki ebszrmb5eb2dnpc7w5v4x3l8eg4j60r 2450393 2450392 2026-08-21T08:36:29Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org 2450393 wikitext text/x-wiki j2f4fvmsmpeodsa9ki3y0ar1te6y3g5 2450394 2450393 2026-08-21T08:43:52Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2001.codfw.wmnet 2450394 wikitext text/x-wiki cghkhnkr9l6yre90ex3sryuoubugmly 2450395 2450394 2026-08-21T08:44:50Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org 2450395 wikitext text/x-wiki fwt054nviw5looaqqsejdk2l78l3kmq 2450396 2450395 2026-08-21T08:44:52Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts netmon2002.wikimedia.org 2450396 wikitext text/x-wiki 7pqixu4kaut9forphhcl81et6axfjy5 2450397 2450396 2026-08-21T08:47:42Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts netmon2002.wikimedia.org 2450397 wikitext text/x-wiki 83pyevkxo8avvw3p0ytgge06nxdcv5z 2450398 2450397 2026-08-21T08:48:03Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts netmon2002.wikimedia.org 2450398 wikitext text/x-wiki tdt1a8prc5o91yupqo8cqawonynlofs 2450399 2450398 2026-08-21T08:49:42Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2001.codfw.wmnet 2450399 wikitext text/x-wiki rhr76o70gmk7ulh2xg8c2r78hi8rg6j 2450400 2450399 2026-08-21T08:49:46Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2001.codfw.wmnet 2450400 wikitext text/x-wiki gkzau5ps2813yakn5z5lgoir4zzaw9d 2450401 2450400 2026-08-21T08:49:50Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2002.codfw.wmnet 2450401 wikitext text/x-wiki 7r1b9ncmf37sv4w0rzahlixx6g6re49 2450403 2450401 2026-08-21T08:54:56Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2002.codfw.wmnet 2450403 wikitext text/x-wiki gybmvy8tx4u3bhocrv664a0inngqe8i 2450404 2450403 2026-08-21T09:00:30Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2002.codfw.wmnet 2450404 wikitext text/x-wiki ipmphgkfjyyvh2b3fi3wlleu1x5cdhi 2450405 2450404 2026-08-21T09:00:32Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2002.codfw.wmnet 2450405 wikitext text/x-wiki mqsrm8slq3ys6byaluy962hnkfpk486 2450406 2450405 2026-08-21T09:00:37Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2003.codfw.wmnet 2450406 wikitext text/x-wiki s6m7owehqr2rn2zcyw96s8wc89vnckx 2450407 2450406 2026-08-21T09:03:20Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb1013.eqiad.wmnet 2450407 wikitext text/x-wiki 872cgpvlvrvwxpfspw5y2d8gfivsodh 2450408 2450407 2026-08-21T09:07:14Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host urldownloader1005.wikimedia.org with OS trixie 2450408 wikitext text/x-wiki numefmeoysf5v42zg5933d7qhgw29yi 2450410 2450408 2026-08-21T09:09:18Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1013.eqiad.wmnet 2450410 wikitext text/x-wiki lm5izp2ylkxphqdpud5vw42qz8zbjen 2450411 2450410 2026-08-21T09:10:46Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2003.codfw.wmnet 2450411 wikitext text/x-wiki 11miufu53e8w7zxkg6v4oga67b32dx1 2450412 2450411 2026-08-21T09:11:16Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb1015.eqiad.wmnet 2450412 wikitext text/x-wiki qca648wq71p15opqjom8mc2jg0c6zj1 2450413 2450412 2026-08-21T09:11:26Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2450413 wikitext text/x-wiki mjmar4jowxglt1s21pufsa5tf1mm7rg 2450414 2450413 2026-08-21T09:11:46Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2450414 wikitext text/x-wiki m0eozkxi31jvrhv9nxgvq3snqxodelj 2450415 2450414 2026-08-21T09:12:38Z Stashbot 7414 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2450415 wikitext text/x-wiki tt3nixhsrdff6anda6hs3h9y49uvgqo 2450416 2450415 2026-08-21T09:13:37Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr[1-2]-eqiad,pfw1-eqiad with reason: upgrade pfw1a-eqiad and pfw1b-eqiad pair 2450416 wikitext text/x-wiki l13stshcex5a3iive5atchpxiegb3ii 2450417 2450416 2026-08-21T09:16:02Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2003.codfw.wmnet 2450417 wikitext text/x-wiki fcfl4rifre9tymhvs6dvo7huwvicrbd 2450418 2450417 2026-08-21T09:16:04Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2003.codfw.wmnet 2450418 wikitext text/x-wiki tnwhnstqpx8gfqnbrmkem6slq33vw7l 2450419 2450418 2026-08-21T09:16:09Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2004.codfw.wmnet 2450419 wikitext text/x-wiki mams1pln3lf7dhtye0rlmhuyomc676a 2450420 2450419 2026-08-21T09:16:33Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1015.eqiad.wmnet 2450420 wikitext text/x-wiki menq8bweodzl5adoy5prjv51usqnxc2 2450421 2450420 2026-08-21T09:21:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2004.codfw.wmnet 2450421 wikitext text/x-wiki qv3p7gfh1hylrtass9yerl3llqmhxi3 2450422 2450421 2026-08-21T09:22:32Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage 2450422 wikitext text/x-wiki 120u0nd0boybnmx1wp6jj0b0echv8td 2450423 2450422 2026-08-21T09:24:41Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2011.codfw.wmnet 2450423 wikitext text/x-wiki 74bd4hmr43cevlcvwoaumfh9r5xg243 2450424 2450423 2026-08-21T09:26:24Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2004.codfw.wmnet 2450424 wikitext text/x-wiki ov89fx1myja5ho1189sgn9cz408jlrq 2450425 2450424 2026-08-21T09:26:27Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2004.codfw.wmnet 2450425 wikitext text/x-wiki 6twhgwv7vyl2ebg7ozcdxa2oainnskj 2450426 2450425 2026-08-21T09:26:32Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet 2450426 wikitext text/x-wiki 24m2xz0sdrhshsxahd7heldti3cuqv0 2450427 2450426 2026-08-21T09:29:50Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage 2450427 wikitext text/x-wiki pzzcugtffrkjrault0o4pxs54wbf6ub 2450428 2450427 2026-08-21T09:30:07Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2011.codfw.wmnet 2450428 wikitext text/x-wiki 933hmtffxorvljlx72k38rp9ipvjl12 2450429 2450428 2026-08-21T09:32:57Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet 2450429 wikitext text/x-wiki 3giwh3qe3vma0o3yuzjhvu09ukwveg1 2450430 2450429 2026-08-21T09:35:16Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database minwikiquote (T429946) 2450430 wikitext text/x-wiki 0njmf2ah9cjbnn44r6q93cw94ed76nz 2450431 2450430 2026-08-21T09:35:29Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database minwikiquote (T429946) 2450431 wikitext text/x-wiki 70y9ly24opk2yzjmjwbq31jauw2bv2q 2450432 2450431 2026-08-21T09:35:53Z Stashbot 7414 fnegri@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database bolwiki (T429954) 2450432 wikitext text/x-wiki pagaz2yqmdz071f7sfiwc9o72mpgjqh 2450433 2450432 2026-08-21T09:36:39Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet 2450433 wikitext text/x-wiki 790ce0yb9bbv8pgmy7lp059cknrmepk 2450434 2450433 2026-08-21T09:38:23Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet 2450434 wikitext text/x-wiki 0q29tewey3vtdynkbqmngybnntej6ys 2450435 2450434 2026-08-21T09:41:28Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet 2450435 wikitext text/x-wiki kpman9tz19f38ahzp65n9j20kxzny24 2450436 2450435 2026-08-21T09:41:30Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet 2450436 wikitext text/x-wiki 90ed4yw40f6t3pwik6vr8jd0nufy6rj 2450437 2450436 2026-08-21T09:41:35Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2006.codfw.wmnet 2450437 wikitext text/x-wiki ilz5vs56974tlz73rasogpwv5xx60te 2450438 2450437 2026-08-21T09:45:03Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1005.wikimedia.org with OS trixie 2450438 wikitext text/x-wiki drew8y0gy4wlsiuv4031mvea3l1fw59 2450439 2450438 2026-08-21T09:46:04Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host urldownloader1006.wikimedia.org with OS trixie 2450439 wikitext text/x-wiki sbimz44hwmonmnw37824vdl7vh1lzdl 2450440 2450439 2026-08-21T09:46:42Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2006.codfw.wmnet 2450440 wikitext text/x-wiki 7gfzhlqgxvpw4354w7590uxtsldbuwd 2450441 2450440 2026-08-21T09:51:33Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2006.codfw.wmnet 2450441 wikitext text/x-wiki co0qhrk1gokxtra98gvmvlvkqkyn4m8 2450442 2450441 2026-08-21T09:51:35Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2006.codfw.wmnet 2450442 wikitext text/x-wiki r4p71csna30k0euui4bi8g8iiems029 2450443 2450442 2026-08-21T09:51:40Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2007.codfw.wmnet 2450443 wikitext text/x-wiki hc21gow2vvdv4b6t87713kwejgju92n 2450444 2450443 2026-08-21T09:56:48Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2007.codfw.wmnet 2450444 wikitext text/x-wiki l2h8u1022v6gzjuogchk6yc1jy0ityb 2450445 2450444 2026-08-21T09:57:35Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage 2450445 wikitext text/x-wiki od3jo9ikw2ycajez9vhsrzs12h0dccq 2450446 2450445 2026-08-21T10:01:37Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2007.codfw.wmnet 2450446 wikitext text/x-wiki cc3bojf95kwq3cmt8bn9xsg29obz113 2450447 2450446 2026-08-21T10:01:40Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2007.codfw.wmnet 2450447 wikitext text/x-wiki 03hdi3hej8b9e6r7nuhfcy26ve1081i 2450448 2450447 2026-08-21T10:01:44Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2008.codfw.wmnet 2450448 wikitext text/x-wiki 247ehpgafpkv0r3y46e18yo813jvimx 2450449 2450448 2026-08-21T10:04:38Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage 2450449 wikitext text/x-wiki bejb4c8palixyzah5sjxztjegv9pbnk 2450450 2450449 2026-08-21T10:06:51Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2008.codfw.wmnet 2450450 wikitext text/x-wiki 991htbv845nb7hs0119b7fz446fn4p7 2450451 2450450 2026-08-21T10:13:19Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2008.codfw.wmnet 2450451 wikitext text/x-wiki 0062064gw4hmtmzt6d1bg2zcrz5yqkx 2450452 2450451 2026-08-21T10:13:23Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2008.codfw.wmnet 2450452 wikitext text/x-wiki 8k0gvmr78o68hf6k9noyspbwu71rvie 2450453 2450452 2026-08-21T10:13:27Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2009.codfw.wmnet 2450453 wikitext text/x-wiki 644tj0hf3ckbp6mgrd6prgi5mmg2mer 2450455 2450453 2026-08-21T10:18:34Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2009.codfw.wmnet 2450455 wikitext text/x-wiki eee6216pm9u6qpny93skl0g30jndygk 2450456 2450455 2026-08-21T10:20:34Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1006.wikimedia.org with OS trixie 2450456 wikitext text/x-wiki 8ihosnknhshtfndf9ufwkaciq9gv9d6 2450457 2450456 2026-08-21T10:24:55Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2009.codfw.wmnet 2450457 wikitext text/x-wiki m0xgkkiyk1kndjdgj6dd8a8jzr7acr7 2450458 2450457 2026-08-21T10:24:59Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2009.codfw.wmnet 2450458 wikitext text/x-wiki swulclwhchaxp9o51xxjf6vzv99dman 2450459 2450458 2026-08-21T10:25:02Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2010.codfw.wmnet 2450459 wikitext text/x-wiki t159u1txczo63x6octcn9pzevvbnsce 2450460 2450459 2026-08-21T10:30:11Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2010.codfw.wmnet 2450460 wikitext text/x-wiki am75w5979x7h71xsc9tozfn25x8yp56 2450461 2450460 2026-08-21T10:33:01Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host urldownloader2005.wikimedia.org with OS trixie 2450461 wikitext text/x-wiki 5bz1mflcer45odzw5pfzvl0ex5l06pg 2450463 2450461 2026-08-21T10:33:34Z Stashbot 7414 fnegri@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database bolwiki (T429954) 2450463 wikitext text/x-wiki qr4yq20wsfrpki6g0i2d5adokny7x1e 2450464 2450463 2026-08-21T10:34:52Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2010.codfw.wmnet 2450464 wikitext text/x-wiki izxzpxdcm6vgtel3bhzvbausm2r4e8z 2450465 2450464 2026-08-21T10:34:55Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2010.codfw.wmnet 2450465 wikitext text/x-wiki myhwexb257jg5y7fiu3ytri5912m877 2450466 2450465 2026-08-21T10:34:59Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2011.codfw.wmnet 2450466 wikitext text/x-wiki 06zlpka70q8xgu5cg9aimvtr8np4bpn 2450467 2450466 2026-08-21T10:40:08Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2011.codfw.wmnet 2450467 wikitext text/x-wiki f9sv06zfgd95n02culfh9s7ctvqikmn 2450470 2450467 2026-08-21T10:43:10Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2011.codfw.wmnet 2450470 wikitext text/x-wiki dwcbjix4g11fvkmv3zqo0xvru2t4wl6 2450471 2450470 2026-08-21T10:43:12Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2011.codfw.wmnet 2450471 wikitext text/x-wiki 6cnmk55988b4xtzyadbeylnojj3dmnm 2450472 2450471 2026-08-21T10:43:16Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-codfw 2450472 wikitext text/x-wiki k6dtvgp86caawjco9qhh6wf8s4nl802 2450473 2450472 2026-08-21T10:53:05Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage 2450473 wikitext text/x-wiki sat9pabvwcht09i4xgxxjc78o4jnukb 2450474 2450473 2026-08-21T10:57:41Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage 2450474 wikitext text/x-wiki bhkckgx1e7f4dr45c4uitly3exegm6l 2450475 2450474 2026-08-21T11:11:33Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2005.wikimedia.org with OS trixie 2450475 wikitext text/x-wiki 4rt6qlteoju27br3ti14j0hgsm3lnuu 2450476 2450475 2026-08-21T11:20:25Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host urldownloader2006.wikimedia.org with OS trixie 2450476 wikitext text/x-wiki g34zeqvto4fphmefmdsutqfqvak48fp 2450479 2450476 2026-08-21T11:38:03Z Stashbot 7414 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage 2450479 wikitext text/x-wiki 0uelbkdfdvj5svz4vkv3pc08485cby8 2450481 2450479 2026-08-21T11:41:45Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2450481 wikitext text/x-wiki daz2mx99tl7sdev7uqisxhxy78mhwax 2450482 2450481 2026-08-21T11:43:38Z Stashbot 7414 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage 2450482 wikitext text/x-wiki ltfz9vuhpk4i4fwnz4ox3mh0h3mnc4r 2450485 2450482 2026-08-21T11:58:28Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2450485 wikitext text/x-wiki oda9xwvns56xxygjh2q0xxzmh87tey0 498 8648 2450174 86615 2026-08-20T15:02:18Z Tara rafeiro 57424 added to my collecting collection of books 📕 and added, “citation” 2450174 wikitext text/x-wiki 09urxx1jz9bwa48slzkypqauy0y8p98 2450232 2450174 2026-08-20T17:03:01Z Taavi 13997 Reverted edit by [[Special:Contributions/Tara rafeiro|Tara rafeiro]] ([[User talk:Tara rafeiro|talk]]) to last revision by [[User:Tim.landscheidt|Tim.landscheidt]] 86615 wikitext text/x-wiki aw5ph1c0jtaebxnw5vurwgjk9ia0s16 0 10397 2450346 2395309 2026-08-21T05:26:20Z RKemper (WMF) 17058 /* Internal endpoints */ Fix inconsequential typo 2450346 wikitext text/x-wiki tu4h8pd9jj1rp4rymucp6rq40w4oz3y 0 17290 2450118 2449959 2026-08-20T13:35:54Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/WikiLambda] Re-enable Catalyst 2450118 wikitext text/x-wiki 7vpo5eec7i1qv3re5lmzjp3twodam22 2450173 2450118 2026-08-20T14:55:06Z Stashbot 7414 James_F: jforrester@integration-castor06:$ sudo rm -rf /srv/castor/castor-mw-ext-and-skins/master/wikilambda-catalyst-end-to-end # Clear stale Catalyst castor npm downloads. 2450173 wikitext text/x-wiki iq1xy16x7hr8jya2u7eyldx3oslij5g 2450324 2450173 2026-08-20T23:26:06Z Stashbot 7414 bd808: deployment-mediawiki14: `systemctl stop php8.3-fpm; sleep 5m; systemctl start php8.3-fpm` -- maybe the bot storm will break if we give fast 500 responses for 5 minutes. 2450324 wikitext text/x-wiki s54iusym3ww16mtwnzhuv3ktpj1xawo 498 30942 2450137 2449421 2026-08-20T13:58:13Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2450137 wikitext text/x-wiki m6vzxt3653x7d8wimluj1uqlsjsgt89 2450231 2450137 2026-08-20T16:57:54Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=97) (T429387) 2450231 wikitext text/x-wiki 78v5xuqrb3v1p9nkxirujhte0jgzdnu 2450249 2450231 2026-08-20T17:31:17Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2450249 wikitext text/x-wiki q4tytlob62uc86y42saxqab41panemk 2450250 2450249 2026-08-20T17:31:22Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=97) (T429387) 2450250 wikitext text/x-wiki isbaw4rw458plui3m2k8mhjk9yxci0t 2450282 2450250 2026-08-20T19:36:42Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2450282 wikitext text/x-wiki b9vi84a8spbj3cntmvorv54w56yaz99 2450329 2450282 2026-08-20T23:50:59Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) 2450329 wikitext text/x-wiki phogggcuknc796hfb1u7b5nq70da7lq 2450332 2450329 2026-08-21T00:08:55Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2450332 wikitext text/x-wiki hbes62cb9zr28ezw3oler6rlcvchk94 2450333 2450332 2026-08-21T00:08:59Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) 2450333 wikitext text/x-wiki 2t62imcohddnecg16sclgqvkenou4y2 2450334 2450333 2026-08-21T00:09:14Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) 2450334 wikitext text/x-wiki i2am7yfcff91onyrbr0o2buuhu8zvd6 2450349 2450334 2026-08-21T06:25:46Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) 2450349 wikitext text/x-wiki tlf5tnhlqkti92ez9awu9gsakk5y9at 0 33280 2450314 2418593 2026-08-20T22:14:30Z BryanDavis 1604 added [[Category:LDAP]] using [[w:Wikipedia:HotCat|HotCat]] 2450314 wikitext text/x-wiki jmynyfukphuttuaj4feh9wza138ywpw 498 55577 2450097 2447872 2026-08-20T13:10:06Z Stashbot 7414 godog: enable maintain-dumps-nfs - T432583 2450097 wikitext text/x-wiki 854xvitpxk04ut2t61hg80fg2c5877z 2450161 2450097 2026-08-20T14:40:00Z Stashbot 7414 taavi: stop running servers to pick up NFS changes T432583 2450161 wikitext text/x-wiki 8d5513vcwyf1qu2slofjeeor6mngxxp 2450353 2450161 2026-08-21T06:59:35Z Stashbot 7414 godog: stop remaining old servers to pick up NFS changes - T432583 2450353 wikitext text/x-wiki bx0ggercfvzahsjvww1t07tdkkza7lz 2450480 2450353 2026-08-21T11:41:43Z Stashbot 7414 godog: cordon paws-127d-mq2566hee6zs-node-4 - T435587 2450480 wikitext text/x-wiki 0v1pkkdhyu9a9vi2mensbujf2hkrd0q 2450484 2450480 2026-08-21T11:47:09Z Stashbot 7414 godog: reboot paws-127d-mq2566hee6zs-node-4 due to high load average - T435587 2450484 wikitext text/x-wiki qkcu7g687qy3rzgjuhwn9t30rfpwiup 0 327375 2450267 2421767 2026-08-20T18:25:22Z Ottomata 88 added UI screenshot 2450267 wikitext text/x-wiki fqlklosl4di7hf41v3vvkng4zh87c2o 116 372938 2450322 2233488 2026-08-20T22:17:49Z BryanDavis 1604 added [[Category:LDAP]] using [[w:Wikipedia:HotCat|HotCat]] 2450322 wikitext text/x-wiki l3va9x5yoi7b2l7oki5zgs070o6l3t2 498 443800 2450323 2449024 2026-08-20T23:14:17Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: rm -rf www-unused-since-2024-09-18/ # old source code, venv etc. – 130 MB / 124 MiB 2450323 wikitext text/x-wiki piwncv2qktrk8ts4heuosf2cvyb5uwc 498 443949 2450126 2449914 2026-08-20T13:42:51Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy ea800c5 (Add functional SPARQL tests against live endpoints) 2450126 wikitext text/x-wiki rzoa78athc6solncu06ytzf4014wy7g 2450127 2450126 2026-08-20T13:42:52Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 881928b (Increase webservice memory limit to 1Gi) 2450127 wikitext text/x-wiki 0skfogwxe5dvgigii1z0l63gf4aqzb4 0 445686 2450335 2449935 2026-08-21T01:23:05Z SFrench-WMF 37362 /* Adding tests */ Minor wording and formatting tweaks around the testwiki test suite 2450335 wikitext text/x-wiki ibj84jehabb9vlgu9qftq2bkcqbbrif 12 446021 2450136 2180346 2026-08-20T13:57:39Z Martin Urbanec (WMF) 37730 update; regenerated via gen_fingerprints on stat1008 (and verified via wmf-known-keys-production script before reconnecting) 2450136 wikitext text/x-wiki tswacebrag4k92ku7bxoylysl7lm3va 0 446051 2450201 2429686 2026-08-20T15:47:16Z MPopov (WMF) 5891 /* Directly from Kafka */ kafkacat -> kcat (upstream change, https://github.com/edenhill/kcat/issues/336) 2450201 wikitext text/x-wiki 16m010544q427a315mub4vfkjoyklcj 2 448526 2450311 2383531 2026-08-20T22:10:39Z BryanDavis 1604 /* Developer accounts in LDAP */ link to new home 2450311 wikitext text/x-wiki lkl4t08ss2m5mjgosv127o9c5ukssje 2450317 2450311 2026-08-20T22:14:58Z BryanDavis 1604 added [[Category:LDAP]] using [[w:Wikipedia:HotCat|HotCat]] 2450317 wikitext text/x-wiki 2to32gnojapvcsiwczumlsx677qfnqx 0 448732 2450062 2436553 2026-08-20T12:39:13Z LBarluzzi-WMF 56575 Add fail mail section for an error to ignore going forward 2450062 wikitext text/x-wiki jysgzizjv6knyzt3f13bkwpkkbdemx3 2450179 2450062 2026-08-20T15:19:39Z LBarluzzi-WMF 56575 Reminder of expected failures for venmo yearly recurring charge in July 2027 see phab T434741 2450179 wikitext text/x-wiki bgly8bdhcsme1v09vbor4b8v1kj3j5b 0 449160 2450331 2449936 2026-08-21T00:00:05Z Dexbot 30554 Bot: Updating the report 2450331 wikitext text/x-wiki 7w6qpruunodpjxgwdnb78h97vn8nmlf 0 451615 2450352 2036365 2026-08-21T06:43:08Z AKlapper (WMF) 421 not production anymore but not yet decom either 2450352 wikitext text/x-wiki bs7f7y22q3oc90a2ri937rpq0054t7h 0 452573 2450320 2072373 2026-08-20T22:15:54Z BryanDavis 1604 added [[Category:LDAP]] using [[w:Wikipedia:HotCat|HotCat]] 2450320 wikitext text/x-wiki r12ma72vt8mazfko5umgde1nlhhuzhu 12 453075 2450141 2092184 2026-08-20T14:00:38Z Martin Urbanec (WMF) 37730 update via gen_fingerprints and verified via wmf-known-keys-production 2450141 wikitext text/x-wiki rj0e74nsuf2c6w3s31oz9e36xso1pin 0 453597 2450121 2448928 2026-08-20T13:40:57Z DCaro (WMF) 18271 added note about components-cli not failing 2450121 wikitext text/x-wiki tk42gh5fjrhfw6m64pc0n87kwgnlu4b 0 453877 2450310 2136087 2026-08-20T22:08:44Z BryanDavis 1604 Make notes from [[User:BryanDavis/LDAP#Developer accounts in LDAP]] more discoverable and likely to be updated and augmented by moving into main namespace 2450310 wikitext text/x-wiki iowae86deyv95epnpusoe9p5tldz26o 2450316 2450310 2026-08-20T22:14:44Z BryanDavis 1604 added [[Category:LDAP]] using [[w:Wikipedia:HotCat|HotCat]] 2450316 wikitext text/x-wiki 8zzceb5dlt8o7jvci5v5bahlfa9ye39 116 453906 2450236 2448997 2026-08-20T17:18:23Z Gitlabaccountapprovalbot 37332 @jatwa was approved. 2450236 wikitext text/x-wiki 2bqdcl3j4za3ree8r7end1s2v6dgajn 12 454180 2450142 2153130 2026-08-20T14:02:18Z Martin Urbanec (WMF) 37730 update via gen_fingerprints and verified via wmf-known-keys-production 2450142 wikitext text/x-wiki 0al4cks2yabghwdbi7ux5uzf0pdvnhs 0 455989 2450483 2449331 2026-08-21T11:44:09Z SFaci-WMF 35637 /* Monitor event production */ 2450483 wikitext text/x-wiki 3nulkykb637rpokvwtlkq8walwvg5by 12 458702 2450230 2380036 2026-08-20T16:52:23Z Taavi-WMF 41365 add instructions for abuse reporting 2450230 wikitext text/x-wiki 59gojuddsxw5r7g0vlodn5u3li3a4nd 0 459290 2450477 2449766 2026-08-21T11:20:56Z APizzata-WMF 47941 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2450477 wikitext text/x-wiki 9eunj843xo6lp0mk3v996mxli8axs9z 2450478 2450477 2026-08-21T11:21:08Z APizzata-WMF 47941 /* YYYY-MM-DD NEXT TUESDAY TRAIN (REPLACE THIS AFTER DEPLOY) */ 2450478 wikitext text/x-wiki ni6z3j32g8g1qoxstf7daq4ipr202rv 2 460039 2450409 2440619 2026-08-21T09:07:40Z Atsuko-WMF 54082 2450409 wikitext text/x-wiki tjglqyc1sj4vzw09r1zz5gohlo2itq6 0 460052 2450308 2449168 2026-08-20T22:02:18Z MPopov (WMF) 5891 Experiment assignment query for app experiments 2450308 wikitext text/x-wiki tubsf7w0kkuo79x6gk8vl5kelwrtt40 0 460059 2450309 2433046 2026-08-20T22:04:34Z AMastilovic-WMF 40067 /* dbt vs Airflow DAGs */ 2450309 wikitext text/x-wiki 7w82h36338rkhttrxsvhz9h0vjkbbt7 0 460515 2450253 2449225 2026-08-20T17:50:39Z Ottomata 88 /* 2. Deploy the lambda to Kubernetes */ 2450253 wikitext text/x-wiki 6asuunx8v4bp2ykxrftxk55xlrcebmw 0 460556 2450081 2449998 2026-08-20T12:59:34Z Atsuko-WMF 54082 /* Why */ 2450081 wikitext text/x-wiki gghqeqly005bcz3plbougygfb70b1rx 2450087 2450081 2026-08-20T13:03:13Z Atsuko-WMF 54082 Added What is changing section 2450087 wikitext text/x-wiki 432g53705a7t5hv7fv8m8zekmxjveyx 2450090 2450087 2026-08-20T13:04:11Z Atsuko-WMF 54082 /* Give me the command */ 2450090 wikitext text/x-wiki fbj6e2a3bxvhv5o0fhxd24z7gnpohfa 2450162 2450090 2026-08-20T14:42:13Z Atsuko-WMF 54082 /* Give me the command */ 2450162 wikitext text/x-wiki n49xa4504jjvv3wej1xpxgwtudd2uzy 0 460561 2450454 2448837 2026-08-21T10:15:19Z KCVelaga (WMF) 38150 +init table 2450454 wikitext text/x-wiki 59fzwmlq7t2grb7qpnla1v0larlgq2h 2450462 2450454 2026-08-21T10:33:03Z KCVelaga (WMF) 38150 /* Datasets */ 2450462 wikitext text/x-wiki firdmoqnwa0tbd4x0zhtb495s6miinj 2450468 2450462 2026-08-21T10:40:55Z KCVelaga (WMF) 38150 /* mrt_api_requests_hourly */ 2450468 wikitext text/x-wiki nmbei6uc096l1oty5jxfhuqtax7lvz3 2450469 2450468 2026-08-21T10:41:08Z KCVelaga (WMF) 38150 /* api_requests_refined */ 2450469 wikitext text/x-wiki lh8xwlvkwq2cnmdxwqdew0x5uefnzzd 0 460571 2450306 2026-08-20T21:58:06Z AMastilovic-WMF 40067 Created page with "<!-- Suggested page title: Scheduling dbt Jobs on Airflow --> This page explains how to schedule dbt jobs on the Data Engineering '''main''' Airflow instance by editing YAML config files. It is a guide for the people who ''write'' those configs — you do not need to touch any Python. == Overview == * The dbt models themselves live in the separate '''<code>dbt-jobs</code>''' repository. This page is only about ''scheduling'' them. * Scheduling is driven by YAML files i..." 2450306 wikitext text/x-wiki g2acees9niho5bv8diq5beck3d2uydt 14 460572 2450321 2026-08-20T22:17:18Z BryanDavis 1604 new cat 2450321 wikitext text/x-wiki tihmvo3gr2hj0et8t5pobijsous5iw5 0 460573 2450371 2026-08-21T07:27:39Z JGiannelos (WMF) 17083 Created page with "Status: '''[[SLO/Runbook#Drafting|draft]]''' <mark>(Replace with "approved" when the SLO is complete, agreed on by all responsible teams, and subject to quarterly reporting. This is the final step. You can still update the SLO afterward, with all teams' agreement.)</mark> == Organizational == <mark>[[SLO/Template_instructions/Organizational|Instructions]]</mark> === Service === <mark>What is the service?</mark> === Teams === <mark>Who are the responsible teams?</mark>..." 2450371 wikitext text/x-wiki jmmgki40egxkm66qvkuapxroxr9ifc4 2450372 2450371 2026-08-21T07:31:32Z JGiannelos (WMF) 17083 2450372 wikitext text/x-wiki dwh5nnymcturp989vdwxa3d1xg8lits 0 460574 2450375 2026-08-21T07:37:50Z JGiannelos (WMF) 17083 Created page with "<mark>To get started, make a copy of this page, named after your service, as a subpage of [[SLO]]. (For example, "SLO/Foobaroid".) Also open the [[SLO/Template_instructions|instructions]], which guide you in more detail through the steps outlined here. Answer the questions in your new copy, removing ''all'' the yellow text, in mark tags, as you go. (Save early and often.)</mark> Status: '''[[SLO/Runbook#Drafting|draft]]''' <mark>(Replace with "approved" when the SLO is..." 2450375 wikitext text/x-wiki c2v9e69iid8s89iqumybv8ad9dm6eay 2450376 2450375 2026-08-21T07:38:13Z JGiannelos (WMF) 17083 2450376 wikitext text/x-wiki dwh5nnymcturp989vdwxa3d1xg8lits 2450379 2450376 2026-08-21T08:02:39Z JGiannelos (WMF) 17083 2450379 wikitext text/x-wiki g8h5b7efa8irfx0xc9hb0t2rrk0pwn4 2450381 2450379 2026-08-21T08:04:51Z JGiannelos (WMF) 17083 2450381 wikitext text/x-wiki 5mz9u6vz8rvrazw1d6ulugsh8rp7kpt 2450382 2450381 2026-08-21T08:08:14Z JGiannelos (WMF) 17083 2450382 wikitext text/x-wiki qcbxi57qxpoxqx8ciznvgvyd8eabx1r 2450383 2450382 2026-08-21T08:09:15Z JGiannelos (WMF) 17083 /* Teams */ 2450383 wikitext text/x-wiki 8ol543iyhfyjwozsdg9fpmfiok0eoqu 2450384 2450383 2026-08-21T08:09:28Z JGiannelos (WMF) 17083 2450384 wikitext text/x-wiki 746qoyxth9nusg8jpj936ua9g0aqq3l 2450385 2450384 2026-08-21T08:12:44Z JGiannelos (WMF) 17083 /* Environmental dependencies */ 2450385 wikitext text/x-wiki sq1n1sth2tv55drsmq5grm34kbl02i1 2450386 2450385 2026-08-21T08:13:12Z JGiannelos (WMF) 17083 2450386 wikitext text/x-wiki n1er20f1e7kp3nyjm993mnx4ir5rbw7 2450387 2450386 2026-08-21T08:17:54Z JGiannelos (WMF) 17083 2450387 wikitext text/x-wiki trnh8aaoh6ea29wowzwpcrjt6j5akb1 2450389 2450387 2026-08-21T08:32:05Z JGiannelos (WMF) 17083 /* Service dependencies */ 2450389 wikitext text/x-wiki rhw4u4m2r3vd7sqpvslhfuvwa8t3tpo 2450402 2450389 2026-08-21T08:53:10Z JGiannelos (WMF) 17083 /* Client-facing */ 2450402 wikitext text/x-wiki pyi449fia6nx16u7xrv4o0i1rvh2l39