Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.14 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2443980 2443973 2026-08-04T12:07:12Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor 2443980 wikitext text/x-wiki sb10shypa3hknylbbwxi82pzksocz7m 2443981 2443980 2026-08-04T12:10:05Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor 2443981 wikitext text/x-wiki 0uqkbgozsgvnyeuf9zi2x8jxskjupow 2443984 2443981 2026-08-04T12:13:47Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor 2443984 wikitext text/x-wiki bs34xnb1u14txdsrv9mgwk29oe545w2 2443987 2443984 2026-08-04T12:18:33Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor 2443987 wikitext text/x-wiki 6kvnc3antd42pd1czreyvi6v2zhpfcv 2444034 2443987 2026-08-04T13:52:38Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer 2444034 wikitext text/x-wiki sbs1g74sxrw35yhqgkbsbnrexmiuree 2444048 2444034 2026-08-04T14:14:41Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer 2444048 wikitext text/x-wiki b2bwx23zd86wbvmu0rrz8bzaizhr3ga 2444064 2444048 2026-08-04T14:32:28Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api 2444064 wikitext text/x-wiki 2o7c74zzq674b7lmm2iov56ddvo0hi9 2444078 2444064 2026-08-04T14:37:57Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api 2444078 wikitext text/x-wiki efjaf82e3qh003r69w861g1dighvt4k 2444087 2444078 2026-08-04T14:54:41Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api 2444087 wikitext text/x-wiki twff48tzimep8ahyx8i6bj1fekkftwo 2444089 2444087 2026-08-04T14:55:35Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api 2444089 wikitext text/x-wiki e39h3piiyvi0lwo70kn2xsk6pduy4zg 2444095 2444089 2026-08-04T15:01:27Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2444095 wikitext text/x-wiki tq7l1s0uv7v38ccsrcryru1vcv30bh5 2444102 2444095 2026-08-04T15:06:32Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder 2444102 wikitext text/x-wiki jpszuw926tz7wg2ukz1bh6g8itvyyio 2444118 2444102 2026-08-04T15:31:15Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2444118 wikitext text/x-wiki hl6yuunua5we8qqabt6ldn1b28tt8ca 2444121 2444118 2026-08-04T15:36:03Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2444121 wikitext text/x-wiki gu5byamyf56ilj42cnui14r81raej7q 2444149 2444121 2026-08-04T16:07:53Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry for certmanager 1.20.3 (T408786) 2444149 wikitext text/x-wiki d2ecyqdknot5zbq4yry661vpf0vnxrg 2444155 2444149 2026-08-04T16:08:48Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry (exit_code=0) for certmanager 1.20.3 (T408786) 2444155 wikitext text/x-wiki 8nme22p2f72ikbdrxje81vk9xnne5gi 2444186 2444155 2026-08-04T16:38:45Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images for certmanager 1.20.3 (T408786) 2444186 wikitext text/x-wiki kmn698cwlwxwl5komlqo4u3wrxrrzwo 2444187 2444186 2026-08-04T16:39:01Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images (exit_code=0) for certmanager 1.20.3 (T408786) 2444187 wikitext text/x-wiki mlo8hbngpylapw4xbm5bqfgmb4njhzm 0 4108 2444103 2443931 2026-08-04T15:07:10Z RLazarus (WMF) 15215 /* Tuesday, August 4 */ rm patch already deployed 2444103 wikitext text/x-wiki h4kyqxwu6yowl1ropgsbrdqcr47qilo 2444163 2444103 2026-08-04T16:12:26Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-08-04}} */ UTC-late infra window 2444163 wikitext text/x-wiki mtwh9m3p8jmookmpna8cpn3sec6pjz2 2444191 2444163 2026-08-04T16:49:53Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-08-04}} */ Add to UTC-late infra window 2444191 wikitext text/x-wiki eg3on5pwl90ld7wx4uda7qjx1ma046t 2444227 2444191 2026-08-04T18:40:41Z ScheduleDeploymentBot 37566 Add [[gerrit:1313827]] to Tuesday, August 04 UTC late backport window 2444227 wikitext text/x-wiki 4l9mhy95v9ilasahc4dd2077tjdtzkr 498 6726 2443976 2443907 2026-08-04T12:04:07Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor 2443976 wikitext text/x-wiki dj3o5luv7be0uhoc8rog0sflm2wa91i 2443979 2443976 2026-08-04T12:06:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor 2443979 wikitext text/x-wiki sixemqd18htawmt58x1cghxe9ik51gl 2444011 2443979 2026-08-04T13:26:12Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer 2444011 wikitext text/x-wiki 6e46zyzdztvoy8m74xplecdv7jv1ijj 2444027 2444011 2026-08-04T13:42:27Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer 2444027 wikitext text/x-wiki msk1x0iezt2imiuyilvzcnd0vri1erv 2444057 2444027 2026-08-04T14:26:23Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api 2444057 wikitext text/x-wiki p6av34subx8zu5hmga5yl2rk6ntknkk 2444063 2444057 2026-08-04T14:30:58Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api 2444063 wikitext text/x-wiki 2zcl216py097lhg520pa367mwwwmsbv 2444084 2444063 2026-08-04T14:51:33Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api 2444084 wikitext text/x-wiki aqtnvrty9tx24b8o480flhd074lrvls 2444086 2444084 2026-08-04T14:52:33Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api 2444086 wikitext text/x-wiki 8fdg946h1otn3mfkees34n6e3wh59sx 2444090 2444086 2026-08-04T14:56:18Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2444090 wikitext text/x-wiki la51r9s70rhfcb9xc549v13xbckeedg 2444094 2444090 2026-08-04T15:00:26Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder 2444094 wikitext text/x-wiki kx1hq9ut7dmdlme8xn4n04jj658i742 2444113 2444094 2026-08-04T15:26:37Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2444113 wikitext text/x-wiki 6rdvl07taqybdqbfcvn2bowhpxkrnlb 2444117 2444113 2026-08-04T15:30:56Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2444117 wikitext text/x-wiki k26emib52ab7vlss29cr0kjcpj6rkpm 0 7919 2443974 2443972 2026-08-04T11:59:41Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1287.eqiad.wmnet 2443974 wikitext text/x-wiki sq8vajupdzk4n9n9m0c8ew0c9uahrux 2443975 2443974 2026-08-04T11:59:55Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1288.eqiad.wmnet 2443975 wikitext text/x-wiki ozqpx7orh3x239n6w0uem8ijgy6vy18 2443977 2443975 2026-08-04T12:05:12Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1288.eqiad.wmnet 2443977 wikitext text/x-wiki pq9sfanm19qrn57zptiehvyhwv5z02d 2443978 2443977 2026-08-04T12:05:26Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1289.eqiad.wmnet 2443978 wikitext text/x-wiki o2nrug59ub6fmoar8ekm2mfdce5ve8b 2443982 2443978 2026-08-04T12:10:39Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1289.eqiad.wmnet 2443982 wikitext text/x-wiki ofzbtb0dds2yduecj3pp9f8gclhi4ps 2443983 2443982 2026-08-04T12:10:50Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2096.codfw.wmnet with reason: host reimage 2443983 wikitext text/x-wiki 9v1u2c02h6s7u3t003lns9sihegvluo 2443985 2443983 2026-08-04T12:14:08Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1141.eqiad.wmnet with OS trixie 2443985 wikitext text/x-wiki mpp820lvx3uy3rrozymr6wwviis10xk 2443986 2443985 2026-08-04T12:14:47Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2096.codfw.wmnet with reason: host reimage 2443986 wikitext text/x-wiki ksc1j5649ao5ljzuu899656g4t18722 2443988 2443986 2026-08-04T12:22:45Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1096.eqiad.wmnet with OS trixie 2443988 wikitext text/x-wiki 7kkjdhs56azzcbh1uu5p0k1mihafb2i 2443989 2443988 2026-08-04T12:33:04Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2096.codfw.wmnet with OS trixie 2443989 wikitext text/x-wiki 2qt3yk3840orr3t1pa0fjj922o8877c 2443991 2443989 2026-08-04T12:38:16Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[1164,1217].eqiad.wmnet with reason: cloning 2443991 wikitext text/x-wiki sps82v274ga3o1dlb0xxiisi8hwqdrk 2443994 2443991 2026-08-04T12:42:14Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1096.eqiad.wmnet with reason: host reimage 2443994 wikitext text/x-wiki bcfeepohdpd7vojyj9q4ed204kjppg6 2443995 2443994 2026-08-04T12:42:55Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 1:00:00 on db1150.eqiad.wmnet with reason: decom 2443995 wikitext text/x-wiki frdlm06hxvv3hk350a0d4g8ovchupjd 2443996 2443995 2026-08-04T12:43:26Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 1:00:00 on db1171.eqiad.wmnet with reason: decom 2443996 wikitext text/x-wiki 1zctu45j4n9drcjj0gf2fnpc4wrk7cj 2443999 2443996 2026-08-04T12:49:04Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1096.eqiad.wmnet with reason: host reimage 2443999 wikitext text/x-wiki 2c023qe8bgl50pv3ofhtbthjuoem8be 2444002 2443999 2026-08-04T13:03:04Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2444002 wikitext text/x-wiki ti7siyoiikw9nnhu6c3m7l4s7mwtkt8 2444005 2444002 2026-08-04T13:05:15Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2444005 wikitext text/x-wiki 8jw97hs4msu810708e5qs1khy6xmft9 2444006 2444005 2026-08-04T13:06:30Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1096.eqiad.wmnet with OS trixie 2444006 wikitext text/x-wiki l6rpezyv7065fe4tgmzbxmt9wch8hrl 2444009 2444006 2026-08-04T13:22:52Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2444009 wikitext text/x-wiki 9yhod72avpy200q6n03mgekvbw17e64 2444010 2444009 2026-08-04T13:22:56Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2444010 wikitext text/x-wiki 6yzvrfrvskui6v28s17xa0ib4p450hn 2444012 2444010 2026-08-04T13:28:27Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1141.eqiad.wmnet 2444012 wikitext text/x-wiki 0v8wzxup3m92of181ebemhpfi3dr2mw 2444013 2444012 2026-08-04T13:28:29Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1141.eqiad.wmnet 2444013 wikitext text/x-wiki efd02szlk2n14mq7t1bx3pvi3k9c35n 2444014 2444013 2026-08-04T13:28:31Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1141.eqiad.wmnet 2444014 wikitext text/x-wiki ic60qh89avy8sojxpl215g6c64tu4lz 2444015 2444014 2026-08-04T13:28:39Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1154.eqiad.wmnet 2444015 wikitext text/x-wiki 5utw9x5k0wtiri1z1jmmnslk9s6xd7g 2444016 2444015 2026-08-04T13:28:42Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1154.eqiad.wmnet 2444016 wikitext text/x-wiki saf9pfyynjprqhmbxsqqb1rqpi721hu 2444017 2444016 2026-08-04T13:29:50Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1154.eqiad.wmnet 2444017 wikitext text/x-wiki p30u4gvnz5xudzq9ayqklffols8zj2s 2444018 2444017 2026-08-04T13:31:10Z Stashbot 7414 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2444018 wikitext text/x-wiki d5nhekq9jec0kqhjvg6jhtzcifykgc6 2444019 2444018 2026-08-04T13:31:40Z Stashbot 7414 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2444019 wikitext text/x-wiki f9dxr6dh9krennq1vjemmmvpn72tuxa 2444020 2444019 2026-08-04T13:31:53Z Stashbot 7414 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2444020 wikitext text/x-wiki b7ncefi3pv9z9ac1ltqgcmq93xj24d1 2444021 2444020 2026-08-04T13:32:41Z Stashbot 7414 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2444021 wikitext text/x-wiki hci190bk087pagz3mktrz8zt4h96570 2444022 2444021 2026-08-04T13:32:54Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2444022 wikitext text/x-wiki 0yvzftbpmfoc46ae5kfo7tihanlss78 2444023 2444022 2026-08-04T13:33:05Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1154.eqiad.wmnet 2444023 wikitext text/x-wiki ralhsk12kwlrkp9dj97av9z7lfy2zce 2444024 2444023 2026-08-04T13:33:38Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2444024 wikitext text/x-wiki 6shy9y3skacixkt2k0cvmnf3u9ye3na 2444025 2444024 2026-08-04T13:38:54Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1320171|Thumbnail: Exclude map figures from multimediaviewer (T433703 T427709)]], [[gerrit:1320159|TimedText: Include SRT-only sources when listing playback tracks (T433666)]] 2444025 wikitext text/x-wiki kka4uxnslp8gj3gd8sxef3459zvds7n 2444026 2444025 2026-08-04T13:41:12Z Stashbot 7414 mszwarc@deploy1003: mszwarc, jforrester: Backport for [[gerrit:1320171|Thumbnail: Exclude map figures from multimediaviewer (T433703 T427709)]], [[gerrit:1320159|TimedText: Include SRT-only sources when listing playback tracks (T433666)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2444026 wikitext text/x-wiki 73zwn17bzazzj77j8eogzl78y0z8yud 2444028 2444026 2026-08-04T13:43:50Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2444028 wikitext text/x-wiki 1621pl720f7kx8gqumftka1op73j6ig 2444029 2444028 2026-08-04T13:44:02Z Stashbot 7414 mszwarc@deploy1003: mszwarc, jforrester: Continuing with deployment 2444029 wikitext text/x-wiki blvwp2xjvn86geijjec96ozc81x9n8f 2444030 2444029 2026-08-04T13:45:48Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2444030 wikitext text/x-wiki ns2jbhdtxgvj752pp4f8zrlqq4y1e9j 2444031 2444030 2026-08-04T13:48:12Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1320171|Thumbnail: Exclude map figures from multimediaviewer (T433703 T427709)]], [[gerrit:1320159|TimedText: Include SRT-only sources when listing playback tracks (T433666)]] (duration: 09m 19s) 2444031 wikitext text/x-wiki a9q0bu97k009d7k116rv22figm2p8zi 2444032 2444031 2026-08-04T13:49:24Z Stashbot 7414 swfrench@cumin2002: conftool action : set/pooled=no; selector: name=wikikube-worker2330.codfw.wmnet 2444032 wikitext text/x-wiki 94s77vydop1r49mccncvlm7mwh603bi 2444033 2444032 2026-08-04T13:49:50Z Stashbot 7414 swfrench@cumin2002: conftool action : set/pooled=yes; selector: name=wikikube-worker2330.codfw.wmnet 2444033 wikitext text/x-wiki i1m9qmty5ba5yraidlund9uc6qwwr71 2444035 2444033 2026-08-04T14:00:07Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw 2444035 wikitext text/x-wiki 671m77xgfjvt3k6dhhlsygoou4r11io 2444036 2444035 2026-08-04T14:00:10Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw 2444036 wikitext text/x-wiki h878twtvy13l3a2iiauebsj6kqvzpov 2444037 2444036 2026-08-04T14:00:11Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw 2444037 wikitext text/x-wiki 9hcmpq8ao6gri3xhxma2mhf9adiy98l 2444038 2444037 2026-08-04T14:03:24Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply logging and security config updates - bking@cumin2003 - T324335 2444038 wikitext text/x-wiki 0zzcglwp6h47oen81tlbwl8gdqtygyc 2444039 2444038 2026-08-04T14:05:08Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1320757|stream: pageview.trending.relative.v1 (T432204)]], [[gerrit:1319086|WikimediaAntiAbuse: Document required load order after Echo (T432452)]], [[gerrit:1320781|UIC: Add user name to server-side instrumentation events (T433816)]], [[gerrit:1320782|UIC: Add user name to server-side instrumentation events (T433816)]] 2444039 wikitext text/x-wiki 41kxh5p911bl06s4xfn83dxicotwz4z 2444040 2444039 2026-08-04T14:07:10Z Stashbot 7414 mszwarc@deploy1003: javiermonton, mszwarc, mpostoronca: Backport for [[gerrit:1320757|stream: pageview.trending.relative.v1 (T432204)]], [[gerrit:1319086|WikimediaAntiAbuse: Document required load order after Echo (T432452)]], [[gerrit:1320781|UIC: Add user name to server-side instrumentation events (T433816)]], [[gerrit:1320782|UIC: Add user name to server-side instrumentation events (T433816)]] synced to the testser 2444040 wikitext text/x-wiki k8y6unxcedgz9upa65yjr858hgndl9n 2444041 2444040 2026-08-04T14:08:59Z Stashbot 7414 mszwarc@deploy1003: javiermonton, mszwarc, mpostoronca: Continuing with deployment 2444041 wikitext text/x-wiki rlm5tcd1ln9k0uzv8hv1wuwe38zh5qf 2444042 2444041 2026-08-04T14:11:26Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2444042 wikitext text/x-wiki ekbhadb9ogsyjv30646y921eegiffbh 2444043 2444042 2026-08-04T14:13:06Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1320757|stream: pageview.trending.relative.v1 (T432204)]], [[gerrit:1319086|WikimediaAntiAbuse: Document required load order after Echo (T432452)]], [[gerrit:1320781|UIC: Add user name to server-side instrumentation events (T433816)]], [[gerrit:1320782|UIC: Add user name to server-side instrumentation events (T433816)]] (duration: 07m 58s) 2444043 wikitext text/x-wiki t7bls81dtj9kwt373dymvz93b6jsnw3 2444044 2444043 2026-08-04T14:13:09Z Stashbot 7414 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2444044 wikitext text/x-wiki freab3agehes20p7q6fbk0x1rtnc8nn 2444045 2444044 2026-08-04T14:13:27Z Stashbot 7414 Msz2001: Finished deployments for UTC afternoon backport window 2444045 wikitext text/x-wiki t6wh3z9dnrismoc7jrcu48ow2s16rsu 2444046 2444045 2026-08-04T14:13:30Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2444046 wikitext text/x-wiki e4beysn7uh6ih2bsdvvnzy2j7yf9q5b 2444047 2444046 2026-08-04T14:13:32Z Stashbot 7414 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2444047 wikitext text/x-wiki d6j9nksmgse1g2y77tow7wgazkgh9w0 2444049 2444047 2026-08-04T14:14:46Z Stashbot 7414 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2444049 wikitext text/x-wiki 0olfxo2y3oa5g8bkdghajod5evuba7d 2444050 2444049 2026-08-04T14:15:41Z Stashbot 7414 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2444050 wikitext text/x-wiki ghhrexajejli2zle61lve5fxzoe7agw 2444051 2444050 2026-08-04T14:16:22Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2444051 wikitext text/x-wiki p9uiy2wsuqgpzkzye5fijznxqw10iyj 2444052 2444051 2026-08-04T14:17:10Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2444052 wikitext text/x-wiki 19syquv43ro5pfhc4n7fgsrd6pscesp 2444053 2444052 2026-08-04T14:26:11Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1154.eqiad.wmnet 2444053 wikitext text/x-wiki rzhllu1b6ygixxts08qzqanrga1ncyt 2444054 2444053 2026-08-04T14:26:15Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1154.eqiad.wmnet 2444054 wikitext text/x-wiki g0pdtoxwu0nnq1nijr7ej6ewpk5mlr4 2444055 2444054 2026-08-04T14:26:18Z Stashbot 7414 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1320941|EventStreamConfig - page_html_feature_counts_change.v1 canary to eventgate-main (T433507)]] 2444055 wikitext text/x-wiki 2x8fiz1xex7i8rh6ngi52pf7mmi0412 2444056 2444055 2026-08-04T14:26:20Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1154.eqiad.wmnet 2444056 wikitext text/x-wiki fkpphjr4lxc6ayrrzy1wdmppq2ul652 2444058 2444056 2026-08-04T14:26:30Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1154.eqiad.wmnet with OS trixie 2444058 wikitext text/x-wiki earxe8v8bugoe3flksu5xqatn7zc21q 2444059 2444058 2026-08-04T14:26:58Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1154 2444059 wikitext text/x-wiki mafrl0wx8usdjhdc184enled5ll8xfz 2444060 2444059 2026-08-04T14:28:16Z Stashbot 7414 otto@deploy1003: otto: Backport for [[gerrit:1320941|EventStreamConfig - page_html_feature_counts_change.v1 canary to eventgate-main (T433507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2444060 wikitext text/x-wiki fv8zw5s2yos5ug5xeucdj1olqk87hk8 2444061 2444060 2026-08-04T14:30:07Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.dns.netbox 2444061 wikitext text/x-wiki g8x1ogw51yrcgjkq6di5293z049zrk7 2444062 2444061 2026-08-04T14:30:56Z Stashbot 7414 otto@deploy1003: otto: Continuing with deployment 2444062 wikitext text/x-wiki kbgd5ggxej5nx9ki3psft0a0289snze 2444065 2444062 2026-08-04T14:34:17Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1154 - jayme@cumin1003" 2444065 wikitext text/x-wiki ewcuntmz4yohzh4bp24bz88akhxe2gr 2444066 2444065 2026-08-04T14:34:22Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1154 - jayme@cumin1003" 2444066 wikitext text/x-wiki tk4zdr3oxr4eve6l43031i1z7ih3x30 2444067 2444066 2026-08-04T14:34:24Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2444067 wikitext text/x-wiki j1l09811ys33pyfmr8fy6bgc4zb3lel 2444068 2444067 2026-08-04T14:34:26Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1154.eqiad.wmnet 108.32.64.10.in-addr.arpa 8.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2444068 wikitext text/x-wiki glonyv28ed6f1fbruehd583g2v4hzxf 2444069 2444068 2026-08-04T14:34:28Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1154.eqiad.wmnet 108.32.64.10.in-addr.arpa 8.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2444069 wikitext text/x-wiki jf8pmhw6zg5kfs5opshoajnf2ks2nyn 2444070 2444069 2026-08-04T14:34:30Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1154 2444070 wikitext text/x-wiki qf7rtslp1v14ezt78hjlavmdt5bfmzm 2444071 2444070 2026-08-04T14:34:57Z Stashbot 7414 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1320941|EventStreamConfig - page_html_feature_counts_change.v1 canary to eventgate-main (T433507)]] (duration: 08m 39s) 2444071 wikitext text/x-wiki 3s3wekbngenr39w8k5x6ltyidb0z9f2 2444072 2444071 2026-08-04T14:36:28Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1154 2444072 wikitext text/x-wiki 7f8hw3kxgyw8boi7fka586h0nj5eowf 2444073 2444072 2026-08-04T14:36:30Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1154 2444073 wikitext text/x-wiki 7a2td341tqnoroa70hch5qlp65wugkr 2444074 2444073 2026-08-04T14:36:55Z Stashbot 7414 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-main: sync 2444074 wikitext text/x-wiki lwo50wfj02lxg9729cg4pli3yz8jv6l 2444075 2444074 2026-08-04T14:37:02Z Stashbot 7414 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-main: sync 2444075 wikitext text/x-wiki 4c3dz4zr1nn55h1aln8zw8jzea4yo9k 2444076 2444075 2026-08-04T14:37:20Z Stashbot 7414 ottomata: roll restart eventgate-main to pick up stream config change - T433507 2444076 wikitext text/x-wiki 2wm6c5woa9hawi88dsjcxhzrnz8ehm4 2444077 2444076 2026-08-04T14:37:37Z Stashbot 7414 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync 2444077 wikitext text/x-wiki cxu4r21u15sj36j0p3mqxtq3pstac4d 2444079 2444077 2026-08-04T14:38:03Z Stashbot 7414 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync 2444079 wikitext text/x-wiki tl91icmz99cprzk44nn3nwi6l50jyxk 2444080 2444079 2026-08-04T14:38:30Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync 2444080 wikitext text/x-wiki 7riuo5sozvxww1jwf3vv3ppp7sqk75r 2444081 2444080 2026-08-04T14:38:53Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync 2444081 wikitext text/x-wiki srgiqgwo7nppqeqy8zwvquk6rljg94q 2444082 2444081 2026-08-04T14:48:07Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply 2444082 wikitext text/x-wiki ogq3kun6mej67ldm0fuikht6y3setyb 2444083 2444082 2026-08-04T14:48:39Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply 2444083 wikitext text/x-wiki i36d31t3qrm70wmhi4we3zkri0bujae 2444085 2444083 2026-08-04T14:51:48Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1154.eqiad.wmnet with reason: host reimage 2444085 wikitext text/x-wiki tmg4mjaboavo0fr5pkq6zqj6cw34cgf 2444088 2444085 2026-08-04T14:55:26Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1154.eqiad.wmnet with reason: host reimage 2444088 wikitext text/x-wiki ggt3172w5o0fopn6m5eviutwm41r9p3 2444091 2444088 2026-08-04T14:58:53Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab2003.codfw.wmnet with reason: deployment 2444091 wikitext text/x-wiki 43ozwiwnledxmmvkya7ol3rdwbzafqj 2444092 2444091 2026-08-04T14:59:28Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab1005.eqiad.wmnet with reason: deployment 2444092 wikitext text/x-wiki fx9wg5cgvg65gf9ihnsxkw5kd34buuy 2444093 2444092 2026-08-04T14:59:52Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab1004.eqiad.wmnet with reason: deployment 2444093 wikitext text/x-wiki aif521ggs5ekcblfohn7eb0r6z9lcdk 2444096 2444093 2026-08-04T15:01:43Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@56f4ffd]: deploy phab2003 for T433981 2444096 wikitext text/x-wiki da1akis5npbgsm9p7tfppbg1icky9ci 2444097 2444096 2026-08-04T15:02:35Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@56f4ffd]: deploy phab2003 for T433981 (duration: 00m 51s) 2444097 wikitext text/x-wiki 55alstw02htk8x0l1xgjba3h8ggla1y 2444098 2444097 2026-08-04T15:04:27Z Stashbot 7414 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2444098 wikitext text/x-wiki 0yi0k74l95aezdr1rqnm292p4z4a3e6 2444099 2444098 2026-08-04T15:05:23Z Stashbot 7414 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2444099 wikitext text/x-wiki 6sljmyu2d13foqhyglr1bnoo0sevomd 2444100 2444099 2026-08-04T15:05:40Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@56f4ffd]: deploy phab1004 for T433981 2444100 wikitext text/x-wiki n5q487pg7k335smeskcdydojo20m9as 2444101 2444100 2026-08-04T15:06:28Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@56f4ffd]: deploy phab1004 for T433981 (duration: 00m 43s) 2444101 wikitext text/x-wiki 7ewl0mdln8qagpjri5idxv1izjcdtuz 2444104 2444101 2026-08-04T15:15:10Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444104 wikitext text/x-wiki r5ti9bf8yhrare3og5gzcb0inxwxywp 2444105 2444104 2026-08-04T15:15:17Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) 2444105 wikitext text/x-wiki c5vty4x2ekmqm4888zph2ptglj59g1e 2444106 2444105 2026-08-04T15:16:13Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444106 wikitext text/x-wiki qvn6bco5opiiu9pghrq2j611062l18l 2444107 2444106 2026-08-04T15:16:19Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1154.eqiad.wmnet with OS trixie 2444107 wikitext text/x-wiki aop9rbx8j8ez7ev8ulqf35rkb2t6by2 2444108 2444107 2026-08-04T15:16:22Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) 2444108 wikitext text/x-wiki b9lbhc7pydfurq9lvnw4h424atd1ylz 2444110 2444108 2026-08-04T15:19:44Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444110 wikitext text/x-wiki ggplnao4wr76xo3k3lnblxlxymqk17m 2444111 2444110 2026-08-04T15:19:52Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) 2444111 wikitext text/x-wiki 6k880r4i1xkumwy3r7l4yqbtigv4nud 2444116 2444111 2026-08-04T15:29:42Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply logging and security config updates - bking@cumin2003 - T324335 2444116 wikitext text/x-wiki bokbflyagsuuoiivjp0r0ytkpzb717c 2444119 2444116 2026-08-04T15:33:25Z Stashbot 7414 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2444119 wikitext text/x-wiki rjzs5zd8856vm1im4f13nsmrsbjvsmr 2444120 2444119 2026-08-04T15:33:48Z Stashbot 7414 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2444120 wikitext text/x-wiki 2xdz6mhxu1iw44kmr6ai87ppcdfo5ed 2444124 2444120 2026-08-04T15:39:02Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply logging and security config updates - bking@cumin2003 - T324335 2444124 wikitext text/x-wiki eer9g4oxgn5tn5jo57hmj9f24rlyhus 2444126 2444124 2026-08-04T15:44:51Z Stashbot 7414 Raine: add php8.5 packages to component/php85 - T432983 2444126 wikitext text/x-wiki sa4xl0pvqsvutef2tp2cgyhfddayoxh 2444129 2444126 2026-08-04T15:49:07Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444129 wikitext text/x-wiki 96akutvfanzld1et4j8o2t0nb9fv16s 2444130 2444129 2026-08-04T15:49:18Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444130 wikitext text/x-wiki rvbiw3nhzenk8g5a0klajetjqkjqr8s 2444131 2444130 2026-08-04T15:55:34Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444131 wikitext text/x-wiki ehyxpnx0wxejemwzz9gwp8twob3a9gl 2444132 2444131 2026-08-04T15:55:43Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) 2444132 wikitext text/x-wiki jh7amj4u5mgdiad4t4dymecp034sd0h 2444133 2444132 2026-08-04T15:55:52Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444133 wikitext text/x-wiki drcvodsqgzmcayorxbrltc8bfhj7tqp 2444134 2444133 2026-08-04T15:56:03Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444134 wikitext text/x-wiki 81r8ns9l7b3bh75azrtsmqz7hnryia7 2444136 2444134 2026-08-04T15:59:07Z Stashbot 7414 robh@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022.mgmt.eqsin.wmnet'] 2444136 wikitext text/x-wiki lg8dso1de528385xznz6ygtjub7dzp2 2444137 2444136 2026-08-04T15:59:10Z Stashbot 7414 robh@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp5022.mgmt.eqsin.wmnet'] 2444137 wikitext text/x-wiki shjmcfvobfz5ugkect5zhdks8mtl6my 2444139 2444137 2026-08-04T15:59:42Z Stashbot 7414 robh@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022.mgmt.eqsin.wmnet'] 2444139 wikitext text/x-wiki f9sfj8dyxtb4vgpkrhf1ejxnxndel2l 2444140 2444139 2026-08-04T15:59:46Z Stashbot 7414 robh@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp5022.mgmt.eqsin.wmnet'] 2444140 wikitext text/x-wiki 58pqzbzf8c82vaj33xxlrl0dbihx2e5 2444142 2444140 2026-08-04T16:04:59Z Stashbot 7414 mutante: gerrit2002/gerrit1003/gerrit2003 - rm /etc/gerrit/ssh_host_rsa_key 2444142 wikitext text/x-wiki c5fk6c69oqljkm7zhhc29cbqufvkduq 2444143 2444142 2026-08-04T16:05:13Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet 2444143 wikitext text/x-wiki eyspcuiqkp5kja643okgjji6goiupgr 2444144 2444143 2026-08-04T16:07:22Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2444144 wikitext text/x-wiki j8nfeh4dj0v583wwh5lrd93nwsmfgxp 2444145 2444144 2026-08-04T16:07:29Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet 2444145 wikitext text/x-wiki afv7k3nec0n6ygg1p5kczua47pup913 2444146 2444145 2026-08-04T16:07:32Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2444146 wikitext text/x-wiki ch1gu292yw2capoxsll9g36z4jqdva2 2444147 2444146 2026-08-04T16:07:34Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2444147 wikitext text/x-wiki 008e9g50eaci051mth17gb1tvoxxqiv 2444148 2444147 2026-08-04T16:07:38Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2444148 wikitext text/x-wiki p1ktm7fk97amimf3ao8n6ga8ncb83zd 2444150 2444148 2026-08-04T16:07:55Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2444150 wikitext text/x-wiki gn84r6802znvmxm6toworkzkifr86rm 2444151 2444150 2026-08-04T16:07:57Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2444151 wikitext text/x-wiki 9boj4t317j101nlbg8kl8ci75acp2vs 2444152 2444151 2026-08-04T16:08:18Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2444152 wikitext text/x-wiki plkf1vqzxn1cp148a759qtntvu3ywd4 2444153 2444152 2026-08-04T16:08:22Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2444153 wikitext text/x-wiki raduysmagt88iajpb2rldeczci3gux8 2444154 2444153 2026-08-04T16:08:40Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2444154 wikitext text/x-wiki s3zly4w2kww7vqj756bh3n49ot8ryf6 2444156 2444154 2026-08-04T16:09:22Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2444156 wikitext text/x-wiki nt2covcoqqru6944rbmlj8wagrxi5n2 2444157 2444156 2026-08-04T16:09:27Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply 2444157 wikitext text/x-wiki 6a8htemzkkukul1t2dh6phq5z38qom9 2444158 2444157 2026-08-04T16:10:10Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply 2444158 wikitext text/x-wiki c5lt2zo8d4gbpxgpvma7vzfgwby70cz 2444159 2444158 2026-08-04T16:10:46Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2444159 wikitext text/x-wiki 84kc0hlr6pgt62hawiqlpsx5ybximgo 2444161 2444159 2026-08-04T16:11:29Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2444161 wikitext text/x-wiki g1mi34brg8w9s3tm0u50h4o7bo8pgel 2444164 2444161 2026-08-04T16:17:43Z Stashbot 7414 swfrench-wmf: reprepro include php8.3_8.3.33-1+wmf12u1 into component/php83 for bookworm-wikimedia 2444164 wikitext text/x-wiki lm0ujdnnvy0vlitc8fyez1khsnu2tj0 2444165 2444164 2026-08-04T16:17:56Z Stashbot 7414 swfrench-wmf: reprepro include php8.3_8.3.33-1+wmf11u1 into component/php83 for bullseye-wikimedia 2444165 wikitext text/x-wiki dshygvvtp8h2pxi3r1rm9i7txf2758x 2444166 2444165 2026-08-04T16:21:51Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance 2444166 wikitext text/x-wiki humgmo64ye7b47993g72ovl90g4wn36 2444167 2444166 2026-08-04T16:22:39Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db1179 (T433990)', diff saved to https://phabricator.wikimedia.org/P95865 and previous config saved to /var/cache/conftool/dbconfig/20260804-162236-ladsgroup.json 2444167 wikitext text/x-wiki 278z8onpra2uwwhh0lhgbkrn5hq8orc 2444169 2444167 2026-08-04T16:23:50Z Stashbot 7414 rzl@deploy1003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply 2444169 wikitext text/x-wiki eie2vnbyn8aijd3qbv61cydbiylxigo 2444170 2444169 2026-08-04T16:24:01Z Stashbot 7414 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply 2444170 wikitext text/x-wiki tcnn7y6rnhs0wq2prxn0qi8fs3rxk0j 2444171 2444170 2026-08-04T16:24:28Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T433990)', diff saved to https://phabricator.wikimedia.org/P95866 and previous config saved to /var/cache/conftool/dbconfig/20260804-162424-ladsgroup.json 2444171 wikitext text/x-wiki 64qnqfaq4oiwugf4s2e8e43jpa6qfh9 2444172 2444171 2026-08-04T16:25:24Z Stashbot 7414 dzahn@cumin1003: START - Cookbook sre.gerrit.restart-gerrit Restarting Gerrit on gerrit2003 2444172 wikitext text/x-wiki nzk4d4q5fxteqeprj1ab97qyfsv5stz 2444173 2444172 2026-08-04T16:25:54Z Stashbot 7414 mutante: restarting gerrit - dropped outdated RSA host key 2444173 wikitext text/x-wiki ji3mbqhaj8a573kyxndagrw9ixao438 2444174 2444173 2026-08-04T16:26:40Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply logging and security config updates - bking@cumin2003 - T324335 2444174 wikitext text/x-wiki ohsfxzimemfcd4sgaee9s9jbjkby6wv 2444175 2444174 2026-08-04T16:26:52Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance 2444175 wikitext text/x-wiki ck3qnac1vosq5kthusqpw9rkynmyl4x 2444176 2444175 2026-08-04T16:27:20Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply 2444176 wikitext text/x-wiki oaxcc3ssq0qqcoy25bozdoqx811a6q0 2444177 2444176 2026-08-04T16:27:40Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db1179 (T433990)', diff saved to https://phabricator.wikimedia.org/P95867 and previous config saved to /var/cache/conftool/dbconfig/20260804-162736-ladsgroup.json 2444177 wikitext text/x-wiki gaxbg67pfyhcmot4vdjcu4c57luw51n 2444178 2444177 2026-08-04T16:27:58Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply 2444178 wikitext text/x-wiki tqaa9q9r4hdebn2k8yad1plufjbkyxk 2444179 2444178 2026-08-04T16:28:00Z Stashbot 7414 dzahn@cumin1003: END (PASS) - Cookbook sre.gerrit.restart-gerrit (exit_code=0) Restarting Gerrit on gerrit2003 2444179 wikitext text/x-wiki 6d241rlkv5kk8qen5uzpguk74fn09xr 2444180 2444179 2026-08-04T16:28:41Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=eqiad 2444180 wikitext text/x-wiki suh5cd3eu1230j898zzvyycglun611s 2444181 2444180 2026-08-04T16:28:59Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw 2444181 wikitext text/x-wiki sv1m7idrxhd4raa1lab9cj24x4no5v8 2444182 2444181 2026-08-04T16:29:01Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw 2444182 wikitext text/x-wiki jal97w4x817anqelp6xeq0x8d3hczk9 2444183 2444182 2026-08-04T16:29:03Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw 2444183 wikitext text/x-wiki 6h439c5sliui4iyd0yiukixzvq6p5rz 2444184 2444183 2026-08-04T16:32:29Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply 2444184 wikitext text/x-wiki gqj614m6v3hnb95hwh40ythjeu4udiq 2444185 2444184 2026-08-04T16:33:16Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply 2444185 wikitext text/x-wiki tns3nwlnl8g273ulvnp4c3s9bm0zsnx 2444189 2444185 2026-08-04T16:48:31Z Stashbot 7414 mutante: gerrit2003 - moving old apache logfiles older than 60 days from /var/log/apache2 to /srv/gerrit/site_path/review_site/logs/old/ 2444189 wikitext text/x-wiki iym7mjhpbsgq8der2hstt69vw9k7n41 2444192 2444189 2026-08-04T16:52:20Z Stashbot 7414 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2187.codfw.wmnet 2444192 wikitext text/x-wiki 4jdr0bcrkdiarl28moc2ndlu8gj0lij 2444193 2444192 2026-08-04T16:52:51Z Stashbot 7414 mutante: gerrit2003:/var/log/apache2# ln -s /srv/gerrit/site_path/review_site/logs/ gerrit 2444193 wikitext text/x-wiki g7ag1r0zvhxym6ig5rxr4iixoanh80u 2444194 2444193 2026-08-04T16:52:53Z Stashbot 7414 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2187.codfw.wmnet 2444194 wikitext text/x-wiki fkwrtwdue6rms7ev35crhu06l5z3x2l 2444195 2444194 2026-08-04T16:54:43Z Stashbot 7414 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker2187.codfw.wmnet with reason: Hardware issue 2444195 wikitext text/x-wiki 217ln8ve0j5uk81x09roqgqgo6pqrmt 2444196 2444195 2026-08-04T17:00:35Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.update-replication 2444196 wikitext text/x-wiki 7kb83qzkeq5f7y9ei0mekrze9evaznj 2444197 2444196 2026-08-04T17:00:47Z Stashbot 7414 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2444197 wikitext text/x-wiki h8gfog03ao9f7udgbbvfztn6fp5hf0z 2444198 2444197 2026-08-04T17:05:36Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Pick up new PHP production image 2444198 wikitext text/x-wiki c836n72uw5ga14v2uved40js15ul8eg 2444199 2444198 2026-08-04T17:28:07Z Stashbot 7414 aokoth@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on phab1005.eqiad.wmnet with reason: Puppet Failure 2444199 wikitext text/x-wiki bemb728ri1nrvs00wbrxke43ixjm71d 2444200 2444199 2026-08-04T17:33:48Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Pick up new PHP production image (duration: 28m 32s) 2444200 wikitext text/x-wiki t2bd4pcyyeduriao5f4q2lsluvbnql2 2444201 2444200 2026-08-04T17:34:03Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T433990)', diff saved to https://phabricator.wikimedia.org/P95868 and previous config saved to /var/cache/conftool/dbconfig/20260804-173359-ladsgroup.json 2444201 wikitext text/x-wiki pg7voikvq0itjx30slk57s0ycidknla 2444202 2444201 2026-08-04T17:44:47Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P95869 and previous config saved to /var/cache/conftool/dbconfig/20260804-174445-ladsgroup.json 2444202 wikitext text/x-wiki cn5k3ygopavu1fp73ssp7pfug9sghpr 2444203 2444202 2026-08-04T17:45:57Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Deploy helmfile-only msmtp timeout override cleanup - T383047 2444203 wikitext text/x-wiki ic5l8z6mm5fi10tbk4gl5n9x6gj2p7m 2444204 2444203 2026-08-04T17:46:59Z Stashbot 7414 swfrench@deploy1003: swfrench: Deploy helmfile-only msmtp timeout override cleanup - T383047 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2444204 wikitext text/x-wiki ajwfm0yeuh3ju3ugyl6fp313vaxlbuz 2444205 2444204 2026-08-04T17:48:13Z Stashbot 7414 swfrench@deploy1003: swfrench: Continuing with deployment 2444205 wikitext text/x-wiki gmuzzu73kc013362kxv5csaa34w8tr1 2444207 2444205 2026-08-04T17:50:03Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Deploy helmfile-only msmtp timeout override cleanup - T383047 (duration: 04m 05s) 2444207 wikitext text/x-wiki e4l7x2h1kefad2qqrgpqw6tpyuhdtuo 2444208 2444207 2026-08-04T17:55:22Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1154.eqiad.wmnet 2444208 wikitext text/x-wiki 1odldoid1n443paoa1s5bbu1grbm32m 2444209 2444208 2026-08-04T17:55:24Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1154.eqiad.wmnet 2444209 wikitext text/x-wiki nllnh86axgi701g3gxl9bqxoj0z4bdy 2444210 2444209 2026-08-04T17:55:27Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1154.eqiad.wmnet 2444210 wikitext text/x-wiki itffsmcf7brn3hljjbvqop23wb4mmro 2444211 2444210 2026-08-04T17:55:34Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P95870 and previous config saved to /var/cache/conftool/dbconfig/20260804-175531-ladsgroup.json 2444211 wikitext text/x-wiki nz7ro0f1fzgvcl1afpjx6fxzb6dssxg 2444213 2444211 2026-08-04T18:06:21Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T433990)', diff saved to https://phabricator.wikimedia.org/P95871 and previous config saved to /var/cache/conftool/dbconfig/20260804-180618-ladsgroup.json 2444213 wikitext text/x-wiki 7eo1lveogb4ag7mvoz01lhtnnase8k0 2444214 2444213 2026-08-04T18:06:36Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1203.eqiad.wmnet with reason: Maintenance 2444214 wikitext text/x-wiki 87gcz1xiznic9yhsuvxobujeyd985xa 2444215 2444214 2026-08-04T18:07:03Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2444215 wikitext text/x-wiki nf6wbgaev9jwguii0p7k2puzxhwv9g7 2444216 2444215 2026-08-04T18:07:25Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db1203 (T433990)', diff saved to https://phabricator.wikimedia.org/P95872 and previous config saved to /var/cache/conftool/dbconfig/20260804-180721-ladsgroup.json 2444216 wikitext text/x-wiki 9etbpqm4ujrge94teijsi4s91kacfrg 2444217 2444216 2026-08-04T18:08:13Z Stashbot 7414 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2444217 wikitext text/x-wiki 3smihwvzpejj5lhdjsa7btwwzitygfo 2444218 2444217 2026-08-04T18:10:43Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2444218 wikitext text/x-wiki icqo6g3hfjg5x6uz7czn55zw8hm4p6a 2444220 2444218 2026-08-04T18:13:42Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2444220 wikitext text/x-wiki j0c9l1ro2590vviynhyfbpxvf6tvrt1 2444221 2444220 2026-08-04T18:14:26Z Stashbot 7414 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2444221 wikitext text/x-wiki 4eo8aqeyvgl0klyugu1kiezcpturtrl 2444222 2444221 2026-08-04T18:14:27Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2444222 wikitext text/x-wiki pyff5r6vd6ma33vnso2g58t2dm026up 2444223 2444222 2026-08-04T18:18:27Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2444223 wikitext text/x-wiki kf14hzul4ejj31fk4fhhar5868z2lc2 2444224 2444223 2026-08-04T18:20:45Z Stashbot 7414 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2444224 wikitext text/x-wiki g5bhxs1byvpr7p74ltpekos77f3zn7q 2444230 2444224 2026-08-04T19:02:13Z Stashbot 7414 mutante: gerrit ssh -p 29418 gerrit.wikimedia.org gerrit index changes 1320979 2444230 wikitext text/x-wiki an7xmyy8d9aq1iu4nmwnzsxziwzy3kl 2444232 2444230 2026-08-04T19:27:41Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T433990)', diff saved to https://phabricator.wikimedia.org/P95874 and previous config saved to /var/cache/conftool/dbconfig/20260804-192738-ladsgroup.json 2444232 wikitext text/x-wiki frxde6gxkipzor09pqkximz88x39jrv 2444233 2444232 2026-08-04T19:38:27Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P95875 and previous config saved to /var/cache/conftool/dbconfig/20260804-193825-ladsgroup.json 2444233 wikitext text/x-wiki lrvwspit7e07u4escqojmg56wx6f1tc 2444234 2444233 2026-08-04T19:45:19Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host mc2046.codfw.wmnet with OS bookworm 2444234 wikitext text/x-wiki p9ri9x829itrt3o8gydwlk7is5x26ad 2444235 2444234 2026-08-04T19:45:49Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.move-vlan for host mc2046 2444235 wikitext text/x-wiki e12aswv2vlszhqa2fct6s1669czu6ku 2444236 2444235 2026-08-04T19:46:10Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2444236 wikitext text/x-wiki pb4px5et3akvdllytmmcjx4v3q5i94k 2444237 2444236 2026-08-04T19:49:13Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P95876 and previous config saved to /var/cache/conftool/dbconfig/20260804-194911-ladsgroup.json 2444237 wikitext text/x-wiki pnzi0guimg9vtwtk6wcfy4x30jhto0m 2444238 2444237 2026-08-04T19:50:29Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host mc2046 - jhancock@cumin2002" 2444238 wikitext text/x-wiki bh3rqzzqqtdeigl3mhojn6ladskgni9 2444239 2444238 2026-08-04T19:50:34Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host mc2046 - jhancock@cumin2002" 2444239 wikitext text/x-wiki jf6z57vxd1y9ldn7buu3vz4lviqp9df 2444240 2444239 2026-08-04T19:50:36Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2444240 wikitext text/x-wiki bfasjmhz92jd9w2prtb8tkgzj9vwziq 2444241 2444240 2026-08-04T19:50:37Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.wipe-cache mc2046.codfw.wmnet 120.16.192.10.in-addr.arpa 0.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2444241 wikitext text/x-wiki 5lc7kyrsszt2sfssqfd1s742lrhn2i4 2444242 2444241 2026-08-04T19:50:39Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mc2046.codfw.wmnet 120.16.192.10.in-addr.arpa 0.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2444242 wikitext text/x-wiki nxc1p06kfvux6o5d3jwz8l6gcpdy0df 2444243 2444242 2026-08-04T19:50:41Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2046 2444243 wikitext text/x-wiki i85wekgfbuiy2z7hy5t0sv36o0lnsyc 2444244 2444243 2026-08-04T19:50:59Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2046 2444244 wikitext text/x-wiki 5ru3ni3goyvekcybdva6mm8fpajepsj 2444245 2444244 2026-08-04T19:51:01Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host mc2046 2444245 wikitext text/x-wiki ff0lhvg5xahyr0hzcg3o8zsohly8yez 2444246 2444245 2026-08-04T20:00:01Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T433990)', diff saved to https://phabricator.wikimedia.org/P95877 and previous config saved to /var/cache/conftool/dbconfig/20260804-195957-ladsgroup.json 2444246 wikitext text/x-wiki sogr435lt5gyh5h8qoahghdko8ffion 2444247 2444246 2026-08-04T20:00:16Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1216.eqiad.wmnet with reason: Maintenance 2444247 wikitext text/x-wiki 18se7ocjjajkncmm6v82c8gspactkfl 2444251 2444247 2026-08-04T20:09:07Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage 2444251 wikitext text/x-wiki oi1t66oxrh3fjjb5ppy13ao4oq1xw3m 2444252 2444251 2026-08-04T20:13:08Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage 2444252 wikitext text/x-wiki cw9ac5e2syjjbrq9ym78rzonirylc1q 2444253 2444252 2026-08-04T20:15:35Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1313827|rest: Add test server option to REST Sandbox for Wikipedia projects (T408816)]] 2444253 wikitext text/x-wiki cv47olv99ibw38tlw72u30yw9ouuwrp 2444254 2444253 2026-08-04T20:21:24Z Stashbot 7414 samtar@deploy1003: samtar, kineticpelagic: Backport for [[gerrit:1313827|rest: Add test server option to REST Sandbox for Wikipedia projects (T408816)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2444254 wikitext text/x-wiki a56nfgp0vrgk2b6n12qhu7cd116cjec 2444255 2444254 2026-08-04T20:28:40Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2046.codfw.wmnet with OS bookworm 2444255 wikitext text/x-wiki q9rl9wt8sea2i4cxqxgwebe1t9z7w63 2444256 2444255 2026-08-04T20:33:42Z Stashbot 7414 samtar@deploy1003: samtar, kineticpelagic: Continuing with deployment 2444256 wikitext text/x-wiki r684jcxqyt8xi65i1s9qvekgg7bknuk 2444257 2444256 2026-08-04T20:40:15Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1313827|rest: Add test server option to REST Sandbox for Wikipedia projects (T408816)]] (duration: 24m 40s) 2444257 wikitext text/x-wiki jv81ib1dxcjn1guft7unaz7z67ejubg 2444258 2444257 2026-08-04T21:13:29Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance 2444258 wikitext text/x-wiki sqp2fp70ggsqrrf6shmigvs5krb8z3u 2444260 2444258 2026-08-04T22:23:00Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1237.eqiad.wmnet with reason: Maintenance 2444260 wikitext text/x-wiki bof0n10m4n63cxq5dvtyk4obabbkwvx 2444261 2444260 2026-08-04T22:23:49Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db1237 (T433990)', diff saved to https://phabricator.wikimedia.org/P95878 and previous config saved to /var/cache/conftool/dbconfig/20260804-222345-ladsgroup.json 2444261 wikitext text/x-wiki 0p9tizx9fj2iqnlwm6oyhz7mm7azyrb 2444263 2444261 2026-08-04T23:11:48Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1237 (T433990)', diff saved to https://phabricator.wikimedia.org/P95879 and previous config saved to /var/cache/conftool/dbconfig/20260804-231144-ladsgroup.json 2444263 wikitext text/x-wiki kdf8lhikl5shu7d4cszxdjij6eeshxw 2444265 2444263 2026-08-04T23:22:33Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1237', diff saved to https://phabricator.wikimedia.org/P95880 and previous config saved to /var/cache/conftool/dbconfig/20260804-232230-ladsgroup.json 2444265 wikitext text/x-wiki itaa1gle66n2eh4ma7qauo6qlp6hbbw 2444266 2444265 2026-08-04T23:33:19Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1237', diff saved to https://phabricator.wikimedia.org/P95881 and previous config saved to /var/cache/conftool/dbconfig/20260804-233317-ladsgroup.json 2444266 wikitext text/x-wiki kx9wvtdmqchbi667mxw9n395abtmben 2444267 2444266 2026-08-04T23:44:08Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1237 (T433990)', diff saved to https://phabricator.wikimedia.org/P95882 and previous config saved to /var/cache/conftool/dbconfig/20260804-234405-ladsgroup.json 2444267 wikitext text/x-wiki sxuhkjm0ke6613qsmzsuyg2qvyyaido 2444268 2444267 2026-08-04T23:44:23Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1264.eqiad.wmnet with reason: Maintenance 2444268 wikitext text/x-wiki 433plg5fgpt6mr8asshdgetwor3h68a 2444269 2444268 2026-08-04T23:45:11Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db1264 (T433990)', diff saved to https://phabricator.wikimedia.org/P95883 and previous config saved to /var/cache/conftool/dbconfig/20260804-234508-ladsgroup.json 2444269 wikitext text/x-wiki 0js6lf9hg5mgl7og3pznse3bex0gayg 2444272 2444269 2026-08-05T00:01:52Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 (T433990)', diff saved to https://phabricator.wikimedia.org/P95884 and previous config saved to /var/cache/conftool/dbconfig/20260805-000148-ladsgroup.json 2444272 wikitext text/x-wiki 6d1hjxya511h8sw802o5iakjn7fq9tg 2444274 2444272 2026-08-05T00:12:38Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P95885 and previous config saved to /var/cache/conftool/dbconfig/20260805-001235-ladsgroup.json 2444274 wikitext text/x-wiki bzkgizhnb1o22hdv9dxaeof6udxykwx 2444275 2444274 2026-08-05T00:23:24Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P95886 and previous config saved to /var/cache/conftool/dbconfig/20260805-002322-ladsgroup.json 2444275 wikitext text/x-wiki kmkfh1lhmmedhhpp8k5ry767upx9lh8 2444276 2444275 2026-08-05T00:34:12Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 (T433990)', diff saved to https://phabricator.wikimedia.org/P95887 and previous config saved to /var/cache/conftool/dbconfig/20260805-003408-ladsgroup.json 2444276 wikitext text/x-wiki raxv572s076ii6kcikn3bgl8949eo8e 2444277 2444276 2026-08-05T00:34:26Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance 2444277 wikitext text/x-wiki 7mad6wstcevlcefsiur9l996fbszcnh 2444280 2444277 2026-08-05T01:29:44Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2186.codfw.wmnet with reason: Maintenance 2444280 wikitext text/x-wiki pe5udjkwsuvwcmjrclljgfm5k7ovtg1 2444281 2444280 2026-08-05T01:30:32Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2186 (T433990)', diff saved to https://phabricator.wikimedia.org/P95888 and previous config saved to /var/cache/conftool/dbconfig/20260805-013029-ladsgroup.json 2444281 wikitext text/x-wiki b9ert71k9ijq8zny5s17sihvjlnlkhw 2444286 2444281 2026-08-05T02:00:54Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 (T433990)', diff saved to https://phabricator.wikimedia.org/P95889 and previous config saved to /var/cache/conftool/dbconfig/20260805-020051-ladsgroup.json 2444286 wikitext text/x-wiki 8yyhnk78vcsbbvud2ogya2dnf3uelo6 2444287 2444286 2026-08-05T02:01:12Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2444287 wikitext text/x-wiki o147m20qoa7uuag9ffclp1js7qa7rh2 2444288 2444287 2026-08-05T02:07:50Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 37s) 2444288 wikitext text/x-wiki 02ramrehuzpylx0ob7ci8dgh8m1njv0 2444289 2444288 2026-08-05T02:11:39Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P95890 and previous config saved to /var/cache/conftool/dbconfig/20260805-021137-ladsgroup.json 2444289 wikitext text/x-wiki 7l5e4vw01u18m7yvnsvexvk9649qonb 2444291 2444289 2026-08-05T02:22:25Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P95891 and previous config saved to /var/cache/conftool/dbconfig/20260805-022223-ladsgroup.json 2444291 wikitext text/x-wiki 86m5c0yt5b48seuhua0acz1x7cmql3g 2444292 2444291 2026-08-05T02:33:13Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 (T433990)', diff saved to https://phabricator.wikimedia.org/P95892 and previous config saved to /var/cache/conftool/dbconfig/20260805-023310-ladsgroup.json 2444292 wikitext text/x-wiki hfv5zl4lw23n8l80s42digurv46hljq 2444293 2444292 2026-08-05T02:33:28Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2196.codfw.wmnet with reason: Maintenance 2444293 wikitext text/x-wiki rg8781xn7n2s9jc7p6kf81b574yg8dn 2444294 2444293 2026-08-05T02:34:16Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2196 (T433990)', diff saved to https://phabricator.wikimedia.org/P95893 and previous config saved to /var/cache/conftool/dbconfig/20260805-023413-ladsgroup.json 2444294 wikitext text/x-wiki 9qbqvk3yn9hwwt197v0xf7pumrrd8ke 2444295 2444294 2026-08-05T03:08:19Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 (T433990)', diff saved to https://phabricator.wikimedia.org/P95894 and previous config saved to /var/cache/conftool/dbconfig/20260805-030815-ladsgroup.json 2444295 wikitext text/x-wiki 33l3nimwlq9izn2imny3rixcqmrch03 2444296 2444295 2026-08-05T03:19:04Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P95895 and previous config saved to /var/cache/conftool/dbconfig/20260805-031902-ladsgroup.json 2444296 wikitext text/x-wiki 20xla6i11z9ue9uwljgl72uocmy76uo 2444297 2444296 2026-08-05T03:29:51Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P95896 and previous config saved to /var/cache/conftool/dbconfig/20260805-032948-ladsgroup.json 2444297 wikitext text/x-wiki oq7cuijud7cj2z5iegqwi79460gf0wy 2444298 2444297 2026-08-05T03:40:40Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 (T433990)', diff saved to https://phabricator.wikimedia.org/P95897 and previous config saved to /var/cache/conftool/dbconfig/20260805-034036-ladsgroup.json 2444298 wikitext text/x-wiki 4oxjnvhzesxv7oxn3e33mhjasqrwlvg 2444299 2444298 2026-08-05T03:40:54Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance 2444299 wikitext text/x-wiki rswchiwgdr4kuzuxcluyd8hywjx1mn9 2444301 2444299 2026-08-05T04:30:42Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2201.codfw.wmnet with reason: Maintenance 2444301 wikitext text/x-wiki oez9knh1i371o6lefb217lif59pbkbw 2444303 2444301 2026-08-05T05:18:54Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2215.codfw.wmnet with reason: Maintenance 2444303 wikitext text/x-wiki nqzj6fp092s9zr74c16zwaxthci381z 2444304 2444303 2026-08-05T05:19:42Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2215 (T433990)', diff saved to https://phabricator.wikimedia.org/P95898 and previous config saved to /var/cache/conftool/dbconfig/20260805-051939-ladsgroup.json 2444304 wikitext text/x-wiki c66fyo1u54djoav2idnmzn5qwztbivo 2444305 2444304 2026-08-05T05:49:21Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 (T433990)', diff saved to https://phabricator.wikimedia.org/P95899 and previous config saved to /var/cache/conftool/dbconfig/20260805-054918-ladsgroup.json 2444305 wikitext text/x-wiki nijg7sy81ch6fsagdiumtat4sxnwl29 2444306 2444305 2026-08-05T06:00:07Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P95900 and previous config saved to /var/cache/conftool/dbconfig/20260805-060004-ladsgroup.json 2444306 wikitext text/x-wiki 6bo38enq109bdqmszm8vvsamz0xp3eb 2444307 2444306 2026-08-05T06:10:53Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P95901 and previous config saved to /var/cache/conftool/dbconfig/20260805-061051-ladsgroup.json 2444307 wikitext text/x-wiki 9s2xt6tw4t66n3hd9tms6aveze6bi4o 2444308 2444307 2026-08-05T06:21:41Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 (T433990)', diff saved to https://phabricator.wikimedia.org/P95902 and previous config saved to /var/cache/conftool/dbconfig/20260805-062137-ladsgroup.json 2444308 wikitext text/x-wiki k8hrueydpd48fkubrsf1m4zl8ttd0vv 2444309 2444308 2026-08-05T06:21:56Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2231.codfw.wmnet with reason: Maintenance 2444309 wikitext text/x-wiki ohjj6iu627c7hjqshox9flonv5vyftg 2444310 2444309 2026-08-05T06:22:44Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2231 (T433990)', diff saved to https://phabricator.wikimedia.org/P95903 and previous config saved to /var/cache/conftool/dbconfig/20260805-062240-ladsgroup.json 2444310 wikitext text/x-wiki qi2vkhc5i6gk54oy4ns73g5y3pmza7g 2444311 2444310 2026-08-05T06:45:02Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2444311 wikitext text/x-wiki 5sqdope1b4iwyujuh9raq7lqv4alup9 2444312 2444311 2026-08-05T06:52:10Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 (T433990)', diff saved to https://phabricator.wikimedia.org/P95904 and previous config saved to /var/cache/conftool/dbconfig/20260805-065206-ladsgroup.json 2444312 wikitext text/x-wiki 0tvdhfxbuspzoncv1xjbepz8pm2qusv 2444313 2444312 2026-08-05T07:02:55Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P95905 and previous config saved to /var/cache/conftool/dbconfig/20260805-070253-ladsgroup.json 2444313 wikitext text/x-wiki 639k1yuvsgipfurpn5cozp55h85w6m7 2444314 2444313 2026-08-05T07:13:42Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P95906 and previous config saved to /var/cache/conftool/dbconfig/20260805-071340-ladsgroup.json 2444314 wikitext text/x-wiki s5uz1uittaxrwrk0bn32m0km2k3mcni 2444315 2444314 2026-08-05T07:19:04Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2444315 wikitext text/x-wiki mdttqkvytxnguvmuafxqopcdlpat8ov 2444316 2444315 2026-08-05T07:21:14Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2444316 wikitext text/x-wiki lsihcazkydr6a1uulfm3jgtse6ja44m 2444317 2444316 2026-08-05T07:24:30Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 (T433990)', diff saved to https://phabricator.wikimedia.org/P95908 and previous config saved to /var/cache/conftool/dbconfig/20260805-072426-ladsgroup.json 2444317 wikitext text/x-wiki mcqexmeiufehj2gulims4n9nopo5vth 2444318 2444317 2026-08-05T07:24:45Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2249.codfw.wmnet with reason: Maintenance 2444318 wikitext text/x-wiki 47ckmwwbbbtrcta0dgfmrn6fcynrqy0 2444319 2444318 2026-08-05T07:25:33Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2249 (T433990)', diff saved to https://phabricator.wikimedia.org/P95909 and previous config saved to /var/cache/conftool/dbconfig/20260805-072529-ladsgroup.json 2444319 wikitext text/x-wiki 1hrj4t5cfb5qj42owfv4trmtkrgs13w 2444320 2444319 2026-08-05T07:53:03Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet,service=s1 2444320 wikitext text/x-wiki emekk99q9ca6grksurm7vmrofjdzypz 2444321 2444320 2026-08-05T07:53:17Z Stashbot 7414 marostegui: Depool clouddb1013:s1 T409557 2444321 wikitext text/x-wiki kwrljavc5kdt5i6juuhaqn7no5m33d1 2444323 2444321 2026-08-05T07:53:46Z Stashbot 7414 marostegui: Depool clouddb1013:s1 T434048 2444323 wikitext text/x-wiki ffq2v38uhpitf2uv6j1df3lg3cqf9xi 2444324 2444323 2026-08-05T07:54:21Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet,service=s2 2444324 wikitext text/x-wiki i2u1fq1gp7av41d36jxww7d4jy2lpwa 2444325 2444324 2026-08-05T07:54:24Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet,service=s7 2444325 wikitext text/x-wiki 6qi0mf2bl2l2mvkmi2brj2cbdz5czll 2444326 2444325 2026-08-05T07:54:35Z Stashbot 7414 marostegui: Depool clouddb1014 (s2,s7) T434048 2444326 wikitext text/x-wiki dzaskmenklt8ztu5p9he6etife1xnm4 2444328 2444326 2026-08-05T07:56:53Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 (T433990)', diff saved to https://phabricator.wikimedia.org/P95910 and previous config saved to /var/cache/conftool/dbconfig/20260805-075650-ladsgroup.json 2444328 wikitext text/x-wiki syaikiql7heg53u5uuoz55q4l7bdkr1 2444329 2444328 2026-08-05T07:57:43Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s4 2444329 wikitext text/x-wiki 5iu66ux4kxhwku9va4teljcuwpv8pdf 2444330 2444329 2026-08-05T07:57:47Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s6 2444330 wikitext text/x-wiki bn5d9mwa3ubd73iqk92l5515liz987x 2444331 2444330 2026-08-05T07:57:58Z Stashbot 7414 marostegui: Depool clouddb1015 (s4,s6) T434048 2444331 wikitext text/x-wiki bqujn5lzn1ih0bgg20ld2ugq78l90aa 2444332 2444331 2026-08-05T07:59:04Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1016.eqiad.wmnet,service=s5 2444332 wikitext text/x-wiki 6shktux5ty2wsapgllh0cw5ggr89gu8 2444333 2444332 2026-08-05T07:59:08Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1016.eqiad.wmnet,service=s8 2444333 wikitext text/x-wiki 838dh6059vpkny9mwn7d374ccs13lku 2444334 2444333 2026-08-05T07:59:21Z Stashbot 7414 marostegui: Depool clouddb1016 (s5,s8) T434048 2444334 wikitext text/x-wiki 61pyt2ut5keuypm526f993jffmkwgp1 2444335 2444334 2026-08-05T08:01:25Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s1 2444335 wikitext text/x-wiki cnnzfcml4musp1spi5279vvdy1kq1r3 2444336 2444335 2026-08-05T08:01:33Z Stashbot 7414 marostegui: Depool clouddb1017 (s1) T434048 2444336 wikitext text/x-wiki fr3u8ymly9y5epj6sy58ew0ldn022kc 2444337 2444336 2026-08-05T08:01:47Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1018.eqiad.wmnet,service=s2 2444337 wikitext text/x-wiki dsqsa4yhzrnpos1b0ibk264qb9sml9i 2444338 2444337 2026-08-05T08:01:51Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1018.eqiad.wmnet,service=s7 2444338 wikitext text/x-wiki d9mfj3zqx7rpfx513hc6mtth4cyy8wn 2444339 2444338 2026-08-05T08:02:01Z Stashbot 7414 marostegui: Depool clouddb1018 (s2,s7) T434048 2444339 wikitext text/x-wiki 6u5fp44npo9qouliq68xsbg22993pq0 2444340 2444339 2026-08-05T08:02:11Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet,service=s5 2444340 wikitext text/x-wiki 0rmunhqh2d01s0ba1poi8na6jl4f7m6 2444341 2444340 2026-08-05T08:02:15Z Stashbot 7414 marostegui@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet,service=s8 2444341 wikitext text/x-wiki qzm9d6xlr9shfyvx519qnnwuw9py64w 2444342 2444341 2026-08-05T08:02:23Z Stashbot 7414 marostegui: Depool clouddb1020 (s5,s8) T434048 2444342 wikitext text/x-wiki moglsl0ga0r6wvw7pyzmcm98nysukgh 2444343 2444342 2026-08-05T08:05:51Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1150.eqiad.wmnet 2444343 wikitext text/x-wiki 9iedk9pm3n2iylaaqtm5jty4nfde59z 2444344 2444343 2026-08-05T08:07:39Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P95911 and previous config saved to /var/cache/conftool/dbconfig/20260805-080737-ladsgroup.json 2444344 wikitext text/x-wiki 76v3acovxnkafp9sc500km8ybb723jv 2444345 2444344 2026-08-05T08:11:03Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.dns.netbox 2444345 wikitext text/x-wiki 2j1mtl36d8j97nicmbjp8lm75v3sq3x 2444346 2444345 2026-08-05T08:14:11Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1155.eqiad.wmnet 2444346 wikitext text/x-wiki mbysfz4i6j4csw4c07br8jg7hsw6gla 2444347 2444346 2026-08-05T08:14:15Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1155.eqiad.wmnet 2444347 wikitext text/x-wiki 2vka5629u1xexb3x17u5402navkwa2a 2444348 2444347 2026-08-05T08:14:51Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1155.eqiad.wmnet 2444348 wikitext text/x-wiki nhcab6wqbqlfb448r3r0j5wnmc2uxxd 2444349 2444348 2026-08-05T08:15:08Z Stashbot 7414 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.14 refs T430833 2444349 wikitext text/x-wiki bmqcmp2hdjboafdumpg8n7hhrz51ahd 2444350 2444349 2026-08-05T08:15:43Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1150.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003" 2444350 wikitext text/x-wiki 8g4qsmaq60h7d12se83ob6b6xn0zztx 2444351 2444350 2026-08-05T08:17:06Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1150.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003" 2444351 wikitext text/x-wiki 7210c9vnfja2grrjl9u2jb1fcjb4do4 2444352 2444351 2026-08-05T08:17:08Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2444352 wikitext text/x-wiki abomw6dyldvtb3f2ue3tjvb4jmpujek 2444353 2444352 2026-08-05T08:17:10Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1150.eqiad.wmnet 2444353 wikitext text/x-wiki 6dxr8oypn04bxgka5w353ny52hxlutr 2444354 2444353 2026-08-05T08:17:42Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1171.eqiad.wmnet 2444354 wikitext text/x-wiki 6psd1aznogvfbwtfdax96sjjf8er0ut 2444355 2444354 2026-08-05T08:18:25Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P95912 and previous config saved to /var/cache/conftool/dbconfig/20260805-081823-ladsgroup.json 2444355 wikitext text/x-wiki m0jd1dxlog6uebjm5tu6yrqog2pda63 2444356 2444355 2026-08-05T08:22:48Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.dns.netbox 2444356 wikitext text/x-wiki rsacdl9ivrn6h4t0fnoq6jnex44cb5m 2444357 2444356 2026-08-05T08:27:41Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1171.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003" 2444357 wikitext text/x-wiki bnst7kg0wy0bza47ti7tvx8g0zopiez 2444358 2444357 2026-08-05T08:29:12Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 (T433990)', diff saved to https://phabricator.wikimedia.org/P95913 and previous config saved to /var/cache/conftool/dbconfig/20260805-082908-ladsgroup.json 2444358 wikitext text/x-wiki 4jpet5t9z00puwoyhecn1r7uk6xjezx 2444359 2444358 2026-08-05T08:29:55Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1171.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003" 2444359 wikitext text/x-wiki bvevcc0iemautkf4l5c8qb10z025ea0 2444360 2444359 2026-08-05T08:29:57Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2444360 wikitext text/x-wiki gvwv30t7gy4ktmg0a6wlop5l8yeudmb 2444361 2444360 2026-08-05T08:29:59Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1171.eqiad.wmnet 2444361 wikitext text/x-wiki q43ci6gov8ei4n8vy0yyx5weduehhug 2444362 2444361 2026-08-05T08:37:56Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1155.eqiad.wmnet with OS trixie 2444362 wikitext text/x-wiki c6j93j3sd8z6v5flh9gpslr9qnht64f 2444363 2444362 2026-08-05T08:38:25Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1155 2444363 wikitext text/x-wiki sxdh6kuztyr82jia66nhup8l6h9pfji 2444366 2444363 2026-08-05T09:05:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Cloning 2444366 wikitext text/x-wiki oiatlf7vehsub0pnvkptz2o90x0oy9b 2444367 2444366 2026-08-05T09:05:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2444367 wikitext text/x-wiki tjwdlmzde1yzaxf0redtr7rcd7cnrh1 2444368 2444367 2026-08-05T09:06:36Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2444368 wikitext text/x-wiki a3jx5tf9avjotqycsx9so02emt9el6z 2444369 2444368 2026-08-05T09:06:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Cloning 2444369 wikitext text/x-wiki 6yqn9tgfuwbhj79cnp7pm3njpa5tbg1 2444370 2444369 2026-08-05T09:08:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444370 wikitext text/x-wiki 2eagvgmboz2fdjqvd6wmut01rxbfbbl 2444371 2444370 2026-08-05T09:08:45Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) 2444371 wikitext text/x-wiki l85dre4yc5e3hxoimqfkfqjxdixon87 2444372 2444371 2026-08-05T09:09:42Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.dns.netbox 2444372 wikitext text/x-wiki e8rondafm7twv6shblamxjub9bpltw8 2444373 2444372 2026-08-05T09:09:44Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2251.codfw.wmnet,db1152.eqiad.wmnet with reason: cloning 2444373 wikitext text/x-wiki 7j4snoak1i83m68hgu9h9m40i482s82 2444374 2444373 2026-08-05T09:10:16Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444374 wikitext text/x-wiki qjhngpmdhvlmry31oj8jx5wfpxsx87v 2444375 2444374 2026-08-05T09:10:27Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) 2444375 wikitext text/x-wiki lgc22axq74lri0sfm6s2ce7g8e48a8r 2444376 2444375 2026-08-05T09:14:04Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1155 - jayme@cumin1003" 2444376 wikitext text/x-wiki 6q9rhemw58df7o05zqj2uts5xhss8wt 2444377 2444376 2026-08-05T09:14:09Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1155 - jayme@cumin1003" 2444377 wikitext text/x-wiki 49r4wibdr2t6j5gc6w61gd78yl6pkzh 2444378 2444377 2026-08-05T09:14:11Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2444378 wikitext text/x-wiki e03hrgidljnhcqfecbdanc1yuues93f 2444379 2444378 2026-08-05T09:14:13Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1155.eqiad.wmnet 109.32.64.10.in-addr.arpa 9.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2444379 wikitext text/x-wiki c02dchruys16mmo731dg8u596hver4l 2444380 2444379 2026-08-05T09:14:16Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1155.eqiad.wmnet 109.32.64.10.in-addr.arpa 9.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2444380 wikitext text/x-wiki fpq4ivdzi1cyg9l4ae5e9bzig29zze1 2444381 2444380 2026-08-05T09:14:18Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1155 2444381 wikitext text/x-wiki a6km172vbat1n9vyx05nme2zqzuja4p 2444382 2444381 2026-08-05T09:17:17Z Stashbot 7414 XioNoX: push pfw policies - T434038 2444382 wikitext text/x-wiki 6ryv6m5fwzswm8aufs4gntgigzrawis 2444383 2444382 2026-08-05T09:20:39Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444383 wikitext text/x-wiki hjsyxbroil73pbpvbasmv7fx8l3xuup 2444384 2444383 2026-08-05T09:20:54Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444384 wikitext text/x-wiki 4yt02huw5rtlbajn42hw62owxno5keo 2444385 2444384 2026-08-05T09:22:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444385 wikitext text/x-wiki gfvpccfd45rirdbhlkfkq4mgapwd88g 2444386 2444385 2026-08-05T09:22:44Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2444386 wikitext text/x-wiki et4iw2eoty4dotoja75mdkztdwbesj4 2444387 2444386 2026-08-05T09:22:48Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2444387 wikitext text/x-wiki dml63vrxdeo5osmcmq10tlaf5ysi8fc 2444388 2444387 2026-08-05T09:22:51Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444388 wikitext text/x-wiki tbgj1i2gqaatmefb3hdeopkb9e0uyqi 2444389 2444388 2026-08-05T09:23:13Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2444389 wikitext text/x-wiki skgo5toxi0qewzw5hjoftk5xkmc0hpl 2444390 2444389 2026-08-05T09:23:16Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2444390 wikitext text/x-wiki 06n3u2zy272w4khgkhe7hfhyvyutdiq 2444391 2444390 2026-08-05T09:23:31Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444391 wikitext text/x-wiki or1ewzobp1wx7ryd7ecrto3iv872u13 2444392 2444391 2026-08-05T09:24:41Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) 2444392 wikitext text/x-wiki 86bgpn6mfesblystogo02ywb7a6eon9 2444393 2444392 2026-08-05T09:24:48Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2444393 wikitext text/x-wiki 3ygxb5fla8df7na41nmupa6e6e6l2dw 2444394 2444393 2026-08-05T09:25:07Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2444394 wikitext text/x-wiki l18ifl82euciq3hcfkj4r2dpsvb074v 2444395 2444394 2026-08-05T09:27:04Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1155 2444395 wikitext text/x-wiki 8wftm7cntvxdgrmcewpu19sysrjp2tu 2444396 2444395 2026-08-05T09:27:07Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1155 2444396 wikitext text/x-wiki n8o731fcuadlfnlrnwem85co1lft4qw 2444397 2444396 2026-08-05T09:31:41Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444397 wikitext text/x-wiki th0yyeqm1u68kphtvuq4bxgbr53c3qh 2444398 2444397 2026-08-05T09:31:55Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444398 wikitext text/x-wiki caqk7nnx1cf1rdwbdrdnqynk0xyzbiu 2444399 2444398 2026-08-05T09:32:14Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444399 wikitext text/x-wiki e7wzky6ya9jr8e6opjnfi2yl6wo28yu 2444400 2444399 2026-08-05T09:32:25Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444400 wikitext text/x-wiki 0g844yz28ho8qbranqfhs9g6nqjcinc 2444401 2444400 2026-08-05T09:41:00Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1155.eqiad.wmnet with reason: host reimage 2444401 wikitext text/x-wiki ia86lhg5omrlr4zqre5zup0m76se6xo 2444402 2444401 2026-08-05T09:43:59Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1155.eqiad.wmnet with reason: host reimage 2444402 wikitext text/x-wiki jcg7bdten4fm54a8ail94gm3sntib9f 2444403 2444402 2026-08-05T09:44:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1152: after cloning 2444403 wikitext text/x-wiki ityik5i92fwnwms0grc0xvhhmtxhhta 2444404 2444403 2026-08-05T09:44:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2444404 wikitext text/x-wiki fgvp3ttfu7fec0y34tdvpbrefcfm1ua 2444405 2444404 2026-08-05T09:44:04Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) 2444405 wikitext text/x-wiki smzk4ngykikl25uie78iobu4x2v12na 2444406 2444405 2026-08-05T09:44:07Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: after cloning 2444406 wikitext text/x-wiki c204yo4bc203exefb1kaqo7r1hnuqii 2444407 2444406 2026-08-05T09:52:14Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Pool back ms1', diff saved to https://phabricator.wikimedia.org/P95918 and previous config saved to /var/cache/conftool/dbconfig/20260805-095212-marostegui.json 2444407 wikitext text/x-wiki gixcxr0n1dstsmdx7uzkoli8r2xw7wi 2444408 2444407 2026-08-05T10:04:01Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1155.eqiad.wmnet with OS trixie 2444408 wikitext text/x-wiki cttrnx8g61tnqecysrgax4maxm87pf6 2444409 2444408 2026-08-05T10:04:13Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444409 wikitext text/x-wiki s04e63qbzlys7wgt5wgvem3uyw02jjn 2444410 2444409 2026-08-05T10:04:23Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444410 wikitext text/x-wiki ojbir3gztogt5rk6rawkb45cku9rdvk 2444411 2444410 2026-08-05T10:04:42Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444411 wikitext text/x-wiki fesihwo4y9xqsbyln8ju9wnqk3rlreh 2444412 2444411 2026-08-05T10:04:52Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444412 wikitext text/x-wiki ghf7330r4l9dg599bznjrdekpa4dg6x 2444413 2444412 2026-08-05T10:05:04Z Stashbot 7414 aikochou@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: sync 2444413 wikitext text/x-wiki 6p6ymauow2l9id0gmaybu5k2ynvy44e 2444414 2444413 2026-08-05T10:05:11Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444414 wikitext text/x-wiki ih25f5tnc9dipqeivdp33j30g36jc5c 2444415 2444414 2026-08-05T10:05:25Z Stashbot 7414 aikochou@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync 2444415 wikitext text/x-wiki jsgfz6rhv3jkcjk7wbfrrs0e72re87h 2444416 2444415 2026-08-05T10:05:42Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2444416 wikitext text/x-wiki h3ti830lnboxy5q0gpxuj02jrx61h1b 2444417 2444416 2026-08-05T10:07:11Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444417 wikitext text/x-wiki m0khe422oy6u1gyj5nqggvyuglo2bix 2444418 2444417 2026-08-05T10:07:26Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444418 wikitext text/x-wiki 235x3ljejuwwszmbztazpdaqv032gji 2444419 2444418 2026-08-05T10:09:15Z Stashbot 7414 aikochou@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: sync 2444419 wikitext text/x-wiki gamsn4y9dmzv5jbs327wxkw1cqonm17 2444420 2444419 2026-08-05T10:09:25Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444420 wikitext text/x-wiki 0p3ji5rv7w4cxphq4opm4hnzcy5lrxg 2444421 2444420 2026-08-05T10:09:38Z Stashbot 7414 aikochou@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: sync 2444421 wikitext text/x-wiki pys57kpi88nktqn0l4ec0u9hfekltlw 2444422 2444421 2026-08-05T10:09:53Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2444422 wikitext text/x-wiki plbpa7b1zqgr539ddwep3nyj7m4elsx 2444423 2444422 2026-08-05T10:10:15Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444423 wikitext text/x-wiki dq46ffjdavaketsnx60odzurlkvperz 2444424 2444423 2026-08-05T10:10:40Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2444424 wikitext text/x-wiki bceslz23o09uxp58fwq8bg39zo5gtje 2444425 2444424 2026-08-05T10:11:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444425 wikitext text/x-wiki 9ncwchvk70x79ucenu6aulsb8imduz7 2444426 2444425 2026-08-05T10:11:46Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444426 wikitext text/x-wiki hi14qc1e68bkw5y3nhzkoa2vt702m91 2444427 2444426 2026-08-05T10:14:23Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444427 wikitext text/x-wiki 2qd1tvdo83qyyw13gqqregkejvs05vg 2444428 2444427 2026-08-05T10:14:42Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2444428 wikitext text/x-wiki 0wk1d36kdcwnwibdu2vyqs1gl8oaz9s 2444429 2444428 2026-08-05T10:14:58Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1155.eqiad.wmnet 2444429 wikitext text/x-wiki 41h44us6j0z95r7jvp7xfwdt5e5ciub 2444430 2444429 2026-08-05T10:15:01Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1155.eqiad.wmnet 2444430 wikitext text/x-wiki hftb5e6qo82nr0xkyzt5sjvsqec1vyu 2444431 2444430 2026-08-05T10:15:03Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1155.eqiad.wmnet 2444431 wikitext text/x-wiki 8rxjt8gr7ik27ip9f05h077vvro3tx4 2444432 2444431 2026-08-05T10:16:46Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444432 wikitext text/x-wiki qoc771101x76pid39gp8takgvbc0u66 2444433 2444432 2026-08-05T10:17:06Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) 2444433 wikitext text/x-wiki b4rinaa5eiwncjvqqd99acnsq1ovolb 2444434 2444433 2026-08-05T10:22:57Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444434 wikitext text/x-wiki ccbp5h45wdux1fi2qs62opbvl42t35v 2444435 2444434 2026-08-05T10:23:11Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444435 wikitext text/x-wiki dedu328tew5vvfqnkn03534l3gpcg94 2444436 2444435 2026-08-05T10:23:20Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444436 wikitext text/x-wiki n4is1sbb1bdsys3ee0usmas9b1letnr 2444437 2444436 2026-08-05T10:23:32Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444437 wikitext text/x-wiki i8jg7kz3xn9ex785may3ag0hchlpio7 2444438 2444437 2026-08-05T10:23:38Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444438 wikitext text/x-wiki d9rt1fgbnwfh06p1kbtl74zu2gp0olk 2444439 2444438 2026-08-05T10:23:51Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444439 wikitext text/x-wiki cugsps87kuyi47ci2b1g1jyfz1206az 2444440 2444439 2026-08-05T10:23:57Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444440 wikitext text/x-wiki ozsltg55g7fxnr7d8b05qsgzhiyis7d 2444441 2444440 2026-08-05T10:24:09Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444441 wikitext text/x-wiki 7cgz1sv6pn5f4pmrjmzwlelismam3i4 2444442 2444441 2026-08-05T10:24:29Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444442 wikitext text/x-wiki 5fl05hhnqdm55bdamfbioeiqqyrfh1z 2444443 2444442 2026-08-05T10:24:39Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444443 wikitext text/x-wiki 4w1eqsqo0p38tug18ilujgn3sl92dma 2444444 2444443 2026-08-05T10:24:46Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444444 wikitext text/x-wiki mrpeiqjb1swr9fos6nlaor886x9xuz7 2444445 2444444 2026-08-05T10:24:55Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444445 wikitext text/x-wiki naf1tzvwpsczton9vkhtjvls7b2i2uf 2444447 2444445 2026-08-05T11:04:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2444447 wikitext text/x-wiki sdsiizdfbwd47ks0esre1eonaax8h2d 2444448 2444447 2026-08-05T11:04:45Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2444448 wikitext text/x-wiki trwwwydp2fgk6f6mjh71nsjuucs9j9m 2444449 2444448 2026-08-05T11:05:34Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444449 wikitext text/x-wiki 2wbeblo0u3dfjioee1xbafnqhcxyjru 2444450 2444449 2026-08-05T11:05:49Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444450 wikitext text/x-wiki jeivltp8dgfny4mn5zxbqjuoo5wi3l3 2444451 2444450 2026-08-05T11:06:02Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444451 wikitext text/x-wiki d96t2jfmhvia7zqi4rt1uodsh4gliil 2444452 2444451 2026-08-05T11:06:12Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444452 wikitext text/x-wiki k2n40m24yoqa6ihhe5ge08f6zzqcu11 2444453 2444452 2026-08-05T11:06:28Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444453 wikitext text/x-wiki a1xyi0ko15zd35jxrt6cez39ocbimqh 2444454 2444453 2026-08-05T11:06:43Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444454 wikitext text/x-wiki aq8u67dsvfinve62x1ax9zsipbhs27m 2444455 2444454 2026-08-05T11:07:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444455 wikitext text/x-wiki 8no4zc6ovcwg6q8ka64cq56ixf5eb2g 2444456 2444455 2026-08-05T11:07:14Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444456 wikitext text/x-wiki jei1oxfwepxc7aju1l5f1u7wj44u96m 2444457 2444456 2026-08-05T11:08:01Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444457 wikitext text/x-wiki ihj22c9be7u63pmyc186lodeems9ctt 2444458 2444457 2026-08-05T11:08:13Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444458 wikitext text/x-wiki 1l9br8d2ow558necngqklaem2kg9fa9 2444459 2444458 2026-08-05T11:08:58Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444459 wikitext text/x-wiki qkc1onosqixpn2vvv84bzl8aprpvdrc 2444460 2444459 2026-08-05T11:09:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444460 wikitext text/x-wiki qfhw7hcysu7pzwcsmp3u0pxfv7kn5gm 2444461 2444460 2026-08-05T11:10:30Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444461 wikitext text/x-wiki bwf6d4di1dqtx5mgiln9dmahp4u1klb 2444462 2444461 2026-08-05T11:10:42Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444462 wikitext text/x-wiki 3glnu9n2bgbha4uzqqr6fl2iwahutp4 2444463 2444462 2026-08-05T11:13:03Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2444463 wikitext text/x-wiki i61slksboiwv58qe4iv5eearjnhr47o 2444464 2444463 2026-08-05T11:13:07Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2444464 wikitext text/x-wiki oxfsci0gl9mpxg18ylykwevm56ll6fb 2444465 2444464 2026-08-05T11:18:33Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2444465 wikitext text/x-wiki 6erizxau429qum6431bigcaa98f1bkc 2444466 2444465 2026-08-05T11:18:46Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444466 wikitext text/x-wiki r20vr3gqh3ghzqzf3p7alsn73idgy8o 2444467 2444466 2026-08-05T11:18:49Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2444467 wikitext text/x-wiki 6lfatj4muvstjzp7c47ycqrgnoa2gqr 2444468 2444467 2026-08-05T11:18:59Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444468 wikitext text/x-wiki ovq81afiadthv4lzmzlxiuyh1ectnhp 2444469 2444468 2026-08-05T11:21:32Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444469 wikitext text/x-wiki t7ssmc7p0of4m884niiv3164g6vk3aj 2444470 2444469 2026-08-05T11:21:43Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444470 wikitext text/x-wiki i5glndy8k1qvdtxi2pf3rj4sqjyn0xl 2444472 2444470 2026-08-05T11:23:01Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444472 wikitext text/x-wiki hehgkqiqfdk02ctmx6tfs05lgnlmmtl 2444473 2444472 2026-08-05T11:23:12Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444473 wikitext text/x-wiki qd4hyg0worgkh055bv8qpyt1z6m3vd7 2444474 2444473 2026-08-05T11:23:16Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444474 wikitext text/x-wiki czbw660nf4k3fmx4ieq2bc523m32w49 2444475 2444474 2026-08-05T11:23:28Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444475 wikitext text/x-wiki gx9j0e6rnhjcip4gnda4dsnzsnicbs4 2444476 2444475 2026-08-05T11:27:55Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444476 wikitext text/x-wiki felzlyvbt7ckliy75vl2s2f0ujfnk5s 2444477 2444476 2026-08-05T11:28:08Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444477 wikitext text/x-wiki 4qftev0t6x23jyn2wovuwxl85ryvbp3 2444478 2444477 2026-08-05T11:34:52Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host mc2046.codfw.wmnet with OS trixie 2444478 wikitext text/x-wiki ogh9ldosqek0hsbhnc6z00b75fb4ahr 2444479 2444478 2026-08-05T11:35:14Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host mc2046 2444479 wikitext text/x-wiki 608jf3joluycwz31qda2jirojrp9bij 2444480 2444479 2026-08-05T11:35:16Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host mc2046 2444480 wikitext text/x-wiki sebgswrxghdudhs58w5erotx7ija880 2444481 2444480 2026-08-05T11:38:41Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444481 wikitext text/x-wiki 3fnmuaoy8yytsjbf8jwva1yozr4c2r2 2444482 2444481 2026-08-05T11:42:14Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444482 wikitext text/x-wiki afseiw9gai48ij0wujy0114d9fm7447 2444483 2444482 2026-08-05T11:43:57Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444483 wikitext text/x-wiki 6z0xpzxd8fslkmdf5lgqenvuu75op10 2444484 2444483 2026-08-05T11:44:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444484 wikitext text/x-wiki 8ems6tcmkfhdo7jc0yhlhuxz3detuwz 2444485 2444484 2026-08-05T11:44:28Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444485 wikitext text/x-wiki km80ckukyh7k36xmxnzif8ap7dd5qtl 2444486 2444485 2026-08-05T11:44:40Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444486 wikitext text/x-wiki 4xvuorrq0yqox5idwc8if7y4m660uo7 2444487 2444486 2026-08-05T11:45:16Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444487 wikitext text/x-wiki qkiatut7mrrhqyhx97j276971fndluu 2444488 2444487 2026-08-05T11:45:28Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444488 wikitext text/x-wiki 0kfvi50u6hml8mk0jj48o6kjaznwozr 2444489 2444488 2026-08-05T11:47:17Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444489 wikitext text/x-wiki ha13jsb9d5u6zlc3of4vk1ialp0tapx 2444490 2444489 2026-08-05T11:47:30Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444490 wikitext text/x-wiki trzkqjo6tncfm3y9yb3g3kni4uiyrnp 2444491 2444490 2026-08-05T11:47:39Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444491 wikitext text/x-wiki 8cymgeddgh3efwm885q9iderpzcj62m 2444492 2444491 2026-08-05T11:47:52Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444492 wikitext text/x-wiki tldh936hqm05c9l0l1nqfldtfbsm2p6 2444493 2444492 2026-08-05T11:47:58Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444493 wikitext text/x-wiki 6eect1sr067hy4shoncgxnuxj5hvwdj 2444494 2444493 2026-08-05T11:48:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444494 wikitext text/x-wiki ti7z2k4mczwszhdmsn1yg6j28ch3zde 2444495 2444494 2026-08-05T11:49:52Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444495 wikitext text/x-wiki sfcfcc2saffcpl1utrywg5f7rl4n3z6 2444496 2444495 2026-08-05T11:50:07Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444496 wikitext text/x-wiki b6ghsy2d4629h8hi6sxmqz94nexl37o 2444497 2444496 2026-08-05T11:50:42Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444497 wikitext text/x-wiki 8mixujll9scemddh6d8poywwvl792ou 2444498 2444497 2026-08-05T11:50:54Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444498 wikitext text/x-wiki qmzcbmm2cak4vr16m711wdgfvbxwbgn 2444499 2444498 2026-08-05T11:52:29Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444499 wikitext text/x-wiki drbwc6g4h2az1nnoxmzumfcy1kanwo5 2444500 2444499 2026-08-05T11:52:41Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444500 wikitext text/x-wiki 58tg95g40ubjujhcegjgg1l1xplcves 2444501 2444500 2026-08-05T11:52:54Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2444501 wikitext text/x-wiki hlnar3xa4a0we11p5wk7l28i5m3bj5o 2444502 2444501 2026-08-05T11:53:05Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2444502 wikitext text/x-wiki 4xwms33t5zc1bk3opgbug8zeo9erqfj 2444503 2444502 2026-08-05T11:53:58Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage 2444503 wikitext text/x-wiki 2zumjnfx3sx2brw2uig7fqyduqv0na4 2444504 2444503 2026-08-05T11:57:24Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage 2444504 wikitext text/x-wiki sddi4q5rcrb3zz4y6bny72fya3j5cc6 0 17290 2444190 2443676 2026-08-04T16:48:48Z Stashbot 7414 James_F: Docker: [php83] Upgrade PHP to 8.3.33 2444190 wikitext text/x-wiki 2p2u0bf3mzmp1acb992jdy8rx1hmo5m 2444231 2444190 2026-08-04T19:06:25Z Stashbot 7414 dancy: Updating buildkitd to v0.32.2 on gitlab-cloud-runners (staging and production) (T433985) 2444231 wikitext text/x-wiki aws7sjkz1or0wd6obxpcfdvpwehk89n 12 18681 2444327 2399971 2026-08-05T07:55:51Z DCaro (WMF) 18271 Added note for continuous jobs 2444327 wikitext text/x-wiki 53kouzfzgygqmluxqip3zwgi76awv2y 498 30942 2444229 2443592 2026-08-04T18:59:53Z Stashbot 7414 andrewbogott: systemctl reset-failed on cloudbackup2003. This patch should prevent future such false alarms: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1321058 2444229 wikitext text/x-wiki bvp3u9a7j3lsdlxu4t69mm3hx68zlbk 2444248 2444229 2026-08-04T20:01:35Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) 2444248 wikitext text/x-wiki t27qexn5py0ogbyjiknd8oy4lepi63w 2444249 2444248 2026-08-04T20:03:18Z Stashbot 7414 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=97) (T429387) 2444249 wikitext text/x-wiki kdae4t48b41j1szooyn5eexcb08k1qz 2444250 2444249 2026-08-04T20:07:03Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) 2444250 wikitext text/x-wiki qyjrkkzepyvczduyi8rbh07doxuqqw8 2444270 2444250 2026-08-04T23:47:17Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) (T429387) 2444270 wikitext text/x-wiki t1tisa7ccejmhbgdpfvmsqxp306uwsx 2444283 2444270 2026-08-05T01:36:53Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) 2444283 wikitext text/x-wiki rpggz26q1tyfqa5oum7nom1nocay7yp 2444284 2444283 2026-08-05T01:36:58Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) (T429387) 2444284 wikitext text/x-wiki n7ia5bnojjy7rnrjrmvsuy3kpisamnt 2444285 2444284 2026-08-05T01:37:09Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) 2444285 wikitext text/x-wiki dm06q58r8dhah1dyjk67mk723r3gei3 2444290 2444285 2026-08-05T02:17:07Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) (T429387) 2444290 wikitext text/x-wiki q6s8grttdbd7f2w1hdvuperooynu9qt 10 54864 2444125 2442602 2026-08-04T15:39:31Z TBurmeister (WMF) 27417 Add link to new page Monitor and analyze tools 2444125 wikitext text/x-wiki gficsp55aazzpfo1nbqvkm0cj8xhp1l 0 441014 2444278 2434450 2026-08-05T01:17:14Z SFrench-WMF 37362 /* Master cluster switchover */ Refresh from T433554 2444278 wikitext text/x-wiki dsuv7hmfl6r0jx812n73dju4sztiu2w 2444279 2444278 2026-08-05T01:22:00Z SFrench-WMF 37362 /* Primary cluster switchover */ Minor tweaks to contextualize phases 2444279 wikitext text/x-wiki 4xj6o2jljixo90zioo6pfd0xgprfq3a 0 441820 2444008 2435151 2026-08-04T13:15:39Z Udehb-WMF 40856 added Outdated tag 2444008 wikitext text/x-wiki 8loryq5hy27wdm916vjtjigzjyco89g 2444114 2444008 2026-08-04T15:29:32Z BTracy-WMF 46548 2444114 wikitext text/x-wiki c5yge1dt3wrqkeorat07n32c0k24e3d 498 443713 2444206 2443797 2026-08-04T17:48:52Z Stashbot 7414 wmbot~valeriobozzolan@tools-bastion-15: deletionbot: released latest changes - up to commit https://gitlab.wikimedia.org/valeriobozzolan/itwiki-deletions-info-bot/-/commit/4ee382a683ce953d8b883a2114beb00d9c874dc2 2444206 wikitext text/x-wiki 37jlh2il6nkh5c7wf28euzp9nbohjpw 0 444130 2444262 2428412 2026-08-04T22:36:09Z RLazarus (WMF) 15215 /* Advanced use cases: using helm */ even read-only helm commands require deploy creds. slightly falsify akosiaris's shell transcript (from helm2) to match 2444262 wikitext text/x-wiki qtdvzej88uqlsnilrjbxbbth7exum7y 12 444256 2444109 2435368 2026-08-04T15:17:13Z TBurmeister (WMF) 27417 Update headings to use consistent verb forms, add section with link to new page 2444109 wikitext text/x-wiki 4jobdh773v0di057jsevm2uaogrp8ub 0 449019 2444446 2367102 2026-08-05T10:30:09Z KBach-WMF 31931 /* Demo server */ Add nftables as a required package 2444446 wikitext text/x-wiki hyzm5bxz3aqfwshbzu7fqvkv7ovu6w4 12 449113 2444302 2430795 2026-08-05T04:56:33Z Raymond Ndibe 16522 updated documentation to add --publish 2444302 wikitext text/x-wiki b04i9chjbd7699yafahw6apz9e96e7g 0 449160 2444168 2443714 2026-08-04T16:23:22Z Dexbot 30554 Bot: Updating the report 2444168 wikitext text/x-wiki koia2dc5xky090b1kad7bljpbyuxhle 2444271 2444168 2026-08-05T00:00:33Z Dexbot 30554 Bot: Updating the report 2444271 wikitext text/x-wiki in1a74nzzbuvjbvvprccqno2mwpx7a5 2444273 2444271 2026-08-05T00:03:34Z Dexbot 30554 Bot: Updating the report 2444273 wikitext text/x-wiki oqoubrzkno9gs8agzzc576gy0advciq 2444282 2444273 2026-08-05T01:30:48Z Dexbot 30554 Bot: Updating the report 2444282 wikitext text/x-wiki tbbalkesrv73id1zws6egg5o3yw3a1l 0 453597 2444188 2443288 2026-08-04T16:42:27Z DCaro (WMF) 18271 /* 2026 */ added the locale fix 2444188 wikitext text/x-wiki lxqwxv7vru5984u9ivq2jdm6fat5n3x 2444300 2444188 2026-08-05T04:19:47Z Raymond Ndibe 16522 added https://phabricator.wikimedia.org/T423408 to changelog 2444300 wikitext text/x-wiki gknxlbm1vgg59bh9y85oz3rxh7j304m 2444322 2444300 2026-08-05T07:53:44Z DCaro (WMF) 18271 Added link to the publish docs 2444322 wikitext text/x-wiki 1oz27hu4eyfssonu8as78kvb0dm9p31 0 459044 2444160 2443669 2026-08-04T16:11:03Z BKing (WMF) 29017 /* Rolling restarts */ 2444160 wikitext text/x-wiki epfveis6a83o5jda9w3ksz73q0g39u9 2444162 2444160 2026-08-04T16:11:35Z BKing (WMF) 29017 /* Rolling restarts */ 2444162 wikitext text/x-wiki 47gsdllvkbbd60guvaplrt1ecepd8ww 0 459128 2444225 2443141 2026-08-04T18:27:47Z BLiviero-WMF 53603 Clarified timeline 2444225 wikitext text/x-wiki sgbc5jjwcs14y3o3ztlxu0mwkfh9747 0 459699 2444264 2443729 2026-08-04T23:15:33Z Stashbot 7414 eileen: civicrm upgraded from 8ed3282e to e229c253 2444264 wikitext text/x-wiki eztokfltt0830kf5m3ksw2h0cbmsk0u 0 460194 2444141 2431936 2026-08-04T16:03:37Z CGoubert-WMF 32775 /* Navigating the checklist */ Clarify component types 2444141 wikitext text/x-wiki b01amjmn6znp9wyto0fuzwt163y67s0 1 460409 2443998 2436729 2026-08-04T12:47:48Z Peter F. Patel-Schneider 42591 /* status of label service queries */ new section 2443998 wikitext text/x-wiki f32bfum5sdqq6tvfgsw92nmxh5m4l2j 0 460417 2443990 2443933 2026-08-04T12:37:34Z Kevin Bazira 14737 /* Deployments (ml-serve-eqiad) */ update ope-a-9b nodes as 1012 was removed by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1319050 2443990 wikitext text/x-wiki 276w1jxbegsoo6kw9vs06ld2wy0qsj5 2443992 2443990 2026-08-04T12:39:39Z Kevin Bazira 14737 /* Deployments (ml-serve-eqiad) */ update qwen3-embedding nodes as 1012 was removed by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1319050 2443992 wikitext text/x-wiki fsehkvg6054cpfto0zp6yzf26duypyl 2443993 2443992 2026-08-04T12:42:08Z Kevin Bazira 14737 /* Internal — project-specific (not public, not in Studio) */ jina-embedding precision is now bf16 2443993 wikitext text/x-wiki 0thookr9pyixazevusv51x0a7b2mvk1 2443997 2443993 2026-08-04T12:43:51Z Kevin Bazira 14737 /* Deployments (ml-serve-eqiad) */ jina-embedding precision is now bf16 2443997 wikitext text/x-wiki a7czkb8j8xg4j68wgneykac1gi4m7yw 2444000 2443997 2026-08-04T12:49:19Z Kevin Bazira 14737 remove stale TODO in the page source 2444000 wikitext text/x-wiki pcwzpq1bzpa4x6c2ic5z0j5s09dcypb 2444001 2444000 2026-08-04T13:00:23Z Kevin Bazira 14737 /* Available Models */ update cope-b-a4b base/type 2444001 wikitext text/x-wiki sc6hmdwrgam3sywedmk6uvvtmv5c3mt 2444003 2444001 2026-08-04T13:03:35Z Kevin Bazira 14737 /* Deployments (ml-serve-eqiad) */ cope-a-9b update weight size 2444003 wikitext text/x-wiki lfi9129ke3pz3sporjq8bmugz23u1kh 2444004 2444003 2026-08-04T13:04:44Z Kevin Bazira 14737 remove stale TODO in the page source 2444004 wikitext text/x-wiki azxoy26tgp6ryxv64or0s0hf9ofonva 2444007 2444004 2026-08-04T13:11:49Z Kevin Bazira 14737 /* Deployments (ml-serve-eqiad) */ update cope-b-a4b weight size 2444007 wikitext text/x-wiki pjjglym4u8433nzllsn5vdabmwpgycm 0 460470 2444259 2439746 2026-08-04T22:07:46Z PPelberg (WMF) 37902 Adding link to VE Suggestion Mode tag in phab 2444259 wikitext text/x-wiki k0rsf9sbra16b6060q7tlrydvlr22hh 2 460515 2444212 2443665 2026-08-04T18:02:34Z Ottomata 88 2444212 wikitext text/x-wiki 8f5v9plqo5va73vwzl0mlygt909vnyu 2444219 2444212 2026-08-04T18:12:43Z Ottomata 88 2444219 wikitext text/x-wiki m848hdgxndrna1950aunyojt8rr7ml1 2444226 2444219 2026-08-04T18:35:37Z Ottomata 88 2444226 wikitext text/x-wiki iyscjpqefef6l6mxho8jdg514vxirys 2444228 2444226 2026-08-04T18:40:47Z Ottomata 88 2444228 wikitext text/x-wiki 6izvqgkt8txj304bjkaljt33clebvcl 12 460517 2444112 2026-08-04T15:25:05Z TBurmeister (WMF) 27417 Create page as draft 2444112 wikitext text/x-wiki ltgv5h1ao3jd9mdtc3mr3rfdd1yx2eg 2444115 2444112 2026-08-04T15:29:42Z TBurmeister (WMF) 27417 Add links to some tools 2444115 wikitext text/x-wiki nmpztvz862ha74pduzkpy4pg6ysmqte 2444122 2444115 2026-08-04T15:38:13Z Taavi-WMF 41365 do not link to tool loading external resources 2444122 wikitext text/x-wiki 9lg17r58jklff6gwwkrtjwwsa54zn6n 2444123 2444122 2026-08-04T15:38:25Z TBurmeister (WMF) 27417 Add phab link and cleanup intro 2444123 wikitext text/x-wiki gb02fpw133x51jzhvzoox9dwxkrz9pg 2444127 2444123 2026-08-04T15:45:15Z TBurmeister (WMF) 27417 Add tool link 2444127 wikitext text/x-wiki lcdhj423kezwtddvifk4iuw2iy7rthf 2444128 2444127 2026-08-04T15:46:50Z TBurmeister (WMF) 27417 Add bg info link 2444128 wikitext text/x-wiki r5vwv63vqelujuiyn44hzog23o42ifo 2444135 2444128 2026-08-04T15:56:33Z TBurmeister (WMF) 27417 More phab links 2444135 wikitext text/x-wiki lczcueoa3ke9u4hlztejmpxbpzc3jl8 2444138 2444135 2026-08-04T15:59:38Z Taavi-WMF 41365 Undid revision [[Special:Diff/2444127|2444127]] by [[Special:Contributions/TBurmeister (WMF)|TBurmeister (WMF)]] ([[User talk:TBurmeister (WMF)|talk]]) per [[Special:Diff/2444122]] 2444138 wikitext text/x-wiki 1yw298qqnk42gsj4hk61ge44i277gnx 2444364 2444138 2026-08-05T09:00:54Z Count Count 11194 /* Analyze tool usage */ add 2444364 wikitext text/x-wiki c6kka6njdnobn3dgr2h4xco9hhunofo 2444365 2444364 2026-08-05T09:01:23Z Count Count 11194 /* Analyze tool usage */ unbold 2444365 wikitext text/x-wiki oiaqkogcavz0b5yqx3acvpqpncisnwf 2444471 2444365 2026-08-05T11:22:37Z Taavi-WMF 41365 remove views.toolforge.org once more 2444471 wikitext text/x-wiki 24idqw0qaa7uc92px344cpbu41ysj8m