Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.13 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2443047 2442947 2026-07-30T13:58:50Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2443047 wikitext text/x-wiki 9b5qhgrt04isx70k6df2es6dboo0kn7 2443048 2443047 2026-07-30T13:59:29Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli 2443048 wikitext text/x-wiki s5wdqjb51h9o15tbgq3jkxcmyr9qa44 2443049 2443048 2026-07-30T14:00:05Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2443049 wikitext text/x-wiki g38n0s6872lwkgic30sd497h96y50xy 2443050 2443049 2026-07-30T14:01:26Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2443050 wikitext text/x-wiki my2jwlrf1dg6tk6zvt8uhtjpvw8tpab 2443072 2443050 2026-07-30T14:25:23Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2443072 wikitext text/x-wiki gpib0vy0tqrhpwt0zbwqvhpn55i75q2 2443078 2443072 2026-07-30T14:28:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api 2443078 wikitext text/x-wiki 0amngag45q1f0x0ivpve9pmcxelywlx 2443080 2443078 2026-07-30T14:30:48Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2443080 wikitext text/x-wiki 6jh00ogsemdzs5h15u4p4elg9afcs77 2443088 2443080 2026-07-30T14:43:30Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2443088 wikitext text/x-wiki jgulcph6j7b23z886sr1h45kz20rfbr 2443105 2443088 2026-07-30T15:00:34Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2443105 wikitext text/x-wiki 41rln8g72ubvtjrgste2zzb3ob5uwd2 2443106 2443105 2026-07-30T15:00:44Z Stashbot 7414 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) 2443106 wikitext text/x-wiki g9hrtqp50cdyy1ao4q8bvcjtokdxecr 2443107 2443106 2026-07-30T15:01:00Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2443107 wikitext text/x-wiki 9hyp5c7wsmpeq3oy0289hyvzxrbkksz 2443109 2443107 2026-07-30T15:03:30Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2443109 wikitext text/x-wiki tq4x37r5dw74qki8bndcishy587v38i 2443111 2443109 2026-07-30T15:04:30Z Stashbot 7414 dcaro: reboot tools-bastion-15, getting out of memory 2443111 wikitext text/x-wiki 06t8fcci3l1iub4o765xrvnr40408kx 2443114 2443111 2026-07-30T15:05:58Z Stashbot 7414 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) 2443114 wikitext text/x-wiki 3cuofb1cbd98byoeh3llib60vta4gsq 2443115 2443114 2026-07-30T15:06:08Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli 2443115 wikitext text/x-wiki l7vfqg3i780vf8363d34e8dis2cr1br 2443116 2443115 2026-07-30T15:06:24Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2443116 wikitext text/x-wiki fnzz5ivdv5cwg74s1v1be4qxehz8hbs 2443117 2443116 2026-07-30T15:08:54Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli 2443117 wikitext text/x-wiki rzjrh7g2ix8p1gn0fcfvzhcgnf7vuy5 2443118 2443117 2026-07-30T15:09:45Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2443118 wikitext text/x-wiki 2hjayi1n4h3ysrc5pr68ff2ym0bo0zo 2443119 2443118 2026-07-30T15:10:57Z Stashbot 7414 godog: hard reboot tools-bastion-15 2443119 wikitext text/x-wiki bspvshm4s8v8d3146gmxlyeuvtela2w 2443129 2443119 2026-07-30T15:18:54Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli 2443129 wikitext text/x-wiki e78dwdt0wr3y8mweojf5a52suxac9mb 2443263 2443129 2026-07-31T11:28:21Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2443263 wikitext text/x-wiki ifqqsn0dbudsarzsy2s872nd1r96hu8 2443264 2443263 2026-07-31T11:36:13Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2443264 wikitext text/x-wiki 00vijsriyq5wdanvaetz89ejuvkvd38 2443269 2443264 2026-07-31T11:53:38Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2443269 wikitext text/x-wiki 4cleo5dpnjscngas0gr25i6pzbu81x5 0 4108 2442986 2442968 2026-07-30T12:30:30Z ScheduleDeploymentBot 37566 Add [[gerrit:1319466]] to Thursday, July 30 UTC afternoon backport window 2442986 wikitext text/x-wiki cp5a774zivyzlknuf8liw64ydns8an7 2442987 2442986 2026-07-30T12:30:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1319467]] to Thursday, July 30 UTC afternoon backport window 2442987 wikitext text/x-wiki by6xhlcvohjaow9wevw2cnfwaqqefjc 2443025 2442987 2026-07-30T13:27:42Z ScheduleDeploymentBot 37566 Add [[gerrit:1319482]] to Thursday, July 30 UTC afternoon backport window 2443025 wikitext text/x-wiki 0eikvtiusytni20nz64he9me831bf03 2443026 2443025 2026-07-30T13:27:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1309552]] to Thursday, July 30 UTC afternoon backport window 2443026 wikitext text/x-wiki bo7y9b953j3ggi0ejfmpuh6fb2zrmtg 2443032 2443026 2026-07-30T13:45:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1319487]] to Thursday, July 30 UTC afternoon backport window 2443032 wikitext text/x-wiki iuhrrxxo9833vyeavzlrsq6wgio20sj 2443033 2443032 2026-07-30T13:45:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1319488]] to Thursday, July 30 UTC afternoon backport window 2443033 wikitext text/x-wiki q9wvdjbzvpe7rcep5eh2vwxafbhsot5 2443187 2443033 2026-07-30T19:19:44Z ScheduleDeploymentBot 37566 Add [[gerrit:1319521]] to Thursday, July 30 UTC late backport window 2443187 wikitext text/x-wiki j77fvuurg7fthbo0ib6lomk6rrfw0qm 2443208 2443187 2026-07-30T23:02:58Z CMing (WMF) 23277 /* {{Deployment_day|date=2026-08-04}} */ 2443208 wikitext text/x-wiki ox6d906shvbjkrwhxv7dqvt1whkxupt 498 6726 2443040 2442944 2026-07-30T13:56:07Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli 2443040 wikitext text/x-wiki 3fkxamio8a00j7g0x12dprwse5131on 2443046 2443040 2026-07-30T13:57:33Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli 2443046 wikitext text/x-wiki 2awnqp5e01orej1au3zq4xzr15exkv5 2443060 2443046 2026-07-30T14:14:12Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2443060 wikitext text/x-wiki iafae4jvf40si4vko4fjlohatxc16tv 2443071 2443060 2026-07-30T14:25:13Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2443071 wikitext text/x-wiki qmp1qa6b2k8swlw6qjcxkdec3tzq93y 2443099 2443071 2026-07-30T14:55:53Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2443099 wikitext text/x-wiki dhpwrdsm72d8rgqkbzcuew1x0im4w2w 2443108 2443099 2026-07-30T15:03:12Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2443108 wikitext text/x-wiki jl9g0d5130iu6xzuxjy91zszry7mapk 2443259 2443108 2026-07-31T11:19:40Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2443259 wikitext text/x-wiki hzv450dnd4bs5vzw1371zbyxpzc48yj 2443260 2443259 2026-07-31T11:27:13Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2443260 wikitext text/x-wiki aixd6za0i136osmgo0qxqdyddsvhpv7 2443267 2443260 2026-07-31T11:44:41Z Stashbot 7414 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2443267 wikitext text/x-wiki 41327lp86kp8dqgc4heipkez69mbjue 2443268 2443267 2026-07-31T11:53:23Z Stashbot 7414 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2443268 wikitext text/x-wiki 6zpk8rkjvgchovwt4xc8zve0ue4ypqm 2443271 2443268 2026-07-31T11:53:43Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2443271 wikitext text/x-wiki 1e3ya4umsh4k2zmujmv41wnw3tlee5x 0 7919 2442974 2442973 2026-07-30T12:02:50Z Stashbot 7414 ayounsi@dns1004: START - running authdns-update 2442974 wikitext text/x-wiki rmdl2l8eweoqzh0dsaxrynrkkgz3ott 2442975 2442974 2026-07-30T12:05:20Z Stashbot 7414 ayounsi@dns1004: END - running authdns-update 2442975 wikitext text/x-wiki tetj19x9v4pftx74zeb3g6alrcgrgxx 2442976 2442975 2026-07-30T12:13:36Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance 2442976 wikitext text/x-wiki mqzubtoepbdtc1c094bmudh8qusa4eh 2442977 2442976 2026-07-30T12:13:58Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2047.codfw.wmnet with reason: Maintenance 2442977 wikitext text/x-wiki gnnd5dxtxvj5gsiqgi12apl8kmax9kr 2442978 2442977 2026-07-30T12:14:06Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es2047 (T431660)', diff saved to https://phabricator.wikimedia.org/P95793 and previous config saved to /var/cache/conftool/dbconfig/20260730-121404-cwilliams.json 2442978 wikitext text/x-wiki i5tf2a34qdw6mtacndmooscrlcejeuy 2442979 2442978 2026-07-30T12:14:57Z Stashbot 7414 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply 2442979 wikitext text/x-wiki 51zb7f72u40dtktngpp9n09jc5su7ri 2442980 2442979 2026-07-30T12:15:26Z Stashbot 7414 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply 2442980 wikitext text/x-wiki sbsk8ordkumu3gvxyo1enmibg8rh5vg 2442981 2442980 2026-07-30T12:18:36Z Stashbot 7414 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2442981 wikitext text/x-wiki 17zi3gppsgr0eqd77hh1r775r192d2x 2442982 2442981 2026-07-30T12:18:44Z Stashbot 7414 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2442982 wikitext text/x-wiki sbn8o38xomqi8a9esi6aab1eurymk88 2442983 2442982 2026-07-30T12:18:54Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2078.codfw.wmnet with OS trixie 2442983 wikitext text/x-wiki ijpe07xfrkwc06dcqi1yupl8xjcjyzg 2442984 2442983 2026-07-30T12:19:08Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2047: Maintenance 2442984 wikitext text/x-wiki q4f8dgqsimwojsw7vbyrrdaor97amaj 2442985 2442984 2026-07-30T12:30:17Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1319162|Remove $wmg hack for UploadStashMaxAge (T119117)]] 2442985 wikitext text/x-wiki ctldtkml107rf6gfwswb5c8wo6hsz4v 2442988 2442985 2026-07-30T12:31:27Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2442988 wikitext text/x-wiki 3kdxk4zzpyiuoylqzz3lsz5k81ilye3 2442990 2442988 2026-07-30T12:32:00Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2442990 wikitext text/x-wiki oqbypz8wuc3ud9w0xtp3sefecfp9vta 2442991 2442990 2026-07-30T12:32:30Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1319162|Remove $wmg hack for UploadStashMaxAge (T119117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2442991 wikitext text/x-wiki 5p0tl72r8v958r1fx8l8tg4qxkknk4f 2442992 2442991 2026-07-30T12:32:38Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2442992 wikitext text/x-wiki 6mt903ahfp9fggmwpfffqzfjkt27wzz 2442993 2442992 2026-07-30T12:32:57Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with deployment 2442993 wikitext text/x-wiki d7t9y3gc0a70yyxtt7bzzuphek6q78v 2442994 2442993 2026-07-30T12:33:32Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2442994 wikitext text/x-wiki pynau6vkqbd9kw591f8vz83k9oeee4k 2442995 2442994 2026-07-30T12:37:09Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319162|Remove $wmg hack for UploadStashMaxAge (T119117)]] (duration: 06m 51s) 2442995 wikitext text/x-wiki r92d1ycze1otwuwyhbpp5bubdywhlfq 2442996 2442995 2026-07-30T12:37:53Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2078.codfw.wmnet with reason: host reimage 2442996 wikitext text/x-wiki cg3452ma4gf4n7z4ltbzpaazst5rb9y 2442997 2442996 2026-07-30T12:44:07Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2078.codfw.wmnet with reason: host reimage 2442997 wikitext text/x-wiki 80oe5b8ynue5ohmh9xdn45tlrbs8hsk 2442998 2442997 2026-07-30T12:59:13Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2038.codfw.wmnet with reason: Maintenance 2442998 wikitext text/x-wiki ftizl1ueewf6y0s3nburgow1u95hohg 2442999 2442998 2026-07-30T12:59:21Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es2038 (T431660)', diff saved to https://phabricator.wikimedia.org/P95797 and previous config saved to /var/cache/conftool/dbconfig/20260730-125919-cwilliams.json 2442999 wikitext text/x-wiki t6mcj93sanylkyzgj3kl5f0w0xwxj4y 2443001 2442999 2026-07-30T13:01:13Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.dns.netbox 2443001 wikitext text/x-wiki 0ikdowdtw5w4s6heghy7vkln164nfut 2443002 2443001 2026-07-30T13:03:34Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2078.codfw.wmnet with OS trixie 2443002 wikitext text/x-wiki l2ntj0wnks36yjwduj6qnxz3jbv20jm 2443003 2443002 2026-07-30T13:04:46Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2038: Maintenance 2443003 wikitext text/x-wiki 0zicowgc286pobbtiy2aspwfa4irtga 2443004 2443003 2026-07-30T13:06:48Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1319466|Instrument link_click server-side instead of client-side (T433257)]], [[gerrit:1319467|Instrument link_click server-side instead of client-side (T433257)]] 2443004 wikitext text/x-wiki 57o0r6cjs7ec25huktxl9dha8q7jxr8 2443005 2443004 2026-07-30T13:07:45Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1048 cloud-private - filippo@cumin1003" 2443005 wikitext text/x-wiki 8qyyzxljdm3m31gknfex9hq06f3jw4f 2443006 2443005 2026-07-30T13:08:04Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1048 cloud-private - filippo@cumin1003" 2443006 wikitext text/x-wiki hw3xc65ktg4bfaqo628bhn8fbd14dh7 2443007 2443006 2026-07-30T13:08:06Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2443007 wikitext text/x-wiki 2wccsfgp1damj84p4632k3ejfyjtfnf 2443008 2443007 2026-07-30T13:08:16Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2047: Maintenance 2443008 wikitext text/x-wiki h0y0lxuovpyklxaexr46zypjgduainh 2443009 2443008 2026-07-30T13:08:34Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1048.eqiad.wmnet with OS trixie 2443009 wikitext text/x-wiki mrmyeu7rs68favw7gi7hbk64uyobrb4 2443010 2443009 2026-07-30T13:08:45Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, stran: Backport for [[gerrit:1319466|Instrument link_click server-side instead of client-side (T433257)]], [[gerrit:1319467|Instrument link_click server-side instead of client-side (T433257)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2443010 wikitext text/x-wiki gwpcw4672m0t65vsbf4fydxb773fwa3 2443011 2443010 2026-07-30T13:10:22Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2038: Maintenance 2443011 wikitext text/x-wiki 75boxhemj3s8htadmae44t4avgqic5z 2443012 2443011 2026-07-30T13:10:27Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2038: Maintenance 2443012 wikitext text/x-wiki 314syaw1r69rp70xm4qocztalwtpiyc 2443013 2443012 2026-07-30T13:10:58Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T433606 2443013 wikitext text/x-wiki 5ieb2m8di5defwhsfc7i44gxjgfqcnz 2443014 2443013 2026-07-30T13:13:17Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, stran: Continuing with deployment 2443014 wikitext text/x-wiki 5k94s991hkitysagwp4rwablneckl14 2443015 2443014 2026-07-30T13:16:18Z Stashbot 7414 bking@cumin2003: conftool action : set/pooled=yes; selector: name=wdqs2022\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main 2443015 wikitext text/x-wiki emgrv9nvo2russpbp9ksl2ss55k70zj 2443016 2443015 2026-07-30T13:17:19Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319466|Instrument link_click server-side instead of client-side (T433257)]], [[gerrit:1319467|Instrument link_click server-side instead of client-side (T433257)]] (duration: 10m 31s) 2443016 wikitext text/x-wiki jr5wtax3ftqequ26gs1giaal5ysjp4j 2443017 2443016 2026-07-30T13:17:22Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance 2443017 wikitext text/x-wiki csba81jkpke0qqf6m1m0nbo54t6ri5q 2443018 2443017 2026-07-30T13:17:29Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es1036 (T431660)', diff saved to https://phabricator.wikimedia.org/P95800 and previous config saved to /var/cache/conftool/dbconfig/20260730-131727-cwilliams.json 2443018 wikitext text/x-wiki fvjl5c8uimuf6qpgbx9669551xta109 2443020 2443018 2026-07-30T13:20:44Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T433606 2443020 wikitext text/x-wiki sf6jbijs3c6euljviu6kz2ggkty9q2s 2443021 2443020 2026-07-30T13:22:42Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Maintenance 2443021 wikitext text/x-wiki cli80y4p0qseuf0wd8hzt762e8treea 2443022 2443021 2026-07-30T13:22:49Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T433606 2443022 wikitext text/x-wiki lsi6u4d4d6ymhvbobdm7l0p73p481rq 2443027 2443022 2026-07-30T13:28:20Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es1036: Maintenance 2443027 wikitext text/x-wiki c4fj0l7u86ouvfqbnl0bd24k8e4ji1r 2443028 2443027 2026-07-30T13:28:25Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Maintenance 2443028 wikitext text/x-wiki sg2hw2e9p5fb118c9g86wy22y0a7hd6 2443029 2443028 2026-07-30T13:32:16Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1319449|SpecialCreateAccount: make user policy-link available again (T430604)]] 2443029 wikitext text/x-wiki h51x3w4wly60gkkrlpfjkwisf1r3325 2443030 2443029 2026-07-30T13:32:44Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T433606 2443030 wikitext text/x-wiki hol5i0nnq2nr9gj2ru7694pqpwbte09 2443031 2443030 2026-07-30T13:45:08Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2443031 wikitext text/x-wiki kf7oo78fuv2prs3ubmyocczho7ygcvs 2443034 2443031 2026-07-30T13:48:25Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2443034 wikitext text/x-wiki of035nmajl2xbssq1ch63pfkpjn8sh8 2443035 2443034 2026-07-30T13:49:47Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2443035 wikitext text/x-wiki blxc5r666qph75t684h4e16omph1qu1 2443036 2443035 2026-07-30T13:50:00Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1319449|SpecialCreateAccount: make user policy-link available again (T430604)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2443036 wikitext text/x-wiki ao5khsdusdkt2zef2einx5iefwyvjmv 2443037 2443036 2026-07-30T13:52:31Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment 2443037 wikitext text/x-wiki fxsnt8o53hj3u4ztccpf0lk1jshd6dk 2443039 2443037 2026-07-30T13:55:27Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2443039 wikitext text/x-wiki bw7bialfg747gmdxdzsbruptl1jukck 2443041 2443039 2026-07-30T13:56:15Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2038: Maintenance 2443041 wikitext text/x-wiki d11z1l1uohdkk0retm26f9nw43d59km 2443042 2443041 2026-07-30T13:56:31Z Stashbot 7414 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2443042 wikitext text/x-wiki cjcy7qrvh42apmjnghlbcsxfjf6l2p1 2443043 2443042 2026-07-30T13:56:34Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2443043 wikitext text/x-wiki 972ovg0t89aux94rdtv4kqdswml74lt 2443044 2443043 2026-07-30T13:56:36Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance 2443044 wikitext text/x-wiki jopm0mnbm3x8tdnf9ecuac4u2dm93at 2443045 2443044 2026-07-30T13:56:45Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es2040 (T431660)', diff saved to https://phabricator.wikimedia.org/P95806 and previous config saved to /var/cache/conftool/dbconfig/20260730-135643-cwilliams.json 2443045 wikitext text/x-wiki b1adg8ys4dfcu6h2kwbjtv10sz3d3ky 2443051 2443045 2026-07-30T14:01:38Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2443051 wikitext text/x-wiki 97u83gsrvxhicgis4lbu7r3xx1gl4ov 2443052 2443051 2026-07-30T14:03:03Z Stashbot 7414 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2443052 wikitext text/x-wiki bvwae46ecd2ji9m78jivsmpkiulvgwi 2443053 2443052 2026-07-30T14:03:13Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2040: Maintenance 2443053 wikitext text/x-wiki 5ux4odp97mt05lwrkvmqaksoxqusj6z 2443054 2443053 2026-07-30T14:03:57Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319449|SpecialCreateAccount: make user policy-link available again (T430604)]] (duration: 31m 41s) 2443054 wikitext text/x-wiki nc95yaaaork6ak5fl4lnemv3f5pwrd7 2443055 2443054 2026-07-30T14:08:50Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2040: Maintenance 2443055 wikitext text/x-wiki 0hkt7w5i7tq3n4k9us8kwrv0bn75nic 2443056 2443055 2026-07-30T14:08:55Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2040: Maintenance 2443056 wikitext text/x-wiki 4x5zsiaycklpszbf50qco8cew9ut1yd 2443057 2443056 2026-07-30T14:09:48Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1319487|SI: Add performer to server-side link_click event (T433257)]], [[gerrit:1319488|SI: Add performer to server-side link_click event (T433257)]] 2443057 wikitext text/x-wiki eiilrrwzy4ktib9e67bdwq10pb2q6g0 2443058 2443057 2026-07-30T14:11:23Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2079.codfw.wmnet with OS trixie 2443058 wikitext text/x-wiki 6zz7grh9xeb6dx9v5ybmd53x9s13970 2443059 2443058 2026-07-30T14:13:40Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1319487|SI: Add performer to server-side link_click event (T433257)]], [[gerrit:1319488|SI: Add performer to server-side link_click event (T433257)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2443059 wikitext text/x-wiki nyub2mwlhfzi359xx86jdx7z8d0j9dx 2443061 2443059 2026-07-30T14:14:24Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1036: Maintenance 2443061 wikitext text/x-wiki e89y5irkfhicfw5ix25340am4j7a3r5 2443062 2443061 2026-07-30T14:14:33Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1038.eqiad.wmnet with reason: Maintenance 2443062 wikitext text/x-wiki fkscd0zhpuuw54bndk48335rmd3z5ur 2443063 2443062 2026-07-30T14:14:41Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es1038 (T431660)', diff saved to https://phabricator.wikimedia.org/P95810 and previous config saved to /var/cache/conftool/dbconfig/20260730-141439-cwilliams.json 2443063 wikitext text/x-wiki mbyyrxk0h8elcz43cblhbqbw1eob9l8 2443064 2443063 2026-07-30T14:14:53Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2443064 wikitext text/x-wiki jrjv3ejqz90a9slb6h8ye2epywr5is6 2443068 2443064 2026-07-30T14:20:04Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es1038: Maintenance 2443068 wikitext text/x-wiki lbbx61l201hd9zrgad6h89o10j8m1wu 2443069 2443068 2026-07-30T14:21:09Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319487|SI: Add performer to server-side link_click event (T433257)]], [[gerrit:1319488|SI: Add performer to server-side link_click event (T433257)]] (duration: 11m 19s) 2443069 wikitext text/x-wiki 3xqoq2sqh5n4oe3ryx16u53ujnmkssx 2443070 2443069 2026-07-30T14:23:51Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1319482|plwikiquote: Set AutoConfirmCount to 25 (T433541)]], [[gerrit:1309552|Revert "Temporarily change plwiki tagline for 1.7M articles" (T430512)]] 2443070 wikitext text/x-wiki 0on15oe99i4wl8xie027r3cq7hczcwf 2443073 2443070 2026-07-30T14:25:40Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es1038: Maintenance 2443073 wikitext text/x-wiki g3rx4ew0158coepo1m1nyeuvbiggm0r 2443074 2443073 2026-07-30T14:25:44Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es1038: Maintenance 2443074 wikitext text/x-wiki 9znyz606zbelcdxdr4dwe97olz2hksp 2443075 2443074 2026-07-30T14:25:52Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1319482|plwikiquote: Set AutoConfirmCount to 25 (T433541)]], [[gerrit:1309552|Revert "Temporarily change plwiki tagline for 1.7M articles" (T430512)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2443075 wikitext text/x-wiki 9vj7ssd7jpeg0jz5ntbvorq01xrm6tl 2443076 2443075 2026-07-30T14:26:14Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2443076 wikitext text/x-wiki 7el71vk2x345w260w0d2h52p1g5fv92 2443077 2443076 2026-07-30T14:27:21Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS trixie 2443077 wikitext text/x-wiki ko0m7obzsbm8w8a711w2fgf4onm5hu9 2443079 2443077 2026-07-30T14:30:21Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319482|plwikiquote: Set AutoConfirmCount to 25 (T433541)]], [[gerrit:1309552|Revert "Temporarily change plwiki tagline for 1.7M articles" (T430512)]] (duration: 06m 31s) 2443079 wikitext text/x-wiki a5k8vizabmhjg03qih9ai3rifhg27rr 2443081 2443079 2026-07-30T14:32:03Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2079.codfw.wmnet with reason: host reimage 2443081 wikitext text/x-wiki 9947r0ou6fvlmesj0d5sjib833quz1h 2443082 2443081 2026-07-30T14:36:13Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2098.codfw.wmnet with OS trixie 2443082 wikitext text/x-wiki t9m2i3zw6hc6tj948nzy7j0076f0dwb 2443083 2443082 2026-07-30T14:39:22Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2079.codfw.wmnet with reason: host reimage 2443083 wikitext text/x-wiki qv4q2ae9an46m3ynlupqbb5qvg26kgr 2443084 2443083 2026-07-30T14:39:54Z Stashbot 7414 sukhe: sukhe@lvs1020:~$ sudo systemctl restart pybal.service 2443084 wikitext text/x-wiki 3c8bpv6wp7n3evqhchmyqz8y3uvfv1m 2443085 2443084 2026-07-30T14:42:19Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1081.eqiad.wmnet with OS trixie 2443085 wikitext text/x-wiki ew5ltn30n3ljd6l3z0cedlo0y8dhr0q 2443086 2443085 2026-07-30T14:42:47Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: cluster=urldownloader,service=squid 2443086 wikitext text/x-wiki 3isqbra4y5x5tv9pkgk7sidy9ycd7wi 2443087 2443086 2026-07-30T14:42:51Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/weight=1; selector: cluster=urldownloader,service=squid 2443087 wikitext text/x-wiki 7laxxdn142rq4agdcqjgpf8g34fo650 2443089 2443087 2026-07-30T14:45:02Z Stashbot 7414 tchin@deploy1003: Started scap sync-world: Backport for [[gerrit:1319489|Monolog: Strip diagnostic context from Monolog-based EventBus events (T433457)]] 2443089 wikitext text/x-wiki 4k1al7x3qkr8v214btw86l11zjpjup5 2443090 2443089 2026-07-30T14:47:01Z Stashbot 7414 tchin@deploy1003: jforrester, tchin: Backport for [[gerrit:1319489|Monolog: Strip diagnostic context from Monolog-based EventBus events (T433457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2443090 wikitext text/x-wiki i9ozgtlotnmwrww74lis1do5s3tfbar 2443091 2443090 2026-07-30T14:47:04Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2443091 wikitext text/x-wiki 1jpjt57qzwq7380qdx36rv1u7efh7vn 2443092 2443091 2026-07-30T14:47:45Z Stashbot 7414 tchin@deploy1003: jforrester, tchin: Continuing with deployment 2443092 wikitext text/x-wiki nm1jtmbjynqh856jo6kjtj3uf7u0fzv 2443094 2443092 2026-07-30T14:51:50Z Stashbot 7414 tchin@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319489|Monolog: Strip diagnostic context from Monolog-based EventBus events (T433457)]] (duration: 06m 48s) 2443094 wikitext text/x-wiki 040drqc39d52xptubvbg2f86rz8ttzc 2443095 2443094 2026-07-30T14:54:30Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2443095 wikitext text/x-wiki p04mn057jun801lifs4hcfzyc8bjz87 2443096 2443095 2026-07-30T14:54:42Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2040: Maintenance 2443096 wikitext text/x-wiki 165sqvpt6oj1hk5xfzlotnn7l59wofg 2443097 2443096 2026-07-30T14:55:03Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2048.codfw.wmnet with reason: Maintenance 2443097 wikitext text/x-wiki p42x7lfkreui12lyhjbffp89m13r02h 2443098 2443097 2026-07-30T14:55:12Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es2048 (T431660)', diff saved to https://phabricator.wikimedia.org/P95816 and previous config saved to /var/cache/conftool/dbconfig/20260730-145510-cwilliams.json 2443098 wikitext text/x-wiki 1yr705gyrhjt1pm7vjhxk8p02aadvof 2443100 2443098 2026-07-30T14:56:09Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2098.codfw.wmnet with reason: host reimage 2443100 wikitext text/x-wiki 6zlhouz1vh4molgo91nzv9jbponbsx0 2443101 2443100 2026-07-30T14:59:42Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2079.codfw.wmnet with OS trixie 2443101 wikitext text/x-wiki c2jsw0rlhzsld52r71vt223b6q6j531 2443102 2443101 2026-07-30T15:00:10Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2098.codfw.wmnet with reason: host reimage 2443102 wikitext text/x-wiki cvkmbpoxq6217js3ezi2415am26vl1f 2443103 2443102 2026-07-30T15:00:15Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es2048: Maintenance 2443103 wikitext text/x-wiki 0ot1l4a9o2jwv18t5kqqjkjx1jlmaqt 2443104 2443103 2026-07-30T15:00:30Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1081.eqiad.wmnet with reason: host reimage 2443104 wikitext text/x-wiki 07d4mekoynlg0qez7da148jvd00rg5l 2443110 2443104 2026-07-30T15:04:23Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1081.eqiad.wmnet with reason: host reimage 2443110 wikitext text/x-wiki enqjtd7vu6i789bn5b72d3ruv9o68rg 2443112 2443110 2026-07-30T15:04:41Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts 2443112 wikitext text/x-wiki lnvmxu74q9jy25i0n5xwgagvd9u6o0o 2443113 2443112 2026-07-30T15:04:57Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts 2443113 wikitext text/x-wiki sggcy2h9ewpv7u54o15w56vdg4x9en0 2443120 2443113 2026-07-30T15:11:33Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1038: Maintenance 2443120 wikitext text/x-wiki kzswnx3rdjwf614551u5akki2neyz2s 2443121 2443120 2026-07-30T15:11:53Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance 2443121 wikitext text/x-wiki egx9whf2hu7aftrdimd0odf65wuorgn 2443122 2443121 2026-07-30T15:12:01Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es1047 (T431660)', diff saved to https://phabricator.wikimedia.org/P95820 and previous config saved to /var/cache/conftool/dbconfig/20260730-151200-cwilliams.json 2443122 wikitext text/x-wiki ee0dsx1ajataa4fiedscmgym0rctr8z 2443123 2443122 2026-07-30T15:13:50Z Stashbot 7414 mvernon@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2097.codfw.wmnet with OS trixie 2443123 wikitext text/x-wiki ev92c6imagf8bpr61qv68kell9slfmn 2443124 2443123 2026-07-30T15:15:44Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2080.codfw.wmnet with OS trixie 2443124 wikitext text/x-wiki dkfs3xxdva5l5f3wdmvzvm06bm0x11i 2443125 2443124 2026-07-30T15:17:01Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es1047: Maintenance 2443125 wikitext text/x-wiki clp3nz528272voipiusrz54kw74j65g 2443126 2443125 2026-07-30T15:17:33Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS trixie 2443126 wikitext text/x-wiki aur38on8297hcmf6gaptnlscq0i3r2j 2443127 2443126 2026-07-30T15:18:25Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin2003" 2443127 wikitext text/x-wiki mod5ppdhee53vxhulk7nw5y9ila06kd 2443128 2443127 2026-07-30T15:18:34Z Stashbot 7414 mvernon@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2097.codfw.wmnet with OS trixie 2443128 wikitext text/x-wiki lim9n4g2dcqxgdb94fsf7voc406uvn0 2443130 2443128 2026-07-30T15:19:35Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin2003" 2443130 wikitext text/x-wiki m335gkpwk9nzum59lhh8wifmtuosymd 2443131 2443130 2026-07-30T15:19:37Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2098.codfw.wmnet with OS trixie 2443131 wikitext text/x-wiki gls2tj8j423hu0xsdufk9r8o79kjmvm 2443132 2443131 2026-07-30T15:20:19Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS trixie 2443132 wikitext text/x-wiki a6plu7nvwx3b5fxeyjrncf946pj75uh 2443133 2443132 2026-07-30T15:23:57Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1081.eqiad.wmnet with OS trixie 2443133 wikitext text/x-wiki qotx83nbegpqdiygcg62vl5gl710yyl 2443134 2443133 2026-07-30T15:30:13Z Stashbot 7414 mvernon@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2097.codfw.wmnet with OS trixie 2443134 wikitext text/x-wiki st9nqn3mcbs9nvsf19hjb5ubqzoxly2 2443135 2443134 2026-07-30T15:30:42Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS trixie 2443135 wikitext text/x-wiki dvjt4mzfncveq5sb8qepcfwqjexy8q8 2443136 2443135 2026-07-30T15:36:29Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2080.codfw.wmnet with reason: host reimage 2443136 wikitext text/x-wiki awkyx2jply63ropd4q9oq3pur0ryjdd 2443137 2443136 2026-07-30T15:38:10Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new logging settings - bking@cumin2003 - T324335 2443137 wikitext text/x-wiki tixy1pmuhnseevs1kjtme7oj1ftf6cj 2443138 2443137 2026-07-30T15:42:16Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2080.codfw.wmnet with reason: host reimage 2443138 wikitext text/x-wiki rkfhkyskzobtyx8pn7b3rn5upspd5is 2443139 2443138 2026-07-30T15:43:52Z Stashbot 7414 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c]: T430880 2443139 wikitext text/x-wiki o09e73bzi13l7mmdlyqz2bfo32fyjqk 2443140 2443139 2026-07-30T15:44:01Z Stashbot 7414 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 24s) 2443140 wikitext text/x-wiki er1y2icd1ljovxasmzz5ga4039v08wt 2443142 2443140 2026-07-30T15:48:17Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2048: Maintenance 2443142 wikitext text/x-wiki hlfpof6coow0d1z9u7gd690esdivkme 2443143 2443142 2026-07-30T15:49:41Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2443143 wikitext text/x-wiki 35pso2rbnt7dmbchvyzwb1dcoh53p7v 2443144 2443143 2026-07-30T15:50:18Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1082.eqiad.wmnet with OS trixie 2443144 wikitext text/x-wiki b0m9nnnwpxkvrxoxaxyodc21rcl8d3k 2443145 2443144 2026-07-30T15:53:18Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance 2443145 wikitext text/x-wiki q2pf354ehog77p4fvo9f6m1pcexp27p 2443146 2443145 2026-07-30T15:53:26Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es1040 (T431660)', diff saved to https://phabricator.wikimedia.org/P95827 and previous config saved to /var/cache/conftool/dbconfig/20260730-155324-cwilliams.json 2443146 wikitext text/x-wiki 7dtjev5knhab37kd47eek6aj6dzrs6l 2443147 2443146 2026-07-30T15:53:42Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage 2443147 wikitext text/x-wiki 7vjgz97u84huxjbh0342nv9vbixgutv 2443149 2443147 2026-07-30T15:58:49Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es1040: Maintenance 2443149 wikitext text/x-wiki n9pe0azwrcpxn7uqrst2y2ecjkerzfe 2443150 2443149 2026-07-30T16:03:45Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new logging settings - bking@cumin2003 - T324335 2443150 wikitext text/x-wiki 6ljdhfcfuqqjwlpj6sp4a5fz4fejp1z 2443151 2443150 2026-07-30T16:04:24Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es1040: Maintenance 2443151 wikitext text/x-wiki rwyfonyh7jk62scqhkpcqu0zz9tc8q3 2443152 2443151 2026-07-30T16:04:29Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es1040: Maintenance 2443152 wikitext text/x-wiki 3ugedfoeqzcldkgmasfs7tsksr2uslj 2443153 2443152 2026-07-30T16:04:46Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2080.codfw.wmnet with OS trixie 2443153 wikitext text/x-wiki eu28vw702u3yeo6p7sh1o3q61a8kqhr 2443154 2443153 2026-07-30T16:05:05Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1047: Maintenance 2443154 wikitext text/x-wiki jd6h5zvp2par84htmq7qvu6fadguqev 2443155 2443154 2026-07-30T16:08:18Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2443155 wikitext text/x-wiki q1piilyygu31uwz932yc4h7mirlzevh 2443156 2443155 2026-07-30T16:08:41Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1082.eqiad.wmnet with reason: host reimage 2443156 wikitext text/x-wiki 32i5p3eej8mmq0emdgxsjnoew4ugv0a 2443157 2443156 2026-07-30T16:13:11Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1082.eqiad.wmnet with reason: host reimage 2443157 wikitext text/x-wiki 6xa0b45wh2i5sqb1dwj1gs0ne3ah86t 2443158 2443157 2026-07-30T16:13:28Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2097.codfw.wmnet with OS trixie 2443158 wikitext text/x-wiki ter8yfqzynvicq6quti908goxokyw4b 2443160 2443158 2026-07-30T16:32:35Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2081.codfw.wmnet with OS trixie 2443160 wikitext text/x-wiki ipefrgjcaa301i6jkyn06wiio12mtaa 2443161 2443160 2026-07-30T16:37:00Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1082.eqiad.wmnet with OS trixie 2443161 wikitext text/x-wiki 0bysh0ppb01py25n490eq7d7ow2or1n 2443162 2443161 2026-07-30T16:46:57Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2081.codfw.wmnet with reason: host reimage 2443162 wikitext text/x-wiki 9v1450zcjirrl306pwccdo9xaj9sthb 2443163 2443162 2026-07-30T16:50:16Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1040: Maintenance 2443163 wikitext text/x-wiki cilamr0lbni7441ysowo3lfr541e9sx 2443164 2443163 2026-07-30T16:50:46Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance 2443164 wikitext text/x-wiki ghqiyrxbw6wz0r1gv0yzo7xlwdocobl 2443165 2443164 2026-07-30T16:50:54Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depooling es1048 (T431660)', diff saved to https://phabricator.wikimedia.org/P95833 and previous config saved to /var/cache/conftool/dbconfig/20260730-165053-cwilliams.json 2443165 wikitext text/x-wiki tklc12mnnawwtl4wt7xhu29hbcq5egc 2443166 2443165 2026-07-30T16:54:12Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2081.codfw.wmnet with reason: host reimage 2443166 wikitext text/x-wiki jugg8pm5b3dvjukrj3tddjbtfuroomw 2443167 2443166 2026-07-30T16:55:47Z Stashbot 7414 root@cumin1003: START - Cookbook sre.mysql.pool pool es1048: Maintenance 2443167 wikitext text/x-wiki 7vbwr651h76n9cevw35imxm82x69edt 2443168 2443167 2026-07-30T17:10:54Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1083.eqiad.wmnet with OS trixie 2443168 wikitext text/x-wiki n7ce9g9lilqktw7pg0ayfu04axawwkw 2443169 2443168 2026-07-30T17:11:06Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new security plugin settings - bking@cumin2003 - T350516 2443169 wikitext text/x-wiki lxzbvitwdiixkrb5iyo385heujgjzov 2443170 2443169 2026-07-30T17:16:56Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2081.codfw.wmnet with OS trixie 2443170 wikitext text/x-wiki mprdd1z589muun5dsaly20zsrjo2cda 2443171 2443170 2026-07-30T17:20:08Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . 2443171 wikitext text/x-wiki 2n7c1ck0z7sd2z1x4ufuqk91q2jbo6w 2443172 2443171 2026-07-30T17:23:13Z Stashbot 7414 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service 2443172 wikitext text/x-wiki 0tresj5vbs1xt7bwlefx4y17wic15aw 2443173 2443172 2026-07-30T17:24:16Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1083.eqiad.wmnet with reason: host reimage 2443173 wikitext text/x-wiki ozpavg4w877uamo0ps53bfauols3lvp 2443174 2443173 2026-07-30T17:26:08Z Stashbot 7414 inflatador: bking@apt1002 `reprepro --noskipold --component thirdparty/opensearch3 update trixie-wikimedia` T433624 2443174 wikitext text/x-wiki 9mucb8ib223tshrpwhtwaxec8jo8jfq 2443176 2443174 2026-07-30T17:29:21Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1083.eqiad.wmnet with reason: host reimage 2443176 wikitext text/x-wiki 11im3a2znfem33h3hgaiyy9574i0agb 2443177 2443176 2026-07-30T17:36:49Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply new security plugin settings - bking@cumin2003 - T350516 2443177 wikitext text/x-wiki t5owtew9je9ofnvuidnltfb6avq2ooh 2443178 2443177 2026-07-30T17:43:51Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1048: Maintenance 2443178 wikitext text/x-wiki 0o2s0larpusl93w0fcgelrgtv60b3y4 2443179 2443178 2026-07-30T17:54:57Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1083.eqiad.wmnet with OS trixie 2443179 wikitext text/x-wiki 7cbx4ik6b54ean1bg30q0zvo5an6uzz 2443180 2443179 2026-07-30T18:15:23Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.13 refs T430832 2443180 wikitext text/x-wiki 387afo8lj5g8qcr8ehx0jiq9u3cym92 2443181 2443180 2026-07-30T18:25:19Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.13 refs T430832 2443181 wikitext text/x-wiki ombpcqein7mvbzi5vefo3utsrj9w83m 2443182 2443181 2026-07-30T18:39:21Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1084.eqiad.wmnet with OS trixie 2443182 wikitext text/x-wiki iptewn7qhk66jd7cf15sykblh6i0tcu 2443183 2443182 2026-07-30T18:40:52Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2443183 wikitext text/x-wiki lp35hplc264bktr601tnwwisom6ojlc 2443184 2443183 2026-07-30T18:41:30Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2443184 wikitext text/x-wiki h4ui1pdt0z7bxttycofrpe6m8wgp04q 2443185 2443184 2026-07-30T18:52:54Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1084.eqiad.wmnet with reason: host reimage 2443185 wikitext text/x-wiki sbmf8ifklsik8bbaz9qmlzelg6gkv0n 2443186 2443185 2026-07-30T18:58:50Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1084.eqiad.wmnet with reason: host reimage 2443186 wikitext text/x-wiki 96mmk5ms4rr1hnfglok7ox2goijyizm 2443188 2443186 2026-07-30T19:20:54Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1084.eqiad.wmnet with OS trixie 2443188 wikitext text/x-wiki mc4uoi31krpglelseyhcz6tduoheklf 2443189 2443188 2026-07-30T19:23:56Z Stashbot 7414 sukhe: sukhe@lvs1019:~$ sudo systemctl restart pybal.service 2443189 wikitext text/x-wiki hiqfocj5u7fuashpbvg74y41x4xas7h 2443190 2443189 2026-07-30T19:24:05Z Stashbot 7414 sukhe: sukhe@lvs1020:~$ sudo systemctl restart pybal.service 2443190 wikitext text/x-wiki terra8kejvxbq96eobt7gbiho4ro151 2443191 2443190 2026-07-30T19:24:07Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443191 wikitext text/x-wiki lbm80d2ueull7wpvym2aej4p9uwwyd3 2443192 2443191 2026-07-30T19:24:18Z Stashbot 7414 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443192 wikitext text/x-wiki hdg6gn2wbzr15c6w0pe8i5zvwjunh9g 2443193 2443192 2026-07-30T19:25:00Z Stashbot 7414 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service 2443193 wikitext text/x-wiki nreaw5e3z1pvbo53ziz7ygr9p72gnww 2443194 2443193 2026-07-30T19:25:41Z Stashbot 7414 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service 2443194 wikitext text/x-wiki 6kl71zhyvsm35mwg0w0k57bmkx5qvnu 2443195 2443194 2026-07-30T19:29:36Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443195 wikitext text/x-wiki jwdvw22mkzznqf6vtbgn4s0fvmu89ut 2443196 2443195 2026-07-30T19:29:48Z Stashbot 7414 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443196 wikitext text/x-wiki j51ky5ieovu1yymyjwo2aii4rnetzig 2443197 2443196 2026-07-30T20:03:57Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1319521|Increase Parsoid image limit to 5000 (T430854)]] 2443197 wikitext text/x-wiki automx1jixqqqnfzoyy8j8ag6wuozr0 2443198 2443197 2026-07-30T20:05:47Z Stashbot 7414 arlolra@deploy1003: osleger, arlolra: Backport for [[gerrit:1319521|Increase Parsoid image limit to 5000 (T430854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2443198 wikitext text/x-wiki 43btsrj8xk7vw611qhpckah9lc7gyn9 2443199 2443198 2026-07-30T20:07:03Z Stashbot 7414 arlolra@deploy1003: osleger, arlolra: Continuing with deployment 2443199 wikitext text/x-wiki sp1vfsftssqxh6379maekhyqitqm0fp 2443200 2443199 2026-07-30T20:13:19Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1319521|Increase Parsoid image limit to 5000 (T430854)]] (duration: 09m 20s) 2443200 wikitext text/x-wiki 1jixfu5opovz9y6qkwdsz8iywppiqts 2443203 2443200 2026-07-30T21:15:01Z Stashbot 7414 maryum: Deployed security fix for T430601 2443203 wikitext text/x-wiki 2vw12ir20tbwv0qc2ctocpnksmucumj 2443204 2443203 2026-07-30T21:22:52Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.276.0" for 3 host(s) 2443204 wikitext text/x-wiki bvr35w1ydouqkhtt2gdg5ddks0o7bwj 2443205 2443204 2026-07-30T21:24:47Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.276.0" completed for 3 hosts 2443205 wikitext text/x-wiki 96lnqa2ouj6ph4gd8hjif6gvp9fuy7t 2443206 2443205 2026-07-30T21:35:18Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.276.1" for 3 host(s) 2443206 wikitext text/x-wiki qyqjdhfvevmoo0ease2bksgfen3zoag 2443207 2443206 2026-07-30T21:37:12Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.276.1" completed for 3 hosts 2443207 wikitext text/x-wiki 93p0c34sb8rjjlilk1g34l51piixkd3 2443210 2443207 2026-07-31T00:46:49Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1316832|Drop old support for non-temporary accounts wikis]] 2443210 wikitext text/x-wiki 2ixbtdsc9unfgdrmgm0745b9i6q80j5 2443211 2443210 2026-07-31T00:48:36Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, jforrester: Backport for [[gerrit:1316832|Drop old support for non-temporary accounts wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2443211 wikitext text/x-wiki mv3pbmt62vlhvkmcx7b3cnplxvgjl9j 2443212 2443211 2026-07-31T00:53:43Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, jforrester: Continuing with deployment 2443212 wikitext text/x-wiki nz6obwx3sv6cpxef81xbec3up0ujj8p 2443213 2443212 2026-07-31T00:57:54Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1316832|Drop old support for non-temporary accounts wikis]] (duration: 11m 04s) 2443213 wikitext text/x-wiki 641fmvpi688ldt9zei4lhq2ko7ujnx6 2443214 2443213 2026-07-31T02:00:40Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2443214 wikitext text/x-wiki odqedmi7ho0qb9c44yalgfhk2dz1646 2443215 2443214 2026-07-31T02:07:34Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 54s) 2443215 wikitext text/x-wiki nrhxdr56jbsv6l7hrgf18ck15vvq3u1 2443216 2443215 2026-07-31T06:34:25Z Stashbot 7414 filippo@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1048.eqiad.wmnet with OS trixie 2443216 wikitext text/x-wiki j9tkgl1967ainrxqbrd509s8n034z43 2443217 2443216 2026-07-31T06:44:18Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1048.eqiad.wmnet with OS trixie 2443217 wikitext text/x-wiki h5992qnzehc3031qdk6binkpnqfrl90 2443218 2443217 2026-07-31T06:48:18Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443218 wikitext text/x-wiki jy26fi93cx24d6isqem4eyn0celad8r 2443219 2443218 2026-07-31T06:48:20Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443219 wikitext text/x-wiki kc0nw8vjibw1fmefpmk7bnymc7xcu5t 2443220 2443219 2026-07-31T06:56:53Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443220 wikitext text/x-wiki q9vwf4w3s5u1rg0v8inrvljs3zntr2q 2443221 2443220 2026-07-31T06:57:01Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443221 wikitext text/x-wiki 35p631jrakv9yalqyy95hp1v8y2o09u 2443222 2443221 2026-07-31T07:09:48Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "cloudvirt1048 - filippo@cumin1003" 2443222 wikitext text/x-wiki 35fw8ugcjzy14ikywz808do932g5ajm 2443223 2443222 2026-07-31T07:09:53Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "cloudvirt1048 - filippo@cumin1003" 2443223 wikitext text/x-wiki ft8jd9t27k0acm14s7ergy2qfx6tiwy 2443224 2443223 2026-07-31T07:11:32Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "cloudvirt1048 - filippo@cumin1003" 2443224 wikitext text/x-wiki st2jm6r8xx7mtvj3vaob9hc52302qqh 2443225 2443224 2026-07-31T07:11:47Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "cloudvirt1048 - filippo@cumin1003" 2443225 wikitext text/x-wiki tw5mbrn5b9cxmttclyne3ogh1u71x5h 2443226 2443225 2026-07-31T07:13:15Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1048.eqiad.wmnet with reason: host reimage 2443226 wikitext text/x-wiki 8hqpk0lb3w5k84l9z96tbzhffcxssfr 2443227 2443226 2026-07-31T07:19:16Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1048.eqiad.wmnet with reason: host reimage 2443227 wikitext text/x-wiki jx8pb34ngqhyiupv33spgfz1cv0zaij 2443230 2443227 2026-07-31T08:00:44Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 2443230 wikitext text/x-wiki 13vjbf5h8fjv7zcgas6qfgdqfkysqts 2443231 2443230 2026-07-31T08:01:08Z Stashbot 7414 filippo@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1048 2443231 wikitext text/x-wiki nxywdy4oa6prz4013h8mh5h4lc1rsab 2443232 2443231 2026-07-31T08:06:04Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 2443232 wikitext text/x-wiki 64sbp5yly5g1ufq4ve3u12e34e1vjnj 2443234 2443232 2026-07-31T08:07:51Z Stashbot 7414 filippo@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1048 2443234 wikitext text/x-wiki gok6dc9e2ny56toevkpvr68w4vpjblz 2443235 2443234 2026-07-31T08:07:54Z Stashbot 7414 filippo@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 2443235 wikitext text/x-wiki djn77z44i0h1d1yteli2327x1rdt4fr 2443236 2443235 2026-07-31T08:08:01Z Stashbot 7414 filippo@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1048 2443236 wikitext text/x-wiki hb2y9qfflgx3nje56gd4ry19v46pc79 2443237 2443236 2026-07-31T08:11:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443237 wikitext text/x-wiki ex8dn36n4vdlscouko4mr280nx4ryr4 2443238 2443237 2026-07-31T08:11:13Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443238 wikitext text/x-wiki jtndjkhfqclchyi7x8xw8sieh3cqzvf 2443239 2443238 2026-07-31T08:35:42Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2443239 wikitext text/x-wiki oujwwgwwus54ok8oo5kd91ohyjgyu40 2443240 2443239 2026-07-31T08:36:14Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2443240 wikitext text/x-wiki 53shyf1gnq5ovnhpkhtvxfwviparzng 2443241 2443240 2026-07-31T08:37:00Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2443241 wikitext text/x-wiki 7otv2v7sfhp6vp20vl6hq03am0e87eo 2443242 2443241 2026-07-31T08:37:56Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2443242 wikitext text/x-wiki 7ptrxn1nh0tdcsov291p9u8hga7okxs 2443243 2443242 2026-07-31T08:42:03Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2443243 wikitext text/x-wiki lq5dwvwpvel7ti1235lg2p6s1flufjb 2443244 2443243 2026-07-31T08:46:59Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443244 wikitext text/x-wiki kxjorc1o213mkz0fdcs4cctm26bskib 2443245 2443244 2026-07-31T08:51:39Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . 2443245 wikitext text/x-wiki 5s23s7vecnovt6uoygl4jllxehvts9z 2443246 2443245 2026-07-31T08:57:10Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443246 wikitext text/x-wiki oyjgsltj4k13pd7lykwweprg38nq01u 2443247 2443246 2026-07-31T09:16:47Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for 10 hosts 2443247 wikitext text/x-wiki mdsi4njk8r23c7y8gf43cbw1fp9acx9 2443248 2443247 2026-07-31T09:18:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 10 hosts 2443248 wikitext text/x-wiki l83azgk2m309tpiwpwh5soy2cfxshqu 2443249 2443248 2026-07-31T09:52:03Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443249 wikitext text/x-wiki qn0qujxr9nyfzfcr8edv8f3i6laewmq 2443250 2443249 2026-07-31T10:02:12Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2443250 wikitext text/x-wiki dmcnugfihigkcq9yi3i8u5e2ua7inm7 2443251 2443250 2026-07-31T10:26:39Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2082.codfw.wmnet with OS trixie 2443251 wikitext text/x-wiki 2uwrkscmfqh72gsjkao65n00jzkj4b3 2443252 2443251 2026-07-31T10:28:29Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2443252 wikitext text/x-wiki lmhci6wtfe54pxdvriqh1cxozoy2ypf 2443254 2443252 2026-07-31T10:40:50Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage 2443254 wikitext text/x-wiki dgamj71ymmnbwahrmnnmjsx7vmw1rq3 2443255 2443254 2026-07-31T10:42:16Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2443255 wikitext text/x-wiki 4vk7dc26k1dctxo0mv62sot7b0g6kbg 2443256 2443255 2026-07-31T10:44:18Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage 2443256 wikitext text/x-wiki lxxgsweqmyrdgf9gqpdvwunx5ojv0ug 2443257 2443256 2026-07-31T11:02:40Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2082.codfw.wmnet with OS trixie 2443257 wikitext text/x-wiki thoiztkrhuduqcgixe9q09v01bm7jjx 2443258 2443257 2026-07-31T11:07:58Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T433606 2443258 wikitext text/x-wiki 1ct4nrddx45jiuqzhrl5p72h5club2j 2443261 2443258 2026-07-31T11:27:41Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T433606 2443261 wikitext text/x-wiki kuoq9lecpz9v390vugzkrko4806knne 2443262 2443261 2026-07-31T11:28:05Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T433606 2443262 wikitext text/x-wiki 2uhzqzyb0eqrbay4b5ynnx0nv17i5j9 2443265 2443262 2026-07-31T11:37:42Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 6515 2443265 wikitext text/x-wiki r7bmb1v8bvc6wvmxc469lutpv82e8xk 2443266 2443265 2026-07-31T11:42:02Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 6515 2443266 wikitext text/x-wiki 2lrbj2ymrmicwf8xjhkjam1y68lrnq5 2443270 2443266 2026-07-31T11:53:40Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T433606 2443270 wikitext text/x-wiki ayiho4pmlacl7qidhqw5i2ckyxxbv21 498 14256 2443202 2438811 2026-07-30T20:48:01Z Stashbot 7414 wikibugs: Updated channels.yaml to: cab24068909455ccc3a11fb58066a1835d53c4024e9a32574b3894a2bcc3f767 2443202 wikitext text/x-wiki kvv61mgvdorlmc33hvam297y186vavt 0 17290 2443175 2442807 2026-07-30T17:28:57Z Stashbot 7414 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover/mediawiki-libs-node-cssjanus/ # T424419 2443175 wikitext text/x-wiki px9qdq060f4nmk69pphdxcl8bmy84tz 12 441210 2443228 1764906 2026-07-31T07:20:41Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Help:Toolforge/OpenSearch]] 2443228 wikitext text/x-wiki lj014octcp7yuxbad5w0hx43nb167ac 0 443680 2443229 2442178 2026-07-31T07:54:44Z JWodstrcil (WMF) 24826 2443229 wikitext text/x-wiki ecr9wxax83mhliztdkivee2sn4s2kcd 2443233 2443229 2026-07-31T08:06:10Z JWodstrcil (WMF) 24826 add warning about CI update and the Rakefile 2443233 wikitext text/x-wiki 7k3yjndcee4b7lhu8jl4sj8zym9gvor 0 448962 2442989 2442372 2026-07-30T12:31:29Z JWodstrcil (WMF) 24826 add update instructions for cert-manager 2442989 wikitext text/x-wiki ke7un7ez9kndhzcvoec7i8p9rlljasb 0 449160 2443000 2442923 2026-07-30T13:00:11Z Dexbot 30554 Bot: Updating the report 2443000 wikitext text/x-wiki a05cjbjv2oxmgjv8fr3wk13ssr52n8k 2443019 2443000 2026-07-30T13:18:16Z Dexbot 30554 Bot: Updating the report 2443019 wikitext text/x-wiki jr9mk9inj1tl9tqubv3jn7hitr0qrz9 2443148 2443019 2026-07-30T15:54:40Z Dexbot 30554 Bot: Updating the report 2443148 wikitext text/x-wiki hnfpz2i3g0beoseovjbipdu4djmbg4x 2443209 2443148 2026-07-31T00:01:47Z Dexbot 30554 Bot: Updating the report 2443209 wikitext text/x-wiki 4kvq6i8k8mgew1w0hhwhp3wo64782l8 0 451533 2443024 2441792 2026-07-30T13:27:15Z APuthin-WMF 52425 Updating to August 2026 meeting agenda 2443024 wikitext text/x-wiki nyib4ddtnf3zsl4ul4kmq9akt1jc2on 116 453906 2443201 2442212 2026-07-30T20:42:16Z Gitlabaccountapprovalbot 37332 horaciocolbert was rejected. 2443201 wikitext text/x-wiki 5y2niwpvzax9e5vqm0g2bev0qpzspmx 0 458576 2443253 2439667 2026-07-31T10:36:31Z BRouberol-WMF 39141 /* Partial dumps */ 2443253 wikitext text/x-wiki qfywau1vff7wwvol331jotwcnyrf9u4 0 459128 2443093 2353734 2026-07-30T14:49:16Z SSapaty (WMF) 28955 /* Timeline */ add timeline related activities 2443093 wikitext text/x-wiki 100ic7efuzuml0yridbtxsbf4wkso7k 2443141 2443093 2026-07-30T15:46:27Z JJMC89 7474 was not on the list 2443141 wikitext text/x-wiki meznx681e95w1hda2c7dl7tu0o772zw 0 459699 2443038 2442041 2026-07-30T13:53:30Z Stashbot 7414 damilare: donorwiki upgraded from d59684ed to 0117d21d 2443038 wikitext text/x-wiki e44oju1zio1whim92trbi0l4b9ydzlo 0 460502 2443023 2026-07-30T13:26:05Z APuthin-WMF 52425 Adding July meeting notes 2443023 wikitext text/x-wiki r8d9m0khfv5o3vtf4ko565ipxvb4dr1 0 460503 2443065 2026-07-30T14:17:08Z FNegri-WMF 32595 create runbook for new alert 2443065 wikitext text/x-wiki 9fqq7efztzpobv7uqgl5xu682h9sc4j 0 460504 2443066 2026-07-30T14:17:49Z FNegri-WMF 32595 create runbook for new alert 2443066 wikitext text/x-wiki rhc24527z5ddsjvnl4rbgjeh9du7tw8 0 460505 2443067 2026-07-30T14:19:55Z FNegri-WMF 32595 create runbook for new alert 2443067 wikitext text/x-wiki anj997cczuqvdldvvzm0x1qtuqzx8id 3 460506 2443159 2026-07-30T16:16:45Z StrikerBot 8475 Welcome to Toolforge! 2443159 wikitext text/x-wiki tpnlaa9e6hqerrqut2l2naog9gkrw4l