Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.5 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2422528 2422219 2026-06-02T15:45:27Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld 2422528 wikitext text/x-wiki kxsjmv4rw1vb0d5cox3msdkq9dj5u4w 2422543 2422528 2026-06-02T15:59:57Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld 2422543 wikitext text/x-wiki 5647ai8w7agygvon5j2sz71paxmln8k 0 4108 2422306 2422206 2026-06-02T12:25:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1296563]] to Tuesday, June 02 UTC afternoon backport window 2422306 wikitext text/x-wiki ixkfswprttfpelt6g8gwxzpd9irrcee 2422307 2422306 2026-06-02T12:25:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1296566]] to Tuesday, June 02 UTC afternoon backport window 2422307 wikitext text/x-wiki 7pkfcsuuebx4etahjm9nonkvkmp4wv4 2422317 2422307 2026-06-02T12:39:47Z ScheduleDeploymentBot 37566 Add [[gerrit:1295574]] to Tuesday, June 02 UTC afternoon backport window 2422317 wikitext text/x-wiki r765vmpptyex3sh5j5tvngk5prxror1 2422337 2422317 2026-06-02T12:56:08Z SFrench-WMF 37362 /* {{Deployment_day|date=2026-06-02}} */ UTC-late infra window 2422337 wikitext text/x-wiki tm8psu7yrmno5rgrsu1flinzz9thx1n 2422346 2422337 2026-06-02T13:03:04Z ScheduleDeploymentBot 37566 Add [[gerrit:1296580]] to Tuesday, June 02 UTC afternoon backport window 2422346 wikitext text/x-wiki 7js3ugfckp3jhs246z8hu02nzzlo65k 2422426 2422346 2026-06-02T14:17:55Z Msz2001 16141 Rescheduling today's UTC afternoon backport window to tomorrow UTC morning (CI issues) 2422426 wikitext text/x-wiki 0imiik251bbxlp2f9kc22a61wbi2iay 2422514 2422426 2026-06-02T15:30:44Z A826 34037 /* {{Deployment_day|date=2026-06-03}} */ removed merged patched 2422514 wikitext text/x-wiki 7r5011qr6r3t85njj9s2gfut1wppm0b 2422545 2422514 2026-06-02T16:04:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1295968]] to Wednesday, June 03 UTC morning backport window 2422545 wikitext text/x-wiki b8r794jmvkveqdtvvrilyv00cz9kyir 2422556 2422545 2026-06-02T16:13:11Z ScheduleDeploymentBot 37566 Add [[gerrit:1296627]] to Wednesday, June 03 UTC morning backport window 2422556 wikitext text/x-wiki irm8g0g7dpsbg9xwm0u4i5xfeqvetrj 2422566 2422556 2026-06-02T16:32:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1296631]] to Wednesday, June 03 UTC afternoon backport window 2422566 wikitext text/x-wiki cc2tkhgvi5gx4wyv2od1fyywcda39sq 2422579 2422566 2026-06-02T16:49:45Z ScheduleDeploymentBot 37566 Add [[gerrit:1296632]] to Wednesday, June 03 UTC morning backport window 2422579 wikitext text/x-wiki 0bfleahei4vw8amnn7q5rx44met4n5g 2422606 2422579 2026-06-02T17:38:38Z ScheduleDeploymentBot 37566 Add [[gerrit:1296649]] to Wednesday, June 03 UTC afternoon backport window 2422606 wikitext text/x-wiki jc71yjiqs9t16b66d2xz9y70gtfx0ng 2422767 2422606 2026-06-03T01:27:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1296703]] to Wednesday, June 03 UTC morning backport window 2422767 wikitext text/x-wiki kgbd43163tc0chyhmidbpp4x7wjy7i2 2422811 2422767 2026-06-03T06:56:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1296713]] to Wednesday, June 03 UTC morning backport window 2422811 wikitext text/x-wiki gg87xjagy52lsasoarnmflqbfmecsjf 2422828 2422811 2026-06-03T07:46:41Z ScheduleDeploymentBot 37566 Add [[gerrit:1296635]] to Wednesday, June 03 UTC morning backport window 2422828 wikitext text/x-wiki gn4yptynk3a8ac6yckp9ph3lcmdy77m 2422891 2422828 2026-06-03T08:40:34Z ScheduleDeploymentBot 37566 Add [[gerrit:1297058]] to Wednesday, June 03 UTC afternoon backport window 2422891 wikitext text/x-wiki 6ms6r8cyn645vgktlya8jkwhyw6kh9c 110 4516 2422774 2189550 2026-06-03T01:58:37Z Krinkle 44 [[Category:Decommissioned services]] 2422774 wikitext text/x-wiki dhpjmqq91o8wsqsizpd2xi1hu6wu9cu 0 4526 2422775 60419 2026-06-03T01:59:20Z Krinkle 44 [[Category:Decommissioned services]] 2422775 wikitext text/x-wiki non0yxui5sxi1nf126qs1zv6pujadlb 498 6726 2422515 2422032 2026-06-02T15:31:31Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld 2422515 wikitext text/x-wiki ajlhxcwg1cq2j1l6x6vgahdq218ueij 2422527 2422515 2026-06-02T15:44:54Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld 2422527 wikitext text/x-wiki kuskubm8e2koc7jd0a1iq4jj4wf8g7g 0 7919 2422278 2422277 2026-06-02T12:00:22Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage 2422278 wikitext text/x-wiki rvir87r4hwtzvi7cxh1pzwm2far9yyi 2422279 2422278 2026-06-02T12:01:48Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS trixie 2422279 wikitext text/x-wiki fn3tqyo0bz8mp1z4qq4r1g3ddy016lt 2422280 2422279 2026-06-02T12:02:43Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage 2422280 wikitext text/x-wiki dny6905y3mjnl6emodicbbk2vsp6jb8 2422281 2422280 2026-06-02T12:04:42Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage 2422281 wikitext text/x-wiki t2q7luidneyo6ecerf54omyx1ueuq91 2422282 2422281 2026-06-02T12:05:20Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2049.codfw.wmnet with reason: host reimage 2422282 wikitext text/x-wiki r22i0boxs04lxr6yqams965owt1tmaw 2422283 2422282 2026-06-02T12:07:11Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296532|hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533|hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] 2422283 wikitext text/x-wiki t2td4m7hbql8lncqi7hq23bt5kgckdk 2422284 2422283 2026-06-02T12:07:16Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet 2422284 wikitext text/x-wiki oyewtl7v4ccljyymf9bw0j47vudpala 2422285 2422284 2026-06-02T12:07:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T426633)', diff saved to https://phabricator.wikimedia.org/P93536 and previous config saved to /var/cache/conftool/dbconfig/20260602-120728-fceratto.json 2422285 wikitext text/x-wiki 5nv8u70l9xbz1n3r4bkf7yr0nj0b8mj 2422286 2422285 2026-06-02T12:07:49Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance 2422286 wikitext text/x-wiki sem5r5dsjqddlkkpcthtl6biw88acfe 2422287 2422286 2026-06-02T12:07:54Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage 2422287 wikitext text/x-wiki 8kawrofqpuis7zcpeouuvp20tgjyekp 2422288 2422287 2026-06-02T12:08:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 (T426633)', diff saved to https://phabricator.wikimedia.org/P93537 and previous config saved to /var/cache/conftool/dbconfig/20260602-120755-fceratto.json 2422288 wikitext text/x-wiki ahlplj81inp9nm8mg7cnv6wdpvvd2tk 2422289 2422288 2026-06-02T12:09:02Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296532|hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533|hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422289 wikitext text/x-wiki e1kuzvd12auaptn4cgougavbo248vhf 2422290 2422289 2026-06-02T12:09:57Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Switch maintenance 2422290 wikitext text/x-wiki ebqljfubxpk9ahbfcfz0tx22irdzpwt 2422291 2422290 2026-06-02T12:10:01Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2161: Migration of db2161.codfw.wmnet completed 2422291 wikitext text/x-wiki 92h1o9ca2pp6ztal7xsbtclwg31raiv 2422292 2422291 2026-06-02T12:11:06Z Stashbot 7414 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt with reason: Switch maintenance 2422292 wikitext text/x-wiki 8f48anf7hl9yl7y1u4mknpy9p7nh5kw 2422293 2422292 2026-06-02T12:11:45Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2049.codfw.wmnet with reason: host reimage 2422293 wikitext text/x-wiki b8j3l8cseopmu2gr8af7nnww97b756v 2422294 2422293 2026-06-02T12:11:55Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2422294 wikitext text/x-wiki hrrvi85suyl74aflpe7p9ri2na9gqss 2422295 2422294 2026-06-02T12:14:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T426633)', diff saved to https://phabricator.wikimedia.org/P93539 and previous config saved to /var/cache/conftool/dbconfig/20260602-121451-fceratto.json 2422295 wikitext text/x-wiki h9ldzwjpqy4inzr0gn5lzto2o52bezm 2422297 2422295 2026-06-02T12:16:14Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296532|hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533|hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] (duration: 09m 02s) 2422297 wikitext text/x-wiki svz71h5tno9a47tfzfsay8dj0mjwc5n 2422299 2422297 2026-06-02T12:17:35Z Stashbot 7414 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie 2422299 wikitext text/x-wiki 7u4mm5zdl9batxxvgj76r2wfs98plc6 2422300 2422299 2026-06-02T12:20:07Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1058.eqiad.wmnet with OS trixie 2422300 wikitext text/x-wiki fbzlsyupsnd8omdrn727hlez93qhg2h 2422301 2422300 2026-06-02T12:20:52Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet 2422301 wikitext text/x-wiki 6zqpgehgmo7ofqubozldo3lrjtrew32 2422302 2422301 2026-06-02T12:20:53Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1060.eqiad.wmnet with OS trixie 2422302 wikitext text/x-wiki gka9jkkw01u9yo9lgf9htdyt6bgwrww 2422303 2422302 2026-06-02T12:21:30Z Stashbot 7414 XioNoX: reboot lsw1-a3-codfw for software upgrade - T427301 2422303 wikitext text/x-wiki i99r1pboygr4nnxlnnng8r0659fx5ri 2422304 2422303 2026-06-02T12:24:39Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1059.eqiad.wmnet with OS trixie 2422304 wikitext text/x-wiki 5q3y8fmxn30slmjz3g2hz2ekfp07bpm 2422305 2422304 2026-06-02T12:25:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93542 and previous config saved to /var/cache/conftool/dbconfig/20260602-122459-fceratto.json 2422305 wikitext text/x-wiki lidz7fvr4evgzi4fwgpksd9eamve2pk 2422308 2422305 2026-06-02T12:28:45Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2049.codfw.wmnet with OS trixie 2422308 wikitext text/x-wiki 4hn8s7lncfmehj5m9phqwvpd2ja1xgy 2422309 2422308 2026-06-02T12:29:30Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1061.eqiad.wmnet with OS trixie 2422309 wikitext text/x-wiki 57ncifzuyz6kixhcuk8xmho9m4hcku8 2422311 2422309 2026-06-02T12:31:42Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422311 wikitext text/x-wiki mqhuykznznfdfh1yu0jqrz18vo8fsft 2422312 2422311 2026-06-02T12:31:54Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2049: repool after upgrade 2422312 wikitext text/x-wiki ploowts9l91qbdxegdm59p2dai6zkjt 2422313 2422312 2026-06-02T12:33:41Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage 2422313 wikitext text/x-wiki cixdm9m7k0gi109hq1hez32a4fthpqd 2422314 2422313 2026-06-02T12:33:51Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2422314 wikitext text/x-wiki o2i0hm5kjks26l0xfpun6rznoc4x5kw 2422315 2422314 2026-06-02T12:35:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93545 and previous config saved to /var/cache/conftool/dbconfig/20260602-123505-fceratto.json 2422315 wikitext text/x-wiki eb9ydn83noi2jwbdc27xy398m0cwqru 2422316 2422315 2026-06-02T12:35:44Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage 2422316 wikitext text/x-wiki qpe10xoloj2vwih3mj1nr7ncbk9kb9f 2422318 2422316 2026-06-02T12:41:16Z Stashbot 7414 topranks: enable bgp graceful-shutdown in underlay on ssw1-a1-codfw T427301 2422318 wikitext text/x-wiki fna6emz3mlpbk1bft3m3zzl302k17dl 2422319 2422318 2026-06-02T12:42:11Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage 2422319 wikitext text/x-wiki 4ih5fxh2h77zr7f1vwqf3kpiu7aug6v 2422320 2422319 2026-06-02T12:42:14Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage 2422320 wikitext text/x-wiki pqhmb06d0sicdrm9j8d6kfcni2v8lrn 2422321 2422320 2026-06-02T12:42:53Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422321 wikitext text/x-wiki f7zflwg9epky7uswt2wxtxcb3fk9ggs 2422322 2422321 2026-06-02T12:43:29Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1060.eqiad.wmnet with OS trixie 2422322 wikitext text/x-wiki 095iftu75aq6vg9htb325w3pzee4ryg 2422323 2422322 2026-06-02T12:45:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T426633)', diff saved to https://phabricator.wikimedia.org/P93547 and previous config saved to /var/cache/conftool/dbconfig/20260602-124512-fceratto.json 2422323 wikitext text/x-wiki bcfs76fq3ifaoxt9ez3klgi7kzcfzfa 2422325 2422323 2026-06-02T12:45:35Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance 2422325 wikitext text/x-wiki hmgez4xatm2hb1whzv8xty5k75yqypg 2422326 2422325 2026-06-02T12:45:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 (T426633)', diff saved to https://phabricator.wikimedia.org/P93548 and previous config saved to /var/cache/conftool/dbconfig/20260602-124541-fceratto.json 2422326 wikitext text/x-wiki brzagxl6bz6qrlzs5dv9n7nqw8co8ew 2422327 2422326 2026-06-02T12:47:19Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1062.eqiad.wmnet with OS trixie 2422327 wikitext text/x-wiki rrwlsx6gsg29szxr07jepee7c1hdtja 2422328 2422327 2026-06-02T12:48:16Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt 2422328 wikitext text/x-wiki pgv7htdakw2o3vjcn4ydnqgy5tmsd9g 2422329 2422328 2026-06-02T12:48:18Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt 2422329 wikitext text/x-wiki n0ejabwl9s9y2a8kbioz4cc7trk67vk 2422330 2422329 2026-06-02T12:49:55Z Stashbot 7414 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1061.eqiad.wmnet with OS trixie 2422330 wikitext text/x-wiki 1nkq7fih1ejhc6w5cpcmixuxnt8drt7 2422331 2422330 2026-06-02T12:50:10Z Stashbot 7414 topranks: enable bgp graceful-shutdown in overlay on ssw1-a1-codfw T427301 2422331 wikitext text/x-wiki 8qp91rcw0u76t0ogkrox8egwptdvjdv 2422332 2422331 2026-06-02T12:52:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T426633)', diff saved to https://phabricator.wikimedia.org/P93550 and previous config saved to /var/cache/conftool/dbconfig/20260602-125223-fceratto.json 2422332 wikitext text/x-wiki 9d21le7p7fxwb9c1dynh65rt5kpwws3 2422333 2422332 2026-06-02T12:52:55Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422333 wikitext text/x-wiki 94mafpoup5tqhpbnuqwusg3ovg7xv6m 2422334 2422333 2026-06-02T12:54:57Z Stashbot 7414 topranks: shutdown sub-interfaces on cr1-codfw et-1/1/5 for row A/B vlans T427301 2422334 wikitext text/x-wiki jo01ozwggn8df9x7qg9792z3qgqiiu9 2422335 2422334 2026-06-02T12:55:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Migration of db2161.codfw.wmnet completed 2422335 wikitext text/x-wiki ebjh8cigtpfpophhbifoe90fskeopis 2422336 2422335 2026-06-02T12:55:34Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2422336 wikitext text/x-wiki cys15429m9msvwektan9xth2fhp9bls 2422339 2422336 2026-06-02T12:57:37Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2422339 wikitext text/x-wiki 0ieeh2ppcyr62819vek8snnlgwtymrj 2422340 2422339 2026-06-02T12:57:41Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2422340 wikitext text/x-wiki sstipw6kxv31pzizfnholo1ld6pp8e7 2422341 2422340 2026-06-02T12:57:45Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2422341 wikitext text/x-wiki 9mwdigroln0ypok3zmme2z9flx4zhix 2422342 2422341 2026-06-02T12:57:52Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2422342 wikitext text/x-wiki 0epjd87m6u5537gkcquz7okr0tv3lrx 2422343 2422342 2026-06-02T12:59:49Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage 2422343 wikitext text/x-wiki kvynf98fagj1uaydn0kudj65lv7jo8a 2422344 2422343 2026-06-02T13:02:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93553 and previous config saved to /var/cache/conftool/dbconfig/20260602-130230-fceratto.json 2422344 wikitext text/x-wiki mzearhnmj8l02brufmc28chzlm3e92b 2422345 2422344 2026-06-02T13:02:40Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream servers 2422345 wikitext text/x-wiki duzcbbgu5upo5ycovc7t736r0l4ej69 2422347 2422345 2026-06-02T13:03:13Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage 2422347 wikitext text/x-wiki 0m3dsomdkvcssa72imf3cj6pdt9ryyj 2422348 2422347 2026-06-02T13:03:16Z Stashbot 7414 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/2 towards lsw1-a3-codfw T427301 2422348 wikitext text/x-wiki asdiltfufiby7ik5tuh48kpwuf117tk 2422349 2422348 2026-06-02T13:03:18Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1001.eqiad.wmnet with OS trixie 2422349 wikitext text/x-wiki g9zzyeg0ez8xsc8d6826ob6l9btbaom 2422350 2422349 2026-06-02T13:03:45Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream servers 2422350 wikitext text/x-wiki jtdizd7yllssrtdrtk21che2mio0bjo 2422351 2422350 2026-06-02T13:04:20Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422351 wikitext text/x-wiki 9pdw54r1bn8r2kf5wj58usdsoznzes8 2422352 2422351 2026-06-02T13:04:22Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422352 wikitext text/x-wiki 5viawug9ji9qokvohxrkwan0fey1udy 2422353 2422352 2026-06-02T13:04:57Z Stashbot 7414 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie 2422353 wikitext text/x-wiki rwqnh1or87njpvzpc3udxb54q8vuf3i 2422354 2422353 2026-06-02T13:06:05Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS trixie 2422354 wikitext text/x-wiki 8t0q4k0res5n1n00hinf4vnkf2usi8c 2422355 2422354 2026-06-02T13:07:26Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie 2422355 wikitext text/x-wiki d13d0dev86dzgtczsg6dpyfk8ko1xrn 2422356 2422355 2026-06-02T13:07:28Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie 2422356 wikitext text/x-wiki 452xm40ce3t6kpvq22pd6w3ozgsy0nc 2422357 2422356 2026-06-02T13:11:43Z Stashbot 7414 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2422357 wikitext text/x-wiki i2z1gl2e9k55c0dtcs4r3wo372h0x6y 2422358 2422357 2026-06-02T13:12:13Z Stashbot 7414 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2422358 wikitext text/x-wiki ng3q5402ch1iqj6xpvtzlwcxw0rcqau 2422359 2422358 2026-06-02T13:12:22Z Stashbot 7414 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2422359 wikitext text/x-wiki rkiqvbcng9riw1w0g4zm76c6wjjd0qo 2422360 2422359 2026-06-02T13:12:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93554 and previous config saved to /var/cache/conftool/dbconfig/20260602-131238-fceratto.json 2422360 wikitext text/x-wiki 7xchuxavfptavnjec5zv5s9w8zo5npd 2422361 2422360 2026-06-02T13:12:49Z Stashbot 7414 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2422361 wikitext text/x-wiki a6z6l3o4qd39ppo53ke5r8g6pctemg6 2422362 2422361 2026-06-02T13:13:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422362 wikitext text/x-wiki rl0x0bk8txz7gl6w904mgoaruyil7xc 2422363 2422362 2026-06-02T13:13:24Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Upgrading db1167.eqiad.wmnet 2422363 wikitext text/x-wiki k5s5t9j9h86efr8t48gzx65wbi9m7ou 2422364 2422363 2026-06-02T13:13:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Upgrading db1167.eqiad.wmnet 2422364 wikitext text/x-wiki 0qyz3qx07axjbs91r0p5oxmnxg6l0vw 2422365 2422364 2026-06-02T13:15:32Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2422365 wikitext text/x-wiki flpqvqhnzxam0j8ob4n5regkxq0jzhs 2422366 2422365 2026-06-02T13:16:20Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1167.eqiad.wmnet with OS trixie 2422366 wikitext text/x-wiki mxiqbdbcieo8ongjg97s0l6698ot0u4 2422367 2422366 2026-06-02T13:17:01Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2422367 wikitext text/x-wiki 13m8nv76ha8eb7iqpphr4npjx4vym9x 2422368 2422367 2026-06-02T13:17:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2049: repool after upgrade 2422368 wikitext text/x-wiki m0vza9cq628he02on2ncg6mz3rc7fcr 2422369 2422368 2026-06-02T13:18:54Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage 2422369 wikitext text/x-wiki 428kuk74mttl1zgxuiuyd7ul2l1jegl 2422370 2422369 2026-06-02T13:19:47Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1062.eqiad.wmnet with OS trixie 2422370 wikitext text/x-wiki 8k3c5829um0w83r9rzmwc45hpuxbcy1 2422371 2422370 2026-06-02T13:19:52Z Stashbot 7414 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie 2422371 wikitext text/x-wiki 66ey98uvffgdoxc6zfymwg4xb7j94c8 2422372 2422371 2026-06-02T13:20:20Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1064.eqiad.wmnet with OS trixie 2422372 wikitext text/x-wiki 9c3zq2l4sh8iqi3594hq73tzdd5uzf7 2422373 2422372 2026-06-02T13:22:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T426633)', diff saved to https://phabricator.wikimedia.org/P93557 and previous config saved to /var/cache/conftool/dbconfig/20260602-132246-fceratto.json 2422373 wikitext text/x-wiki effpmg6vgl4h1nqs42tsmsjwvlkujjl 2422374 2422373 2026-06-02T13:23:08Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance 2422374 wikitext text/x-wiki 6d9m0n4nlyrcg8xd0ns92otv2odpnl8 2422375 2422374 2026-06-02T13:23:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 (T426633)', diff saved to https://phabricator.wikimedia.org/P93558 and previous config saved to /var/cache/conftool/dbconfig/20260602-132314-fceratto.json 2422375 wikitext text/x-wiki gmbotixit5854l66ahim78uvgjsr1qi 2422376 2422375 2026-06-02T13:23:50Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage 2422376 wikitext text/x-wiki b4p0hwauy1o1w23nu9utomhzzkrj9bh 2422377 2422376 2026-06-02T13:24:42Z Stashbot 7414 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/4 towards lsw1-a5-codfw T427301 2422377 wikitext text/x-wiki pke05ysn3e1nsi4vua21ym10h2lxhjh 2422378 2422377 2026-06-02T13:25:39Z Stashbot 7414 slyngshede@dns1004: START - running authdns-update 2422378 wikitext text/x-wiki qjsqxz442kkgdkhgq4y8ktw3n52c7ya 2422379 2422378 2026-06-02T13:27:22Z Stashbot 7414 slyngshede@dns1004: END - running authdns-update 2422379 wikitext text/x-wiki tb25bbdvb4ilf05nyh9gi1yna7ec2lx 2422380 2422379 2026-06-02T13:30:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T426633)', diff saved to https://phabricator.wikimedia.org/P93559 and previous config saved to /var/cache/conftool/dbconfig/20260602-132959-fceratto.json 2422380 wikitext text/x-wiki mvoau1q7r7ssrtyeef8b6nq947349g6 2422381 2422380 2026-06-02T13:31:25Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage 2422381 wikitext text/x-wiki hrrewkjjhqvuzljpnz36r7p84key6ns 2422382 2422381 2026-06-02T13:32:46Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage 2422382 wikitext text/x-wiki 72sdswtu19noefljx64fw8x3vpuyfgu 2422383 2422382 2026-06-02T13:33:17Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply 2422383 wikitext text/x-wiki 71ssrdikadtbgx5vs5tlktzsa6v6mao 2422384 2422383 2026-06-02T13:33:19Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply 2422384 wikitext text/x-wiki kvawfo78ivu1c9smnwb9axdslpzei9j 2422385 2422384 2026-06-02T13:34:05Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply 2422385 wikitext text/x-wiki e99zpt9r8c8tp5pv9m7ican0ran7wtb 2422386 2422385 2026-06-02T13:34:11Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply 2422386 wikitext text/x-wiki iqz1bqwjkdg2k37r4roqq1zexq48ap9 2422387 2422386 2026-06-02T13:35:14Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie 2422387 wikitext text/x-wiki djddmm9qoyytmqmyxu50o4388f9slwl 2422388 2422387 2026-06-02T13:35:19Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie 2422388 wikitext text/x-wiki gvsdfhoh3umfvnbocyrb1w0k7ny3jrj 2422389 2422388 2026-06-02T13:38:44Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage 2422389 wikitext text/x-wiki qobcms29ekgksc8s5glffcuoem15kwn 2422390 2422389 2026-06-02T13:40:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93560 and previous config saved to /var/cache/conftool/dbconfig/20260602-134007-fceratto.json 2422390 wikitext text/x-wiki ithg0mkfauis7zi2de557dt66hoqjfn 2422391 2422390 2026-06-02T13:40:38Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS trixie 2422391 wikitext text/x-wiki d4wnebfrwbsj61ne1thoaervjab9t85 2422392 2422391 2026-06-02T13:42:53Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage 2422392 wikitext text/x-wiki m8gyn2onv52k4dwtaphqp8e0n0jf8g6 2422393 2422392 2026-06-02T13:43:49Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS trixie 2422393 wikitext text/x-wiki ls5ys6ve3f92tyxs9cm16clkroxgkaz 2422394 2422393 2026-06-02T13:47:19Z Stashbot 7414 topranks: revert all config to normal on cr1-codfw and ssw1-a1-codfw 2422394 wikitext text/x-wiki ro5uxy7k6nqpi08fk0ps67per4x3i4c 2422395 2422394 2026-06-02T13:50:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93561 and previous config saved to /var/cache/conftool/dbconfig/20260602-135015-fceratto.json 2422395 wikitext text/x-wiki 2ubsi5wpvv0hwzub8l0ednmy5vixwik 2422396 2422395 2026-06-02T13:51:07Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422396 wikitext text/x-wiki sm76cjfwndc4q8f5ral85fonrmn08pd 2422397 2422396 2026-06-02T13:51:52Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422397 wikitext text/x-wiki 655nlqs3rv3zbzt1bdz0obrgq53qjgv 2422398 2422397 2026-06-02T13:55:55Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1167.eqiad.wmnet with OS trixie 2422398 wikitext text/x-wiki 2p5bzf593vx6cr2sb6uwd7slglxu7et 2422399 2422398 2026-06-02T13:56:28Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage 2422399 wikitext text/x-wiki lxgiezkzmkv62qass3hvio2otdiv6z2 2422401 2422399 2026-06-02T14:00:01Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1064.eqiad.wmnet with OS trixie 2422401 wikitext text/x-wiki d1enlxi7rubtigsk1ovtrhrmc6hx1od 2422402 2422401 2026-06-02T14:00:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T426633)', diff saved to https://phabricator.wikimedia.org/P93562 and previous config saved to /var/cache/conftool/dbconfig/20260602-140022-fceratto.json 2422402 wikitext text/x-wiki bz7olg5lo6058mw0833ocrbb3u11mvy 2422403 2422402 2026-06-02T14:00:26Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet 2422403 wikitext text/x-wiki pstzuas6ymjfzcgfnhzyhls0rufg7nw 2422404 2422403 2026-06-02T14:00:39Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet 2422404 wikitext text/x-wiki stx0h97za80addu8lfkh36brsotxv0h 2422405 2422404 2026-06-02T14:00:46Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage 2422405 wikitext text/x-wiki n1mtaxd3gsg75b61jtldazlrnalnujc 2422406 2422405 2026-06-02T14:01:12Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS trixie 2422406 wikitext text/x-wiki g3ewa8u8pn97rr1d4cumeo3yndbcwct 2422407 2422406 2026-06-02T14:01:14Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance 2422407 wikitext text/x-wiki b23gn07x76neinmg19vspwkyyjkqhdm 2422408 2422407 2026-06-02T14:01:34Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance 2422408 wikitext text/x-wiki hzz8ueagdbd85gir0jcgnj6uxjxvs1v 2422409 2422408 2026-06-02T14:01:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1156 (T426633)', diff saved to https://phabricator.wikimedia.org/P93563 and previous config saved to /var/cache/conftool/dbconfig/20260602-140140-fceratto.json 2422409 wikitext text/x-wiki 1yo11wucgijam0dusoca32mfhmqlfjv 2422411 2422409 2026-06-02T14:05:27Z Stashbot 7414 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422411 wikitext text/x-wiki js4ew0urh28a7wyd8b6si6f4w30hyfz 2422413 2422411 2026-06-02T14:08:44Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete growthexperiments-homepage-variant # T417621 2422413 wikitext text/x-wiki kpg18u1qdp262gf1hw4uxgpenrv0he4 2422414 2422413 2026-06-02T14:09:05Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1048.eqiad.wmnet 2422414 wikitext text/x-wiki 5qft20acwlxk79p2se5vwh39moegym2 2422415 2422414 2026-06-02T14:09:08Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete --nowarn growthexperiments-homepage-variant # T417621 2422415 wikitext text/x-wiki bra6u53x53xuqyorl5kqi8oblwajrvy 2422416 2422415 2026-06-02T14:10:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T426633)', diff saved to https://phabricator.wikimedia.org/P93564 and previous config saved to /var/cache/conftool/dbconfig/20260602-141019-fceratto.json 2422416 wikitext text/x-wiki iichnofyuoaa0jg2sw9r5kccneghep1 2422418 2422416 2026-06-02T14:13:56Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage 2422418 wikitext text/x-wiki hoijpa7a4voplf2wh2ryw174mgmio2x 2422419 2422418 2026-06-02T14:14:52Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2422419 wikitext text/x-wiki cdpo1096hoyqwt7didolmcm573ze5v6 2422420 2422419 2026-06-02T14:15:49Z Stashbot 7414 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie 2422420 wikitext text/x-wiki ixknn23l64b8vjankqa079a0dv31azb 2422421 2422420 2026-06-02T14:16:47Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS trixie 2422421 wikitext text/x-wiki mzcm1ta52atyn0jfs9u6ervkinvt8jc 2422422 2422421 2026-06-02T14:17:04Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1167.eqiad.wmnet 2422422 wikitext text/x-wiki scuh0ujhqdp6sxp26gydxzkis5bnbfk 2422423 2422422 2026-06-02T14:17:06Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1167.eqiad.wmnet 2422423 wikitext text/x-wiki 5gjv9u3wmgke0te362k3zgdefl6t6ot 2422424 2422423 2026-06-02T14:17:21Z Stashbot 7414 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie 2422424 wikitext text/x-wiki fctfx6x5aalm8q1qjz53uw2m2irdu0f 2422425 2422424 2026-06-02T14:17:23Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS trixie 2422425 wikitext text/x-wiki sjzj2rg6xneujfwg53hmygfiamtrb07 2422427 2422425 2026-06-02T14:20:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93566 and previous config saved to /var/cache/conftool/dbconfig/20260602-142027-fceratto.json 2422427 wikitext text/x-wiki 78f59rs5k0b6jon38a1p3p3imas6z7c 2422428 2422427 2026-06-02T14:20:31Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage 2422428 wikitext text/x-wiki tkoqtykmxjdb1ugdp5u6auhhdgps58w 2422429 2422428 2026-06-02T14:21:19Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Repooling after Icing wait-for-green timeout 2422429 wikitext text/x-wiki ms9zlqm5iv60v4zto9y5jqd7q2nqpws 2422431 2422429 2026-06-02T14:25:01Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422431 wikitext text/x-wiki nmlbfmwburi4i15cecd5octefg3xd96 2422433 2422431 2026-06-02T14:30:20Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage 2422433 wikitext text/x-wiki oqe08r201t1t73womqpyutmrd7x4tm0 2422434 2422433 2026-06-02T14:30:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93569 and previous config saved to /var/cache/conftool/dbconfig/20260602-143035-fceratto.json 2422434 wikitext text/x-wiki jc3pligy5ndw1be84cmv6uvs9gk79nv 2422435 2422434 2026-06-02T14:34:52Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage 2422435 wikitext text/x-wiki k2y823kqhpmwf4k09h53f3mxs1wjdpe 2422436 2422435 2026-06-02T14:36:50Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS trixie 2422436 wikitext text/x-wiki dk242yw0gmf6umj23vwbc0m6tnnv1nx 2422437 2422436 2026-06-02T14:37:25Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS trixie 2422437 wikitext text/x-wiki 5g37fu62hp1smcarjha0q06eb3n0bsd 2422438 2422437 2026-06-02T14:37:32Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422438 wikitext text/x-wiki qg32ozm1o4lc50tvo7vrky5v36emcys 2422439 2422438 2026-06-02T14:37:34Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2422439 wikitext text/x-wiki 06lgy1sdt0j50ik05fef8pm4b9nd8kq 2422440 2422439 2026-06-02T14:37:36Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1048.eqiad.wmnet 2422440 wikitext text/x-wiki b2oi9ni9ebincfu8ui21radw1pm7izh 2422441 2422440 2026-06-02T14:37:59Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422441 wikitext text/x-wiki gq710txyhlnfzv8wuryyv3nc3xcza5x 2422443 2422441 2026-06-02T14:38:43Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2422443 wikitext text/x-wiki 3jmjya2mzliqqknkrdpcv7snp58m7ud 2422444 2422443 2026-06-02T14:38:46Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2422444 wikitext text/x-wiki rjh2k6i8vs7a5pxybj8qbe5qojka5vu 2422445 2422444 2026-06-02T14:38:58Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422445 wikitext text/x-wiki 5nymalev77ayv4nbd7r2eulce7px39a 2422446 2422445 2026-06-02T14:40:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T426633)', diff saved to https://phabricator.wikimedia.org/P93571 and previous config saved to /var/cache/conftool/dbconfig/20260602-144043-fceratto.json 2422446 wikitext text/x-wiki 3n3yboau8ccg317mzos07pf72c6gcki 2422447 2422446 2026-06-02T14:40:50Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422447 wikitext text/x-wiki t7fc8gfigigs3c13baut44fi1mnt6vk 2422448 2422447 2026-06-02T14:41:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2158: Repooling 2422448 wikitext text/x-wiki 9p42r030snbi1lohxr4sy7mewq73p0k 2422449 2422448 2026-06-02T14:41:04Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance 2422449 wikitext text/x-wiki 84jxj0li9kzyx9aqtdzwzxevq6yl971 2422450 2422449 2026-06-02T14:41:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1182 (T426633)', diff saved to https://phabricator.wikimedia.org/P93573 and previous config saved to /var/cache/conftool/dbconfig/20260602-144110-fceratto.json 2422450 wikitext text/x-wiki jmsvhl3ylg6hsjn8zfl37rd2ljyeg9u 2422451 2422450 2026-06-02T14:41:21Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Repooling 2422451 wikitext text/x-wiki k8nezf1n6reyfnazlyjjsh6os6dyxhq 2422452 2422451 2026-06-02T14:41:23Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache 2422452 wikitext text/x-wiki mo6zc5m3purooosj2756b8zgqpjih27 2422453 2422452 2026-06-02T14:41:36Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2422453 wikitext text/x-wiki 9ij6b73ynfrj4f55h48ezr78lozd6el 2422454 2422453 2026-06-02T14:41:37Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Repooling 2422454 wikitext text/x-wiki hxsf87va7r8n5m5tibltfbs37cepm7k 2422455 2422454 2026-06-02T14:41:46Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2158.codfw.wmnet 2422455 wikitext text/x-wiki inljkdmmo75294zz8dii99cwe9wqw66 2422456 2422455 2026-06-02T14:41:48Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2158.codfw.wmnet 2422456 wikitext text/x-wiki ahmanbix1nu6xpqsbtu6snnovyp2h6f 2422457 2422456 2026-06-02T14:41:56Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2250.codfw.wmnet 2422457 wikitext text/x-wiki 67e7a0e85yrpazuq3pi14io5nmutf53 2422458 2422457 2026-06-02T14:41:58Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2250.codfw.wmnet 2422458 wikitext text/x-wiki 4f4k7zngms1bzq0evsdr43wraldnx97 2422459 2422458 2026-06-02T14:42:04Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for pc2021.codfw.wmnet 2422459 wikitext text/x-wiki 695sv5z36dtdd2cx9znrxscfjzps6e4 2422460 2422459 2026-06-02T14:42:06Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for pc2021.codfw.wmnet 2422460 wikitext text/x-wiki swqsx4eo4uue3la6kqsc0msk45fxlkz 2422461 2422460 2026-06-02T14:49:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T426633)', diff saved to https://phabricator.wikimedia.org/P93575 and previous config saved to /var/cache/conftool/dbconfig/20260602-144935-fceratto.json 2422461 wikitext text/x-wiki ep3fmtz74wqi3u8xj91ikcoyhsun46o 2422462 2422461 2026-06-02T14:50:18Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage 2422462 wikitext text/x-wiki e61g1vdbek49v3uwj6h94siz669kb4q 2422463 2422462 2026-06-02T14:50:53Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422463 wikitext text/x-wiki cdwqv0d1j1zkzpnq7tgeetflp636fj4 2422464 2422463 2026-06-02T14:51:05Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS trixie 2422464 wikitext text/x-wiki njr98yvwscgev3uza9khqc3ow3hmhxn 2422465 2422464 2026-06-02T14:52:02Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1049.eqiad.wmnet 2422465 wikitext text/x-wiki 4htvz5gj454xgvsb1vxol6ug6lf4ux8 2422466 2422465 2026-06-02T14:52:23Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2422466 wikitext text/x-wiki 36xe16l58wq2x7k4d36wrb0m06g2714 2422467 2422466 2026-06-02T14:52:29Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply 2422467 wikitext text/x-wiki hoa4pctoznc9eluc8edoqc6d3beb1zo 2422468 2422467 2026-06-02T14:54:32Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage 2422468 wikitext text/x-wiki p9u4j5dfexld0p2l19e9hlbr0k1l74p 2422472 2422468 2026-06-02T14:59:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93578 and previous config saved to /var/cache/conftool/dbconfig/20260602-145943-fceratto.json 2422472 wikitext text/x-wiki 2zfjyvt4v1qxqhnw7ofwa0i9ebri78o 2422478 2422472 2026-06-02T15:01:16Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2422478 wikitext text/x-wiki 4e3wkrkg06dklmwammufk69eiz880a3 2422479 2422478 2026-06-02T15:02:21Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS trixie 2422479 wikitext text/x-wiki f53tgoj2gb5cu8jfvknpz4hg1fui6s4 2422480 2422479 2026-06-02T15:02:51Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296514|[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] 2422480 wikitext text/x-wiki h9jewjejbadlst5zydu81kmxusifeap 2422481 2422480 2026-06-02T15:05:48Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422481 wikitext text/x-wiki te5s1mqc7vjll4kgo4s5wx31ypb71di 2422482 2422481 2026-06-02T15:06:06Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422482 wikitext text/x-wiki 6pvx0krn7fvbf6q91rs1a1lnisyq9a9 2422483 2422482 2026-06-02T15:06:08Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2422483 wikitext text/x-wiki kaqj2jcnnoebjmbrvlb4cbkvnc3ym5q 2422484 2422483 2026-06-02T15:06:10Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1049.eqiad.wmnet 2422484 wikitext text/x-wiki fayi8ahw3gq3x9x407ud8x41vgvp6yi 2422485 2422484 2026-06-02T15:06:44Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1050.eqiad.wmnet 2422485 wikitext text/x-wiki 4vxhm6vuokbvdavg69ptoa19l8705mz 2422486 2422485 2026-06-02T15:06:48Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Repooling after Icing wait-for-green timeout 2422486 wikitext text/x-wiki gipyhc8ox77hi84ub1uywrzvroi2ven 2422487 2422486 2026-06-02T15:09:11Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296514|[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] (duration: 06m 22s) 2422487 wikitext text/x-wiki sbq2lsczb05xrmqwn9yv5dn2wi4lxy3 2422488 2422487 2026-06-02T15:09:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93580 and previous config saved to /var/cache/conftool/dbconfig/20260602-150951-fceratto.json 2422488 wikitext text/x-wiki dtwgqupx8cgpdifntw7y0guqw5z7tjk 2422489 2422488 2026-06-02T15:12:02Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS trixie 2422489 wikitext text/x-wiki i55xjpkm3419mv9hjdz192pk1lm4t7c 2422490 2422489 2026-06-02T15:12:46Z Stashbot 7414 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie 2422490 wikitext text/x-wiki kahks0irnrf68wk7y1pdnu4uwxea1ds 2422491 2422490 2026-06-02T15:13:26Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295502|Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106|Remove unused 'writeapi' right]], [[gerrit:1296566|Clean up bot password configuration]], [[gerrit:1296563|Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574|cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582|Us 2422491 wikitext text/x-wiki 78jbp7vhz8mz4z0hder61znr5201nwb 2422492 2422491 2026-06-02T15:14:25Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2422492 wikitext text/x-wiki pyu0sjhvwieb3gmgtm6uysthg2d6u5g 2422493 2422492 2026-06-02T15:15:13Z Stashbot 7414 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Backport for [[gerrit:1295502|Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106|Remove unused 'writeapi' right]], [[gerrit:1296566|Clean up bot password configuration]], [[gerrit:1296563|Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574|cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582 2422493 wikitext text/x-wiki cbvi6uh4w07iuourotg0g3xntsf6nmv 2422494 2422493 2026-06-02T15:15:20Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage 2422494 wikitext text/x-wiki g9k69ay4stgnplpjdkf0zpjk4ukh42l 2422495 2422494 2026-06-02T15:17:53Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply 2422495 wikitext text/x-wiki gi1dplyljusd5qfq28t7lxud1eq00zy 2422496 2422495 2026-06-02T15:17:57Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply 2422496 wikitext text/x-wiki 654td3ta9pftfjsu5bvhnkqd2sms4kp 2422497 2422496 2026-06-02T15:18:07Z Stashbot 7414 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie 2422497 wikitext text/x-wiki 0062nuehd3n0mv26yo5f8xjwg3wvcf3 2422498 2422497 2026-06-02T15:18:26Z Stashbot 7414 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Continuing with deployment 2422498 wikitext text/x-wiki l4t1fmtaci8szh5q3iw7ll40iyatu8u 2422499 2422498 2026-06-02T15:18:56Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS trixie 2422499 wikitext text/x-wiki 255hpa0thqc8xy8s1mevavp09723vgt 2422500 2422499 2026-06-02T15:19:16Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply 2422500 wikitext text/x-wiki s5efcijqysekk7nyxrpzn55gt4ml7hl 2422501 2422500 2026-06-02T15:19:21Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply 2422501 wikitext text/x-wiki nfktj2uwntl6sv4sdkn5kzk0dbzu3uu 2422502 2422501 2026-06-02T15:20:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T426633)', diff saved to https://phabricator.wikimedia.org/P93582 and previous config saved to /var/cache/conftool/dbconfig/20260602-151958-fceratto.json 2422502 wikitext text/x-wiki jsw6ncdr6g8bk3n44uecklya3nkouf7 2422503 2422502 2026-06-02T15:20:20Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance 2422503 wikitext text/x-wiki 9jtf31hibgvcvm7ij8ufp792wtagvpm 2422505 2422503 2026-06-02T15:20:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1188 (T426633)', diff saved to https://phabricator.wikimedia.org/P93583 and previous config saved to /var/cache/conftool/dbconfig/20260602-152026-fceratto.json 2422505 wikitext text/x-wiki fo403mgnuqor81wopzksbaayd3q9q1a 2422506 2422505 2026-06-02T15:20:35Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage 2422506 wikitext text/x-wiki hmy5they7djr2ar14bwex16qxvp92pn 2422507 2422506 2026-06-02T15:22:42Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295502|Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106|Remove unused 'writeapi' right]], [[gerrit:1296566|Clean up bot password configuration]], [[gerrit:1296563|Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574|cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582|U 2422507 wikitext text/x-wiki d2885725bbmwfmzypzvmy771828cl04 2422509 2422507 2026-06-02T15:26:30Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: Repooling 2422509 wikitext text/x-wiki 35tbxr8nbovaudljb8up08vj5kfumcm 2422510 2422509 2026-06-02T15:27:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T426633)', diff saved to https://phabricator.wikimedia.org/P93585 and previous config saved to /var/cache/conftool/dbconfig/20260602-152726-fceratto.json 2422510 wikitext text/x-wiki ey36umhehg6mk0ps2tp3t7u9y9m8d4d 2422512 2422510 2026-06-02T15:29:57Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply 2422512 wikitext text/x-wiki dygmqzueho8nawtritykutu5vp9uj4p 2422513 2422512 2026-06-02T15:30:02Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply 2422513 wikitext text/x-wiki 02dr6u1lk57yfh3e3pjll3kay7kcwir 2422516 2422513 2026-06-02T15:31:45Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage 2422516 wikitext text/x-wiki ai1us7otj9pvmxqhuo53p1mnang5zrv 2422517 2422516 2026-06-02T15:32:12Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply 2422517 wikitext text/x-wiki n9j1xk67lf6t7nff6ze2r0c9nqhwmwo 2422518 2422517 2026-06-02T15:32:17Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply 2422518 wikitext text/x-wiki i3z93lv603ie2u8amegzbgrwqj9mzm4 2422520 2422518 2026-06-02T15:35:32Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage 2422520 wikitext text/x-wiki efpn5u8hiwoep3tl5xskmwg3ek0qrhe 2422521 2422520 2026-06-02T15:36:36Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS trixie 2422521 wikitext text/x-wiki hhwpb1og9bgj04ydc7rho3xunbi5e2l 2422522 2422521 2026-06-02T15:37:35Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1071.eqiad.wmnet with OS trixie 2422522 wikitext text/x-wiki sy2ydz6ybgfu1aa5asx8uv1njz165cy 2422523 2422522 2026-06-02T15:37:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93586 and previous config saved to /var/cache/conftool/dbconfig/20260602-153734-fceratto.json 2422523 wikitext text/x-wiki rq63lu9i5bmhqhcqjaeoikdrp6l4y9h 2422524 2422523 2026-06-02T15:40:19Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296558|hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568|hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] 2422524 wikitext text/x-wiki o2wcdjt3ntdtl9zo5eaqo3eo62gim38 2422525 2422524 2026-06-02T15:42:04Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296558|hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568|hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422525 wikitext text/x-wiki puxh7ixht8wna930csh96i3k1lx1ob0 2422526 2422525 2026-06-02T15:43:32Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2422526 wikitext text/x-wiki mel08zyyy40zt1srd670wrvrgplps70 2422529 2422526 2026-06-02T15:47:43Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296558|hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568|hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] (duration: 07m 24s) 2422529 wikitext text/x-wiki 30dcasnj7nah30ub8s1k77iypw9vn1y 2422530 2422529 2026-06-02T15:47:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93587 and previous config saved to /var/cache/conftool/dbconfig/20260602-154742-fceratto.json 2422530 wikitext text/x-wiki mcvgkepj3f9fef0wej0cejbgukgshgp 2422532 2422530 2026-06-02T15:49:54Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295909|hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] 2422532 wikitext text/x-wiki b5lqxhrvc7sem8ut0b0lgtbhnoib9q6 2422533 2422532 2026-06-02T15:50:18Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage 2422533 wikitext text/x-wiki o8bjirb8htnvnzwqarruqrs29ldigf5 2422534 2422533 2026-06-02T15:51:41Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295909|hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422534 wikitext text/x-wiki fzi356645u9jvi4r4vh13taa8zuk2kt 2422535 2422534 2026-06-02T15:51:53Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1070.eqiad.wmnet with OS trixie 2422535 wikitext text/x-wiki 6ithvot5ydm1wo8hbmq9qqta4g57117 2422536 2422535 2026-06-02T15:53:04Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1072.eqiad.wmnet with OS trixie 2422536 wikitext text/x-wiki awynxzukfg6rmryhqlle6ql5dwy0kcq 2422537 2422536 2026-06-02T15:54:07Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage 2422537 wikitext text/x-wiki 6r09k8bfm1cs2zxfr22p8713fmk59on 2422538 2422537 2026-06-02T15:57:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T426633)', diff saved to https://phabricator.wikimedia.org/P93588 and previous config saved to /var/cache/conftool/dbconfig/20260602-155749-fceratto.json 2422538 wikitext text/x-wiki kotruk656bhgp2quuhxn4hctywe7ujf 2422539 2422538 2026-06-02T15:58:11Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance 2422539 wikitext text/x-wiki qlgmsilgt9ueap75jgyl018sszkh0xy 2422540 2422539 2026-06-02T15:58:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1197 (T426633)', diff saved to https://phabricator.wikimedia.org/P93589 and previous config saved to /var/cache/conftool/dbconfig/20260602-155817-fceratto.json 2422540 wikitext text/x-wiki 60wnm9gzjzmm1svv34veuqxoelplvru 2422541 2422540 2026-06-02T15:59:05Z Stashbot 7414 kharlan@deploy1003: kharlan: Rolling back deployment 2422541 wikitext text/x-wiki 0rqtxbjo7sfyqdcuplcvztd7xan00ws 2422542 2422541 2026-06-02T15:59:44Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295909|hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] (duration: 09m 48s) 2422542 wikitext text/x-wiki 6xedw1n8jfsolxr0z6mlhrkacx6oro0 2422544 2422542 2026-06-02T16:03:23Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296624|Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] 2422544 wikitext text/x-wiki 050ahhc4ebbbz3wd7q4y9d7r1sr5jvm 2422546 2422544 2026-06-02T16:05:11Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296624|Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422546 wikitext text/x-wiki c0tkqeoggrhmrjw4q7bke5bvh50p9ag 2422547 2422546 2026-06-02T16:05:25Z Stashbot 7414 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage 2422547 wikitext text/x-wiki bi7xtbek20hb8oh7kc34d5uxrttj5yx 2422548 2422547 2026-06-02T16:05:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T426633)', diff saved to https://phabricator.wikimedia.org/P93590 and previous config saved to /var/cache/conftool/dbconfig/20260602-160527-fceratto.json 2422548 wikitext text/x-wiki qgvjr6w5ehef8wcx5afwpcl65h0tt60 2422550 2422548 2026-06-02T16:05:48Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage 2422550 wikitext text/x-wiki bihg5ndhp6iet4aw0vikz3lseagvpm8 2422551 2422550 2026-06-02T16:05:50Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2422551 wikitext text/x-wiki rlee7ylb8legjlw7nqa18de4bi2zrpz 2422552 2422551 2026-06-02T16:09:54Z Stashbot 7414 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage 2422552 wikitext text/x-wiki 2mxsksajibdfc6go62kt1bsga08g5iv 2422553 2422552 2026-06-02T16:10:03Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296624|Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 06m 40s) 2422553 wikitext text/x-wiki njtvl3ho51agb94m68mhlykkv5vkjf4 2422554 2422553 2026-06-02T16:10:51Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1071.eqiad.wmnet with OS trixie 2422554 wikitext text/x-wiki nj7x6o1tw97vskfnqpjdrm91t2pefko 2422558 2422554 2026-06-02T16:14:18Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage 2422558 wikitext text/x-wiki 8t0wzs4yseevstwcjqiryx9t8bwc2mk 2422559 2422558 2026-06-02T16:15:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93591 and previous config saved to /var/cache/conftool/dbconfig/20260602-161534-fceratto.json 2422559 wikitext text/x-wiki phh2ha9jmkq285cfla0lajt3k2kpoan 2422561 2422559 2026-06-02T16:25:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93593 and previous config saved to /var/cache/conftool/dbconfig/20260602-162542-fceratto.json 2422561 wikitext text/x-wiki b9dx1owkrwjvwfe6umspb5fnwq4w02x 2422562 2422561 2026-06-02T16:27:38Z Stashbot 7414 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2006.codfw.wmnet with OS trixie 2422562 wikitext text/x-wiki 2vf2e87meik2ya24bqdpb64qh6szauh 2422563 2422562 2026-06-02T16:29:47Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2422563 wikitext text/x-wiki ol71n3l6pnjn0j5eaaou8y8vii0iwf7 2422564 2422563 2026-06-02T16:30:09Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2422564 wikitext text/x-wiki abbdtx4tq93ywo6hycyjbqrgtkkiexi 2422565 2422564 2026-06-02T16:30:52Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1072.eqiad.wmnet with OS trixie 2422565 wikitext text/x-wiki pz8rzm82lyrtfb8kdn44jbotru6ao1g 2422567 2422565 2026-06-02T16:34:01Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2422567 wikitext text/x-wiki 97f3uatnje1b4j2mb7rh81jttvln9p1 2422568 2422567 2026-06-02T16:34:05Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2422568 wikitext text/x-wiki fn3fzgq9t8mttuv7ig41pb67tst8e5t 2422569 2422568 2026-06-02T16:35:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T426633)', diff saved to https://phabricator.wikimedia.org/P93594 and previous config saved to /var/cache/conftool/dbconfig/20260602-163550-fceratto.json 2422569 wikitext text/x-wiki prfi15hsq9e0rdr9l3r4yzj1zdhfeky 2422570 2422569 2026-06-02T16:35:59Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2422570 wikitext text/x-wiki 15d19fn8n3j3p9tim7g03d2vu6td9yg 2422571 2422570 2026-06-02T16:36:05Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply 2422571 wikitext text/x-wiki lwxktfnpgzra7kmxofcc9yum9789nin 2422572 2422571 2026-06-02T16:36:16Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance 2422572 wikitext text/x-wiki mnyy4pngyk0z5xok93a96sv4zfuj7v7 2422573 2422572 2026-06-02T16:36:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1229 (T426633)', diff saved to https://phabricator.wikimedia.org/P93595 and previous config saved to /var/cache/conftool/dbconfig/20260602-163622-fceratto.json 2422573 wikitext text/x-wiki 79ee473gymo7aeqodomdxrenax9l6xo 2422574 2422573 2026-06-02T16:43:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T426633)', diff saved to https://phabricator.wikimedia.org/P93596 and previous config saved to /var/cache/conftool/dbconfig/20260602-164328-fceratto.json 2422574 wikitext text/x-wiki bqoaej2uqw180lsbwpaci7wrv7s1h42 2422575 2422574 2026-06-02T16:47:32Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422575 wikitext text/x-wiki 628er5t7n7hp74eijqcy53mswxjj90y 2422576 2422575 2026-06-02T16:48:11Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422576 wikitext text/x-wiki 0kewbybjnzw9wfko1di72one3q9om2h 2422577 2422576 2026-06-02T16:48:12Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2422577 wikitext text/x-wiki 6f6m3v7bjjtcs10p32ylg2iv2rsbo6l 2422578 2422577 2026-06-02T16:48:14Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1050.eqiad.wmnet 2422578 wikitext text/x-wiki pjrmfmaxwnsru2kv57fzobgvkcvx3da 2422580 2422578 2026-06-02T16:50:00Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1051.eqiad.wmnet 2422580 wikitext text/x-wiki t0mnztvgkthewtkw2uejmqsx3e3m02r 2422581 2422580 2026-06-02T16:53:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93597 and previous config saved to /var/cache/conftool/dbconfig/20260602-165336-fceratto.json 2422581 wikitext text/x-wiki m681cg2tgmd82wal92im1fxbqu1fbgh 2422582 2422581 2026-06-02T17:03:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93598 and previous config saved to /var/cache/conftool/dbconfig/20260602-170344-fceratto.json 2422582 wikitext text/x-wiki 1yyqd8wxlas1fypw71sy0u90pe3vbm3 2422584 2422582 2026-06-02T17:04:27Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2422584 wikitext text/x-wiki 5sud25bj95ilwrcc6p3sie5op7fsviz 2422585 2422584 2026-06-02T17:13:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T426633)', diff saved to https://phabricator.wikimedia.org/P93599 and previous config saved to /var/cache/conftool/dbconfig/20260602-171354-fceratto.json 2422585 wikitext text/x-wiki fhy96etr5heja5zdlqkoicm3g6lyj8v 2422586 2422585 2026-06-02T17:14:16Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance 2422586 wikitext text/x-wiki l5jt2moc31a22mto2mzcjk6qr7mxl7z 2422587 2422586 2026-06-02T17:14:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1233 (T426633)', diff saved to https://phabricator.wikimedia.org/P93600 and previous config saved to /var/cache/conftool/dbconfig/20260602-171422-fceratto.json 2422587 wikitext text/x-wiki gm5a1mt57egsy1cw3slyrdaguvpjhz3 2422595 2422587 2026-06-02T17:21:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T426633)', diff saved to https://phabricator.wikimedia.org/P93601 and previous config saved to /var/cache/conftool/dbconfig/20260602-172135-fceratto.json 2422595 wikitext text/x-wiki qz1xetwtgf0x2kkomq09mludo3bih64 2422602 2422595 2026-06-02T17:31:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93602 and previous config saved to /var/cache/conftool/dbconfig/20260602-173143-fceratto.json 2422602 wikitext text/x-wiki l6v6rrm9ocd7qsz97nmxczo8lyka42p 2422607 2422602 2026-06-02T17:41:52Z Stashbot 7414 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296615|feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614|feat(cleanMentorList): Add a feature flag (T427386)]] 2422607 wikitext text/x-wiki 779j715nwljk5jsumcluy6u4j40c07e 2422608 2422607 2026-06-02T17:41:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93603 and previous config saved to /var/cache/conftool/dbconfig/20260602-174150-fceratto.json 2422608 wikitext text/x-wiki e9k735xag97uutxcjn0j85ic455jj1u 2422609 2422608 2026-06-02T17:42:25Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2422609 wikitext text/x-wiki k6ol3mwy9fwhe76opel61b51ik0djj4 2422610 2422609 2026-06-02T17:42:52Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2422610 wikitext text/x-wiki arz4l5msuwalotj4hx6j14p0xi8892j 2422611 2422610 2026-06-02T17:42:54Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2422611 wikitext text/x-wiki 81f738asjvdugz24ilcrhlijeyhfxdo 2422612 2422611 2026-06-02T17:43:06Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2422612 wikitext text/x-wiki rls001zb4x0b8pxll57iielytx40rui 2422613 2422612 2026-06-02T17:43:07Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2422613 wikitext text/x-wiki ctnx420ulisly2oolwuq4pcm39g4kep 2422614 2422613 2026-06-02T17:43:21Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2422614 wikitext text/x-wiki j29yq8j2e1p06ligc7e3k46tuuwmc8x 2422615 2422614 2026-06-02T17:43:23Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2422615 wikitext text/x-wiki 76ln19r9qouvk2nqpm5c9sgofa86cmn 2422616 2422615 2026-06-02T17:43:37Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2422616 wikitext text/x-wiki c8iq2xoi2vengoyglpd3lq38lf4yie5 2422617 2422616 2026-06-02T17:43:39Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2422617 wikitext text/x-wiki agyllhv2j6ggt1whgs5qc380oywyh0b 2422618 2422617 2026-06-02T17:43:55Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2422618 wikitext text/x-wiki 7jdzsl9ur8xyf9fxg32mt9sos2sdhrj 2422619 2422618 2026-06-02T17:43:57Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2422619 wikitext text/x-wiki 26c3kx74lj13v6vaoljiln3mpkohz9j 2422620 2422619 2026-06-02T17:44:19Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2422620 wikitext text/x-wiki jmh0mfk2qe8v3gzw54kz94o5h9xw4eh 2422622 2422620 2026-06-02T17:47:20Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2422622 wikitext text/x-wiki l0o6qrrnni3mmcn5h0bxlsc2dcdamyz 2422623 2422622 2026-06-02T17:48:03Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2422623 wikitext text/x-wiki 8eucijn7eyh17ny0wfd1cm18lsarb9b 2422624 2422623 2026-06-02T17:48:34Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2422624 wikitext text/x-wiki 48wegbqr5acv53an80gabeky3qdcqhx 2422625 2422624 2026-06-02T17:49:10Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2422625 wikitext text/x-wiki mb63ydv7ele5tvn4v2y605n4mi94s15 2422626 2422625 2026-06-02T17:49:42Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2422626 wikitext text/x-wiki 07ypddc074u7i1qek17c9czxdjtiy2w 2422627 2422626 2026-06-02T17:50:00Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2422627 wikitext text/x-wiki pezaoq3lkk5s4zej2ndtbq05bak7lv8 2422628 2422627 2026-06-02T17:50:32Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2422628 wikitext text/x-wiki 3t6wn52ra831oie63x6cr6bjouybpkd 2422629 2422628 2026-06-02T17:50:56Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2422629 wikitext text/x-wiki 3ftubfze955fl5zoqnk0q7vphealj72 2422630 2422629 2026-06-02T17:51:28Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2422630 wikitext text/x-wiki afrcvx5cgo959ozmasib6j2uaxj50jj 2422631 2422630 2026-06-02T17:51:52Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2422631 wikitext text/x-wiki tdl1sgx7v28j122zceovd4vqdlb3kf7 2422632 2422631 2026-06-02T17:51:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T426633)', diff saved to https://phabricator.wikimedia.org/P93604 and previous config saved to /var/cache/conftool/dbconfig/20260602-175157-fceratto.json 2422632 wikitext text/x-wiki f3fojozyqyhy8cne6k4me7b2utmlz8a 2422633 2422632 2026-06-02T17:52:20Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance 2422633 wikitext text/x-wiki hntixbryxfpb8br3rlsisl5ibhgio5c 2422634 2422633 2026-06-02T17:52:23Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2422634 wikitext text/x-wiki 1sxzg8zs8xtzndh4qfzlel3lwr2p1z0 2422635 2422634 2026-06-02T17:52:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1254 (T426633)', diff saved to https://phabricator.wikimedia.org/P93605 and previous config saved to /var/cache/conftool/dbconfig/20260602-175227-fceratto.json 2422635 wikitext text/x-wiki 17s5ya5dot56qzwz73j4gtnll1pjrp6 2422637 2422635 2026-06-02T17:53:24Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2422637 wikitext text/x-wiki cqb6vnazjy0dzumnst213ewhidg3tqo 2422638 2422637 2026-06-02T17:55:54Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422638 wikitext text/x-wiki 1nhie5d8friotc7w82z83xjbbf7l1zx 2422639 2422638 2026-06-02T17:56:30Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422639 wikitext text/x-wiki f1q5scmy5swdh2ylm36zbw89y8hlltn 2422640 2422639 2026-06-02T17:56:31Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2422640 wikitext text/x-wiki 4b0hq0mfz47oovmhx8rf2nkezvkvxr6 2422641 2422640 2026-06-02T17:56:33Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1051.eqiad.wmnet 2422641 wikitext text/x-wiki 5i0tnj20rullaltc7qesk5o9i4b8f57 2422642 2422641 2026-06-02T17:57:59Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2422642 wikitext text/x-wiki 5dex5bjy4sofvru27iybmov7r7dqae5 2422643 2422642 2026-06-02T17:58:23Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2422643 wikitext text/x-wiki ndwbojwlfr1vq3afdmvmcv9i264c6j2 2422645 2422643 2026-06-02T17:59:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 (T426633)', diff saved to https://phabricator.wikimedia.org/P93607 and previous config saved to /var/cache/conftool/dbconfig/20260602-175933-fceratto.json 2422645 wikitext text/x-wiki gl3v2czs2jw4f4ensy7rczkecq4uwmt 2422646 2422645 2026-06-02T18:00:52Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1052.eqiad.wmnet 2422646 wikitext text/x-wiki mvm89r0y4h203c586tez407vbfu2s33 2422647 2422646 2026-06-02T18:01:33Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1296615|feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614|feat(cleanMentorList): Add a feature flag (T427386)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422647 wikitext text/x-wiki syi5exikh6iqgeijgz8ygagvydp72gw 2422648 2422647 2026-06-02T18:01:51Z Stashbot 7414 urbanecm@deploy1003: urbanecm: Continuing with deployment 2422648 wikitext text/x-wiki sn17dso3ymzx02un6e7sylkolusdurk 2422649 2422648 2026-06-02T18:01:53Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2422649 wikitext text/x-wiki kqxhmubyrvdtyt2bhcahsyowjsrtyh8 2422651 2422649 2026-06-02T18:02:24Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2422651 wikitext text/x-wiki 22pzto0zjbhu6hdtgkrymo1a1tnyave 2422652 2422651 2026-06-02T18:02:39Z Stashbot 7414 swfrench-wmf: reverting shellbox to 2026-05-20-192555 due to errors in shellbox-syntaxhighlight 2422652 wikitext text/x-wiki quhwi2lf3kxb4on53b5t3izos5onruh 2422653 2422652 2026-06-02T18:02:55Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2422653 wikitext text/x-wiki rduyos8xble3p9wxpkz1x9nz7sqa326 2422654 2422653 2026-06-02T18:04:30Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2422654 wikitext text/x-wiki ba1ni41fkqw7i8dxavidoopaeotxddb 2422655 2422654 2026-06-02T18:05:01Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2422655 wikitext text/x-wiki 3mgnfnvrxfvqokt4erreg5pqglzt6f6 2422656 2422655 2026-06-02T18:05:16Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2422656 wikitext text/x-wiki ae1qdxp22bzp3hr60rusrcfxi6g555w 2422657 2422656 2026-06-02T18:05:47Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2422657 wikitext text/x-wiki gwtb8g95xgq86zydtszyht4sgew0fer 2422658 2422657 2026-06-02T18:05:50Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2422658 wikitext text/x-wiki gc6aoi63cf1o7vppc9iv3viizck184s 2422659 2422658 2026-06-02T18:06:21Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2422659 wikitext text/x-wiki 0ztceeun9exgnps1o1bwpnr361ineer 2422660 2422659 2026-06-02T18:06:52Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2422660 wikitext text/x-wiki f46t2xc3yrqshb8n4wz5mhfowa44i5t 2422661 2422660 2026-06-02T18:07:23Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2422661 wikitext text/x-wiki jybuw776wkubs27nk7j01rsi0ab8hkd 2422662 2422661 2026-06-02T18:08:41Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2422662 wikitext text/x-wiki a2v4f77ourcge4vn5jo6cfqslkstd5l 2422663 2422662 2026-06-02T18:09:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93608 and previous config saved to /var/cache/conftool/dbconfig/20260602-180941-fceratto.json 2422663 wikitext text/x-wiki bqhichxs1t3cjks2skwiegacs0pwjet 2422664 2422663 2026-06-02T18:10:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2422664 wikitext text/x-wiki 8xf88y3ifihhgw9lqvksn9jnaddhtqj 2422665 2422664 2026-06-02T18:12:17Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2422665 wikitext text/x-wiki kfx9cpeoz1do8ee8wt5membk13qesz0 2422666 2422665 2026-06-02T18:12:34Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2422666 wikitext text/x-wiki mhxuzru7u7zjlbatk9smttkvpfq15r8 2422667 2422666 2026-06-02T18:12:36Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2422667 wikitext text/x-wiki kwr9cq5wzxrgo6hx8zmshxk9is1ocl7 2422668 2422667 2026-06-02T18:13:01Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2422668 wikitext text/x-wiki 0cwbcy9o1jm77hvvevqn95kkjcvd4vh 2422669 2422668 2026-06-02T18:13:02Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2422669 wikitext text/x-wiki miczcb48sqp9konfucqr8bfpb8065b5 2422670 2422669 2026-06-02T18:13:11Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2422670 wikitext text/x-wiki 4sw74x0horbswu1z8b58wznh03bnxit 2422671 2422670 2026-06-02T18:13:13Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2422671 wikitext text/x-wiki lod54g83qy52yp6zj9uoo2yrr7pbe4j 2422672 2422671 2026-06-02T18:13:21Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2422672 wikitext text/x-wiki 7zas3jxlhg7aq78829i2dpcu0l7s4p6 2422673 2422672 2026-06-02T18:13:22Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2422673 wikitext text/x-wiki 1p51d38u5ucbtpfytbbqlko0ho3d7in 2422675 2422673 2026-06-02T18:13:34Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2422675 wikitext text/x-wiki ne7k86kuhwmv6boht75uz5zzw4xa1ae 2422676 2422675 2026-06-02T18:13:36Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2422676 wikitext text/x-wiki jcqsutwsih41s2c0wxcckgfjbpzuqok 2422677 2422676 2026-06-02T18:13:40Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2422677 wikitext text/x-wiki dgukd2e727f9eov7d6kp5b2vh2m4r8g 2422679 2422677 2026-06-02T18:16:02Z Stashbot 7414 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296615|feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614|feat(cleanMentorList): Add a feature flag (T427386)]] (duration: 34m 09s) 2422679 wikitext text/x-wiki 5rgflkbu9l9a852wiozkf50jojyadag 2422683 2422679 2026-06-02T18:19:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93609 and previous config saved to /var/cache/conftool/dbconfig/20260602-181949-fceratto.json 2422683 wikitext text/x-wiki 71p3xqpw67yf3sokygw8apf8ftvqj7g 2422685 2422683 2026-06-02T18:21:43Z Stashbot 7414 Daimona: Running query from T427962#11978299 in x1.wikishared 2422685 wikitext text/x-wiki bcxbbltn60emi3iwjad2htc8uaex8hk 2422686 2422685 2026-06-02T18:25:01Z Stashbot 7414 dancy: Train is blocked at testwikis on https://phabricator.wikimedia.org/T427935 2422686 wikitext text/x-wiki ow60zqwqeyhp3yc3oygyb24iwa30vkp 2422687 2422686 2026-06-02T18:25:24Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422687 wikitext text/x-wiki jnm1ipyak3vc1uyujtch09caddmcc9m 2422689 2422687 2026-06-02T18:26:04Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422689 wikitext text/x-wiki ennoues93vwti8zauynbcwmusigsqeh 2422690 2422689 2026-06-02T18:26:06Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2422690 wikitext text/x-wiki ml6bsf5d9ehu1zv4drwl57z7gobamwp 2422691 2422690 2026-06-02T18:26:08Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1052.eqiad.wmnet 2422691 wikitext text/x-wiki qj9kt8dpnuv0ind2lmf4w614xjfoed7 2422692 2422691 2026-06-02T18:27:50Z Stashbot 7414 mutante: gerrit delete unused plugin projects: barricade, WikimediaBlocks and WikimediaWebSessions 2422692 wikitext text/x-wiki q53hfxme7i7wtj4okbw9zbgqj428thr 2422693 2422692 2026-06-02T18:29:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 (T426633)', diff saved to https://phabricator.wikimedia.org/P93610 and previous config saved to /var/cache/conftool/dbconfig/20260602-182956-fceratto.json 2422693 wikitext text/x-wiki hgpt907ya7d6998qp7qy3kfbdzv448q 2422695 2422693 2026-06-02T18:30:17Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance 2422695 wikitext text/x-wiki 5y9j85oroyxmfk2oen473jeopk4nrts 2422696 2422695 2026-06-02T18:30:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1259 (T426633)', diff saved to https://phabricator.wikimedia.org/P93611 and previous config saved to /var/cache/conftool/dbconfig/20260602-183023-fceratto.json 2422696 wikitext text/x-wiki f3qmuek34rm7y2w2yrm97y1alvketp8 2422697 2422696 2026-06-02T18:33:55Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1053.eqiad.wmnet 2422697 wikitext text/x-wiki sqlaupsam7vot9vyz6uza327qpu1643 2422699 2422697 2026-06-02T18:37:05Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2422699 wikitext text/x-wiki e0wmbi0ch70k21bn15jmy9zrbdv3353 2422700 2422699 2026-06-02T18:37:23Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2422700 wikitext text/x-wiki 5rah34y41fu4ejjozfsiltjnchqk2yx 2422701 2422700 2026-06-02T18:37:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 (T426633)', diff saved to https://phabricator.wikimedia.org/P93612 and previous config saved to /var/cache/conftool/dbconfig/20260602-183749-fceratto.json 2422701 wikitext text/x-wiki 09dhznazkduhrdvyyf0lxlsa6yoaxuq 2422702 2422701 2026-06-02T18:38:33Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2422702 wikitext text/x-wiki o8vu9owyjs6l24vd83eiff0ds8stcgm 2422703 2422702 2026-06-02T18:38:39Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2422703 wikitext text/x-wiki 7xeotv4ezdiaqzw7q0gld9wvct9h743 2422704 2422703 2026-06-02T18:38:50Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2422704 wikitext text/x-wiki 6dvrmsdlex3tdd9048pf7yw14oti7s6 2422708 2422704 2026-06-02T18:47:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93614 and previous config saved to /var/cache/conftool/dbconfig/20260602-184757-fceratto.json 2422708 wikitext text/x-wiki ouzt5pclsr7iu7zp4lfvh6p34pq99gq 2422710 2422708 2026-06-02T18:58:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93615 and previous config saved to /var/cache/conftool/dbconfig/20260602-185804-fceratto.json 2422710 wikitext text/x-wiki dt430konqk3cjvm13y624hpw5fdw25l 2422713 2422710 2026-06-02T19:05:22Z Stashbot 7414 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.5 refs T423914 2422713 wikitext text/x-wiki loui97343fnbwx9si59yhgfnbv9tpqc 2422715 2422713 2026-06-02T19:08:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 (T426633)', diff saved to https://phabricator.wikimedia.org/P93616 and previous config saved to /var/cache/conftool/dbconfig/20260602-190811-fceratto.json 2422715 wikitext text/x-wiki hvsgeoxzn76zs33lennod2re1csmnxp 2422716 2422715 2026-06-02T19:09:01Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance 2422716 wikitext text/x-wiki ayq5srzskhx3dta1p6h1f01eu7uxaxy 2422717 2422716 2026-06-02T19:09:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 (T426633)', diff saved to https://phabricator.wikimedia.org/P93617 and previous config saved to /var/cache/conftool/dbconfig/20260602-190907-fceratto.json 2422717 wikitext text/x-wiki 837xsnxvdvl069xb5artvvc0fdd62wb 2422725 2422717 2026-06-02T19:37:33Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422725 wikitext text/x-wiki iv1ua8thxxk21a65vga2sy8ni6o6yai 2422728 2422725 2026-06-02T19:48:26Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422728 wikitext text/x-wiki o9dg37l0k1zr9qdgcih9u7mqqcfha4v 2422729 2422728 2026-06-02T19:48:28Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2422729 wikitext text/x-wiki 22ufekfppp1volerd4txhnj02k1hyrw 2422730 2422729 2026-06-02T19:48:29Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1053.eqiad.wmnet 2422730 wikitext text/x-wiki hk42ey3v4t63jx54vdnhy6odshq792m 2422734 2422730 2026-06-02T20:03:49Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1054.eqiad.wmnet 2422734 wikitext text/x-wiki jybeuun44wxxctfyqotpzjvw2gbp3u9 2422735 2422734 2026-06-02T20:09:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T426633)', diff saved to https://phabricator.wikimedia.org/P93618 and previous config saved to /var/cache/conftool/dbconfig/20260602-200922-fceratto.json 2422735 wikitext text/x-wiki arlkxdopprgkc372f5iq6y3ydsubt1e 2422736 2422735 2026-06-02T20:19:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93619 and previous config saved to /var/cache/conftool/dbconfig/20260602-201929-fceratto.json 2422736 wikitext text/x-wiki sefnbmowl5650gvo3mo85w8gtakbfgy 2422737 2422736 2026-06-02T20:20:15Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.dns.netbox 2422737 wikitext text/x-wiki 3ndh9ajtm647n0thhns2s04de4ioued 2422738 2422737 2026-06-02T20:26:07Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422738 wikitext text/x-wiki 6f7l9la4j5jc85fkeub95ph1gzn0thb 2422739 2422738 2026-06-02T20:27:29Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" 2422739 wikitext text/x-wiki gwo858ph1q19pwqb29uhkym33y2hd2q 2422740 2422739 2026-06-02T20:27:31Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2422740 wikitext text/x-wiki 0i2cmu1cgnsw55g0iwjzrxrf95bxd9i 2422741 2422740 2026-06-02T20:27:33Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1054.eqiad.wmnet 2422741 wikitext text/x-wiki c7fufq8xrc5bvjo4yp1hx5hluzat0bp 2422742 2422741 2026-06-02T20:29:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93620 and previous config saved to /var/cache/conftool/dbconfig/20260602-202937-fceratto.json 2422742 wikitext text/x-wiki cgj6n1vm57lxzxhj9kreriueoy1oyvm 2422744 2422742 2026-06-02T20:39:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T426633)', diff saved to https://phabricator.wikimedia.org/P93621 and previous config saved to /var/cache/conftool/dbconfig/20260602-203945-fceratto.json 2422744 wikitext text/x-wiki 17eka9khvte2pwurchsmki21nrpuunc 2422749 2422744 2026-06-02T22:05:19Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296551|hCaptcha: Enable for badlogin on group0 wikis (T426875)]] 2422749 wikitext text/x-wiki mxf49daajd8wx8ah0eqojt8fu2g9s3t 2422750 2422749 2026-06-02T22:07:17Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296551|hCaptcha: Enable for badlogin on group0 wikis (T426875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422750 wikitext text/x-wiki rb14z7r1zy2zqarsatlggcx1wuletng 2422751 2422750 2026-06-02T22:09:32Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2422751 wikitext text/x-wiki 520cpfk4v2xpaiyc9yqwub5zizdgnvh 2422752 2422751 2026-06-02T22:10:20Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2422752 wikitext text/x-wiki gbqwpdtm1edrxmtc8x6ozyuwohcr7m4 2422753 2422752 2026-06-02T22:10:40Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2422753 wikitext text/x-wiki amioi8lsdz10u0moakuj21vulpk9zta 2422754 2422753 2026-06-02T22:13:50Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296551|hCaptcha: Enable for badlogin on group0 wikis (T426875)]] (duration: 08m 31s) 2422754 wikitext text/x-wiki d2q9q0av00r6u7iu51r991bq5oi2yk1 2422755 2422754 2026-06-02T22:15:31Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296689|hCaptcha: Correct inaccurate comment]] 2422755 wikitext text/x-wiki rouamv13uej3g3ehz7s6kpeistmacwh 2422757 2422755 2026-06-02T22:17:30Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296689|hCaptcha: Correct inaccurate comment]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422757 wikitext text/x-wiki r0txb62wvdtrmkng1blrmf4w3apov39 2422758 2422757 2026-06-02T22:17:49Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2422758 wikitext text/x-wiki ddp9j6rw7is8wnrokxpmdsaiqhdulv3 2422759 2422758 2026-06-02T22:18:29Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply 2422759 wikitext text/x-wiki 5d2l6fg33z6d5w8ozzrtofjq9ss4302 2422760 2422759 2026-06-02T22:18:47Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply 2422760 wikitext text/x-wiki f3a2j2njb6kw9fgpthhowyy4g27haeo 2422761 2422760 2026-06-02T22:21:58Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296689|hCaptcha: Correct inaccurate comment]] (duration: 06m 27s) 2422761 wikitext text/x-wiki h3fg77svzvgdrztbp6flhv2lwml6z0a 2422786 2422761 2026-06-03T05:16:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422786 wikitext text/x-wiki 8w9yt0srm67acdydruytouj8ckr4m4v 2422787 2422786 2026-06-03T05:17:12Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1056: Upgrading es1056.eqiad.wmnet 2422787 wikitext text/x-wiki ltzl7d9qgho39587rshajvnzdh7n39e 2422788 2422787 2026-06-03T05:17:43Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1056: Upgrading es1056.eqiad.wmnet 2422788 wikitext text/x-wiki grmxrhwl15ve28ajin0alf9y5p1l4xj 2422789 2422788 2026-06-03T05:18:22Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1056.eqiad.wmnet with OS trixie 2422789 wikitext text/x-wiki 9u9ywib3ohftmb2n0m0dqxxl52aqf7c 2422790 2422789 2026-06-03T05:33:23Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage 2422790 wikitext text/x-wiki c8ptjbebq38d3txopiq1bhhxe07jmi5 2422791 2422790 2026-06-03T05:39:28Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage 2422791 wikitext text/x-wiki tsw8jox3j3jf1k3o05nj8l5mtwx66qc 2422793 2422791 2026-06-03T05:55:25Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1056.eqiad.wmnet with OS trixie 2422793 wikitext text/x-wiki m47kr6a6tzmqvxrn7nurldj75chnt98 2422794 2422793 2026-06-03T05:59:12Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422794 wikitext text/x-wiki jz3uxyp07zozrd90quk7yytkd0u7mof 2422795 2422794 2026-06-03T05:59:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1056: repool after upgrade 2422795 wikitext text/x-wiki d9hhci5bnisopl90vze0imhpkeo35w3 2422796 2422795 2026-06-03T06:09:04Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422796 wikitext text/x-wiki lg1tvog786aepa00v6hgeoswkyb249a 2422797 2422796 2026-06-03T06:09:26Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2231: Upgrading db2231.codfw.wmnet 2422797 wikitext text/x-wiki i6f4qv6aw9h8p0sf59f79ly47x7o54y 2422798 2422797 2026-06-03T06:09:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2231: Upgrading db2231.codfw.wmnet 2422798 wikitext text/x-wiki 466re2a5vm9hrardsfbu2tyufvd3i9m 2422799 2422798 2026-06-03T06:19:48Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2231.codfw.wmnet with OS trixie 2422799 wikitext text/x-wiki 3k4g5kg5opbmtm1e9zc4aavgqfgcdf6 2422802 2422799 2026-06-03T06:36:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2231.codfw.wmnet with reason: host reimage 2422802 wikitext text/x-wiki bzwb6nrl15x8vfngw513hk7el4yeq8w 2422803 2422802 2026-06-03T06:40:02Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2231.codfw.wmnet with reason: host reimage 2422803 wikitext text/x-wiki 7ceqs4kbi3bohcbln535pv8db8ha5bx 2422804 2422803 2026-06-03T06:44:54Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1056: repool after upgrade 2422804 wikitext text/x-wiki ax6bp2v66h4kbdxtgsx42mx0g5grwix 2422805 2422804 2026-06-03T06:45:36Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422805 wikitext text/x-wiki mybmqmsgw02qa29xbhidqbjm1qyje68 2422806 2422805 2026-06-03T06:45:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1049: Upgrading es1049.eqiad.wmnet 2422806 wikitext text/x-wiki j25tn14573j6kd08usiybteu3w93h6o 2422807 2422806 2026-06-03T06:46:26Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2056 to es2 codfw primary T427875', diff saved to https://phabricator.wikimedia.org/P93632 and previous config saved to /var/cache/conftool/dbconfig/20260603-064623-marostegui.json 2422807 wikitext text/x-wiki pdhgoohog29xxqawn7dxrhxdy6e90bo 2422808 2422807 2026-06-03T06:46:57Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1049: Upgrading es1049.eqiad.wmnet 2422808 wikitext text/x-wiki p09b1mbexk1th505n3dk4nmph0goxfv 2422810 2422808 2026-06-03T06:52:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1049.eqiad.wmnet with OS trixie 2422810 wikitext text/x-wiki 34oauwjs04q197poc94g7rionaodwj4 2422812 2422810 2026-06-03T06:57:18Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2231.codfw.wmnet with OS trixie 2422812 wikitext text/x-wiki g6j2re1z5x5i4xhnl4vcx2p4qj9dv9x 2422813 2422812 2026-06-03T07:04:44Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422813 wikitext text/x-wiki p6gxml88wp1t00g9zuxpscyrqwti0k7 2422814 2422813 2026-06-03T07:07:14Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2231: repool after maintenance 2422814 wikitext text/x-wiki s1fejijgi7ffwnggbftsn75rdqj6npb 2422815 2422814 2026-06-03T07:07:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage 2422815 wikitext text/x-wiki lujlvuxul642m8epg2coiaxevlh7seh 2422816 2422815 2026-06-03T07:14:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage 2422816 wikitext text/x-wiki 3a0rrbc49qblpgbzkmtrr3nntlqh89g 2422817 2422816 2026-06-03T07:16:38Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296516|Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517|Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] 2422817 wikitext text/x-wiki mvtsux7ljqinwf05tslpcgjyxdu2ty9 2422819 2422817 2026-06-03T07:32:20Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1049.eqiad.wmnet with OS trixie 2422819 wikitext text/x-wiki a4itue9j4mv6mmuvhuz4ow4ldpspmyb 2422820 2422819 2026-06-03T07:35:19Z Stashbot 7414 mszwarc@deploy1003: mszwarc, stran: Backport for [[gerrit:1296516|Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517|Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422820 wikitext text/x-wiki 2bc8litl4zo91srfnqsxq33dyvle6sv 2422821 2422820 2026-06-03T07:36:00Z Stashbot 7414 mszwarc@deploy1003: mszwarc, stran: Continuing with deployment 2422821 wikitext text/x-wiki afh6r2rn5nen5g8mnchyf44w9m9y9k9 2422822 2422821 2026-06-03T07:37:23Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422822 wikitext text/x-wiki 6nc9czk9wkgxvcxxjerjx5430pg5v4a 2422823 2422822 2026-06-03T07:37:48Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1049: repool after upgrade 2422823 wikitext text/x-wiki 050qyfc6817ilr1tdd67g9n8nf6ra0e 2422824 2422823 2026-06-03T07:42:53Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1056 to es2 eqiad primary T427875', diff saved to https://phabricator.wikimedia.org/P93637 and previous config saved to /var/cache/conftool/dbconfig/20260603-074250-marostegui.json 2422824 wikitext text/x-wiki ahrtrd5k21c23oq615ui0fzhf16iu5e 2422825 2422824 2026-06-03T07:43:31Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2422825 wikitext text/x-wiki riv6hi8xqpk587jz56tah36q5zgq3bk 2422827 2422825 2026-06-03T07:44:57Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2422827 wikitext text/x-wiki ist4kzdmhgm7rctdao87mpz6oai0hf1 2422829 2422827 2026-06-03T07:48:51Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296516|Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517|Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] (duration: 32m 13s) 2422829 wikitext text/x-wiki hjyseyqtvldj4f11zf7zp2wfhlkbbry 2422830 2422829 2026-06-03T07:50:20Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296580|Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703|jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713|conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627|Add missing lazy img to carousel (T427821)]], [[gerrit:1295968|MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]] 2422830 wikitext text/x-wiki 1hxkpzcweizdym71a4fe5x18mhonj92 2422831 2422830 2026-06-03T07:50:36Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422831 wikitext text/x-wiki rni6zt3uj6sqy0t0shmdbiv4vpltco7 2422832 2422831 2026-06-03T07:50:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2054: Upgrading es2054.codfw.wmnet 2422832 wikitext text/x-wiki iavjnw7faqazmi6hju4h7qtz5vy97ii 2422833 2422832 2026-06-03T07:51:18Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2054: Upgrading es2054.codfw.wmnet 2422833 wikitext text/x-wiki lay9abnoptxr8dqcz6xn5a4slszzfuk 2422834 2422833 2026-06-03T07:52:13Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2054.codfw.wmnet with OS trixie 2422834 wikitext text/x-wiki c16ewgto2485390cd0qunc6oi2j8s49 2422835 2422834 2026-06-03T07:52:39Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2231: repool after maintenance 2422835 wikitext text/x-wiki 05nlaosittvxy7dqqpuy62ogcovqjzb 2422836 2422835 2026-06-03T07:54:24Z Stashbot 7414 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Backport for [[gerrit:1296580|Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703|jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713|conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627|Add missing lazy img to carousel (T427821)]], [[gerrit:1295968|MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T42 2422836 wikitext text/x-wiki rfi2hkfc1rjip3srswd1k2hsd7y9ol5 2422837 2422836 2026-06-03T07:57:00Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream server 2422837 wikitext text/x-wiki tj4qynwfl44t24vdmrt3oxw7cv819dm 2422838 2422837 2026-06-03T07:57:09Z Stashbot 7414 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Continuing with deployment 2422838 wikitext text/x-wiki 1eno3kju3qnswu2aii3z4a5iarw3kxa 2422839 2422838 2026-06-03T07:57:25Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream server 2422839 wikitext text/x-wiki 058o147ymazq1kslfu2ljby7c61ii2o 2422840 2422839 2026-06-03T08:00:04Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2422840 wikitext text/x-wiki f7cj5p9k48ss4qanw9iwqtrsb5rnt80 2422841 2422840 2026-06-03T08:01:15Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Repooling 2422841 wikitext text/x-wiki 0d67c8mkibt4c3bo47st0ri3ktrfft0 2422842 2422841 2026-06-03T08:01:18Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Repooling 2422842 wikitext text/x-wiki mgfpozrkry4smyn606glqeer6rfz00u 2422843 2422842 2026-06-03T08:01:26Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422843 wikitext text/x-wiki svc4e3jmnmyqsvr5g4bfar0759wy8so 2422844 2422843 2026-06-03T08:01:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422844 wikitext text/x-wiki jv81qo07insjqay6355vgiqt7gg6jhc 2422845 2422844 2026-06-03T08:01:40Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2215: Upgrading db2215.codfw.wmnet 2422845 wikitext text/x-wiki oqe8ye7jvjfieqjb9gtnnsqtc07vt88 2422846 2422845 2026-06-03T08:01:46Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1211: Upgrading db1211.eqiad.wmnet 2422846 wikitext text/x-wiki mncuq0los6pill6opdn27h1sbx3rma6 2422847 2422846 2026-06-03T08:01:56Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2422847 wikitext text/x-wiki si9bwh1vk9n0b77i9onq5bcwwj1elgo 2422848 2422847 2026-06-03T08:02:01Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2215: Upgrading db2215.codfw.wmnet 2422848 wikitext text/x-wiki 2leckg04qsjbwufuqs3gzc11l2ojvgh 2422849 2422848 2026-06-03T08:02:36Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1211: Upgrading db1211.eqiad.wmnet 2422849 wikitext text/x-wiki ioqtqlnte70283m3imoqago3t1cozyf 2422850 2422849 2026-06-03T08:03:18Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2215.codfw.wmnet with OS trixie 2422850 wikitext text/x-wiki ruyzj09z55c08djrbv8rtr2qbhmrale 2422851 2422850 2026-06-03T08:03:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance 2422851 wikitext text/x-wiki 1prx2v22udzcchvt3dt0mv19w1um3hm 2422852 2422851 2026-06-03T08:03:42Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1211.eqiad.wmnet with OS trixie 2422852 wikitext text/x-wiki k86vbc7xnczdau5rdn1v1tv1j6pm263 2422853 2422852 2026-06-03T08:03:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 (T426633)', diff saved to https://phabricator.wikimedia.org/P93643 and previous config saved to /var/cache/conftool/dbconfig/20260603-080346-fceratto.json 2422853 wikitext text/x-wiki ibj6j8bcu53munsypn1f0i9q17wllsa 2422854 2422853 2026-06-03T08:04:02Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296580|Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703|jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713|conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627|Add missing lazy img to carousel (T427821)]], [[gerrit:1295968|MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)] 2422854 wikitext text/x-wiki 94j23c9urqjltaarti1qdpqgl8tv359 2422855 2422854 2026-06-03T08:05:34Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296632|Image Browsing: add accessible labels to carousel elements (T407793)]] 2422855 wikitext text/x-wiki ipuyc9bnyi6xcaoc4dfludmjriasijs 2422857 2422855 2026-06-03T08:08:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2054.codfw.wmnet with reason: host reimage 2422857 wikitext text/x-wiki psbzx8emxwdm0q76op83uy52tdm2zpq 2422859 2422857 2026-06-03T08:14:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2054.codfw.wmnet with reason: host reimage 2422859 wikitext text/x-wiki bcs7b2hacmq9d8v9ihmhb7pv3jzo6bh 2422860 2422859 2026-06-03T08:16:51Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2422860 wikitext text/x-wiki 3ssznkjxainm3bln41ig84qux5nypqn 2422861 2422860 2026-06-03T08:17:13Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2422861 wikitext text/x-wiki pz8jqz3hkkjils6x8onfs90859590uy 2422862 2422861 2026-06-03T08:17:32Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2422862 wikitext text/x-wiki 2w3o0eg9m6j5p5ir5o0zal5dpt4l571 2422863 2422862 2026-06-03T08:17:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T426633)', diff saved to https://phabricator.wikimedia.org/P93645 and previous config saved to /var/cache/conftool/dbconfig/20260603-081756-fceratto.json 2422863 wikitext text/x-wiki c4lp906zfww7jvlci1gdtie7tv0w5l7 2422864 2422863 2026-06-03T08:18:13Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2422864 wikitext text/x-wiki 4f30q13xxy6kjd4g0o7l6ofrlw3s10s 2422865 2422864 2026-06-03T08:18:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage 2422865 wikitext text/x-wiki h03o1g2t5rhb4p0eqc28c6bmcbp1ajf 2422866 2422865 2026-06-03T08:22:47Z Stashbot 7414 mszwarc@deploy1003: mlitn, mszwarc: Backport for [[gerrit:1296632|Image Browsing: add accessible labels to carousel elements (T407793)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422866 wikitext text/x-wiki pfuz7acrubxpp8bx2z1c8sou5h9xxvj 2422867 2422866 2026-06-03T08:22:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2215.codfw.wmnet with reason: host reimage 2422867 wikitext text/x-wiki p0pxeyy6kfbt8qu6x81kv8juc5qeee0 2422868 2422867 2026-06-03T08:23:13Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1049: repool after upgrade 2422868 wikitext text/x-wiki jr1dw5oljf68eqnyaupzqjkludyy92h 2422869 2422868 2026-06-03T08:24:03Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage 2422869 wikitext text/x-wiki rj8savpihv1nge6l3wpjdj5y06769ww 2422870 2422869 2026-06-03T08:25:15Z Stashbot 7414 mszwarc@deploy1003: mlitn, mszwarc: Continuing with deployment 2422870 wikitext text/x-wiki k0t7vn62mtn4htsrgvy71urgd5rkt7p 2422874 2422870 2026-06-03T08:28:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93647 and previous config saved to /var/cache/conftool/dbconfig/20260603-082804-fceratto.json 2422874 wikitext text/x-wiki tbzgbd6cxmishdfpbogap8thmv5q0x3 2422875 2422874 2026-06-03T08:28:15Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2215.codfw.wmnet with reason: host reimage 2422875 wikitext text/x-wiki qq3ppt7t40vlbz6s6zvpfvvqwsovl8s 2422876 2422875 2026-06-03T08:29:33Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. 2422876 wikitext text/x-wiki 4breeus4uqoobqcczkmanh7y27rt7hw 2422877 2422876 2026-06-03T08:30:23Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. 2422877 wikitext text/x-wiki ngjjlzgj8wnb47f6l8ca82z7e7e95zl 2422878 2422877 2026-06-03T08:31:04Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2054.codfw.wmnet with OS trixie 2422878 wikitext text/x-wiki 0sxr8tag1xhiwn4rkmqx1bapl77m4df 2422879 2422878 2026-06-03T08:31:27Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. 2422879 wikitext text/x-wiki 78rmenvzmiyslqrla00srptm9f5mas3 2422880 2422879 2026-06-03T08:31:56Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2422880 wikitext text/x-wiki 0lmg0rsd9c3jezqomc0r3pouk0jkky1 2422881 2422880 2026-06-03T08:33:16Z Stashbot 7414 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2422881 wikitext text/x-wiki c6nrj6o96sf694owsmbf4q0f77wy6z2 2422882 2422881 2026-06-03T08:33:43Z Stashbot 7414 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2422882 wikitext text/x-wiki oa3ynhb5oyivjr94k8gb58z1t3farpb 2422883 2422882 2026-06-03T08:34:15Z Stashbot 7414 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2422883 wikitext text/x-wiki 48cg6y6jwyxtk8t4y4x712cbt8ano1s 2422884 2422883 2026-06-03T08:34:54Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422884 wikitext text/x-wiki oiamo3bsfuggiohu48sxi8t9gucwwmy 2422885 2422884 2026-06-03T08:35:04Z Stashbot 7414 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2422885 wikitext text/x-wiki ovjcahwfu86mhnlpqvayq6ryq4kbbfr 2422886 2422885 2026-06-03T08:35:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054.codfw.wmnet: After reimage 2422886 wikitext text/x-wiki s5mnmcnl5c9kfdzyysadboie9mzqdim 2422887 2422886 2026-06-03T08:35:51Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2054.codfw.wmnet: After reimage 2422887 wikitext text/x-wiki jesfj4iuxqschxoknx2unwn0pjydpje 2422888 2422887 2026-06-03T08:36:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054: repool after upgrade 2422888 wikitext text/x-wiki 8zw1c0om9d8ugseg356m8r0dm470nh5 2422889 2422888 2026-06-03T08:37:45Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296632|Image Browsing: add accessible labels to carousel elements (T407793)]] (duration: 32m 11s) 2422889 wikitext text/x-wiki ghmf7r4xzldqzdblblghxa4xwti3q35 2422890 2422889 2026-06-03T08:38:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93649 and previous config saved to /var/cache/conftool/dbconfig/20260603-083811-fceratto.json 2422890 wikitext text/x-wiki m5gh6wi2bscyyc1fe6stwto0im0j5ul 2422892 2422890 2026-06-03T08:41:10Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1211.eqiad.wmnet with OS trixie 2422892 wikitext text/x-wiki j0k5nfrng5od69ft3mv8k038k3d9roh 2422893 2422892 2026-06-03T08:41:24Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296635|Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] 2422893 wikitext text/x-wiki 8oj2uwahyks2k25fq6ivozj9shbn5fr 2422894 2422893 2026-06-03T08:43:28Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1211: Migration of db1211.eqiad.wmnet completed 2422894 wikitext text/x-wiki 5geb86sn08yqcck3bt8h897ypi01xs9 2422895 2422894 2026-06-03T08:45:20Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.discovery.service-route check docker-registry: maintenance 2422895 wikitext text/x-wiki mc0xixz174tja2htfs17bq7r8f6b19f 2422896 2422895 2026-06-03T08:45:22Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check docker-registry: maintenance 2422896 wikitext text/x-wiki baln88zc6p2y0kl4ch0pkbrrqfu3yyp 2422897 2422896 2026-06-03T08:45:32Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2215.codfw.wmnet with OS trixie 2422897 wikitext text/x-wiki sww29n6ibvch4whlc0mbx1gdvffc5mh 2422898 2422897 2026-06-03T08:47:25Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296635|Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422898 wikitext text/x-wiki gg0jjxnti70hm8n1ovz34h6ntaumvbr 2422899 2422898 2026-06-03T08:48:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T426633)', diff saved to https://phabricator.wikimedia.org/P93651 and previous config saved to /var/cache/conftool/dbconfig/20260603-084819-fceratto.json 2422899 wikitext text/x-wiki ja6k2v5r96vd0eid1bc6rohr2yiorec 2422900 2422899 2026-06-03T08:48:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance 2422900 wikitext text/x-wiki osc1863jac7aryu3jityzru7ewv6fji 2422901 2422900 2026-06-03T08:48:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1166 (T426633)', diff saved to https://phabricator.wikimedia.org/P93652 and previous config saved to /var/cache/conftool/dbconfig/20260603-084846-fceratto.json 2422901 wikitext text/x-wiki 2ftv9y5kpk2hn2gqwg6wh8fva2cybve 2422902 2422901 2026-06-03T08:50:45Z Stashbot 7414 kharlan@deploy1003: kharlan: Rolling back deployment 2422902 wikitext text/x-wiki rvnl5248oqbcqm9tnw196bdya6igkye 2422903 2422902 2026-06-03T08:51:52Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1022-1023].eqiad.wmnet 2422903 wikitext text/x-wiki 791chnj1b1m0e0fuyogoltv9p9rtvf9 2422904 2422903 2026-06-03T08:51:54Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1022-1023].eqiad.wmnet 2422904 wikitext text/x-wiki ndkd4o8cw1mda7333h3ixmucq4yy4g8 2422905 2422904 2026-06-03T08:52:11Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet 2422905 wikitext text/x-wiki esumejp5d224gmi0kjo9sm4b6q20wxx 2422906 2422905 2026-06-03T08:52:13Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet 2422906 wikitext text/x-wiki htumcnwawkainh8biy83h5rtilg4k65 2422907 2422906 2026-06-03T08:52:58Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2215: Migration of db2215.codfw.wmnet completed 2422907 wikitext text/x-wiki otjhmks8fo47orc526sz97hb3kt53lo 2422908 2422907 2026-06-03T08:53:08Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296635|Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 11m 43s) 2422908 wikitext text/x-wiki ewvkoj4as08skl67qc5sb452ra3uk67 2422909 2422908 2026-06-03T08:55:07Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297064|Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] 2422909 wikitext text/x-wiki 2bdwd1xgrhe2pz3mosssyi1fe36y2g0 2422910 2422909 2026-06-03T08:59:05Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297064|Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422910 wikitext text/x-wiki rxl63r9wg3kgv57zzc53v50eczuih9b 2422911 2422910 2026-06-03T08:59:41Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2422911 wikitext text/x-wiki hjywz3mpdsk3e51fri4zdt5rre0zeq1 2422912 2422911 2026-06-03T09:00:16Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" 2422912 wikitext text/x-wiki 04uhzibdu1jnd4426m7hjdsfbg8tspf 2422913 2422912 2026-06-03T09:00:18Z Stashbot 7414 ayounsi@cumin1003: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" 2422913 wikitext text/x-wiki 2ku9nuw0gk37mnbcgsxkq3gc6d77qsw 2422914 2422913 2026-06-03T09:00:37Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - T422043" 2422914 wikitext text/x-wiki 6vqsrwpha8jqkb32pnmwx7dg8dppyrd 2422915 2422914 2026-06-03T09:00:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T426633)', diff saved to https://phabricator.wikimedia.org/P93656 and previous config saved to /var/cache/conftool/dbconfig/20260603-090056-fceratto.json 2422915 wikitext text/x-wiki f4v9jxwmso0hcc2bokhwshiuewgu064 2422916 2422915 2026-06-03T09:01:21Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - T422043" 2422916 wikitext text/x-wiki 1pkg8by460g85tan6omctqtdjhxsdgh 2422917 2422916 2026-06-03T09:04:20Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2422917 wikitext text/x-wiki bdp2fy3cin6340ibbo73nod84xaf3e5 2422918 2422917 2026-06-03T09:05:55Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply 2422918 wikitext text/x-wiki r2ozwisfbzcf4q33cahp57klzuc8979 2422919 2422918 2026-06-03T09:06:01Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297064|Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 10m 54s) 2422919 wikitext text/x-wiki cnl4hev81tvb862o1cbosjwgiqq7wk4 2422920 2422919 2026-06-03T09:06:21Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply 2422920 wikitext text/x-wiki sncubt59g9rplv6ld6s3wty1ygqcip2 2422921 2422920 2026-06-03T09:07:50Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297065|Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] 2422921 wikitext text/x-wiki d7nyktrpgxgnrrdhlcoixcrcsp5bywc 2422922 2422921 2026-06-03T09:09:47Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297065|Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422922 wikitext text/x-wiki kglvssl21biwcocq3nzirbna4j7xyt2 2422923 2422922 2026-06-03T09:10:42Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2422923 wikitext text/x-wiki aho7e6ddtt5fxvu7jfz5v9aly1ttidr 2422924 2422923 2026-06-03T09:11:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93659 and previous config saved to /var/cache/conftool/dbconfig/20260603-091104-fceratto.json 2422924 wikitext text/x-wiki q7f03qqmyzoey384oe33w7e7a24gdzp 2422925 2422924 2026-06-03T09:14:57Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297065|Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] (duration: 07m 06s) 2422925 wikitext text/x-wiki kkf04whsnkskf0tw82diezyk9hsv9tz 2422926 2422925 2026-06-03T09:20:01Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297069|hCaptcha: Collect risk score for blocked account creations (T427784)]] 2422926 wikitext text/x-wiki 90xd417oxlegnhe8qvl2xlgwvnbqe07 2422927 2422926 2026-06-03T09:20:03Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2422927 wikitext text/x-wiki si2jck2f9n4at877mtofvt4apn1zdly 2422928 2422927 2026-06-03T09:21:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93661 and previous config saved to /var/cache/conftool/dbconfig/20260603-092111-fceratto.json 2422928 wikitext text/x-wiki ivbev9q9vw561r4knft2pieqv9hvdc1 2422929 2422928 2026-06-03T09:21:27Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply 2422929 wikitext text/x-wiki 7mjhv82dsyay0s39hzcrxsyurwwbwgr 2422930 2422929 2026-06-03T09:21:38Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2054: repool after upgrade 2422930 wikitext text/x-wiki 7okih6ohsmzmx7jk82887uqpopvyo4v 2422931 2422930 2026-06-03T09:21:40Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply 2422931 wikitext text/x-wiki econgtup4qdkvo5f1tc2bmtb9in7gk4 2422932 2422931 2026-06-03T09:21:53Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297069|hCaptcha: Collect risk score for blocked account creations (T427784)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422932 wikitext text/x-wiki eqnzxmidc9y8t2yb3ffiealn3y71imk 2422933 2422932 2026-06-03T09:22:14Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422933 wikitext text/x-wiki 467k8leq4287yu63wi4zx8mizyq26zt 2422934 2422933 2026-06-03T09:22:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1053: Upgrading es1053.eqiad.wmnet 2422934 wikitext text/x-wiki emqvlqjxoplq195dxp3avx2f3t3wj6q 2422935 2422934 2026-06-03T09:23:15Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2422935 wikitext text/x-wiki 1tzuc0x71km3q1tc8wb1byjl1beclch 2422936 2422935 2026-06-03T09:23:54Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1053: Upgrading es1053.eqiad.wmnet 2422936 wikitext text/x-wiki gidd7xs9mgd5ojo982ohpk21rqbnlav 2422937 2422936 2026-06-03T09:24:24Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" 2422937 wikitext text/x-wiki 5f2kh3ydzd61ykq4jxiywxluics9ota 2422938 2422937 2026-06-03T09:24:28Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" 2422938 wikitext text/x-wiki s3ujd20ntkio256mv2kna1jalcqy4mi 2422939 2422938 2026-06-03T09:24:30Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2422939 wikitext text/x-wiki 5avdphlf1ya2r4de8p7sfm8k10milcv 2422940 2422939 2026-06-03T09:25:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1053.eqiad.wmnet with OS trixie 2422940 wikitext text/x-wiki rm7p7oy4lrc25fnneb8xoz3t85vvsin 2422941 2422940 2026-06-03T09:27:27Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297069|hCaptcha: Collect risk score for blocked account creations (T427784)]] (duration: 07m 26s) 2422941 wikitext text/x-wiki inx9n3lx4jfott2ftuphui4u09fjpkl 2422942 2422941 2026-06-03T09:28:57Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1211: Migration of db1211.eqiad.wmnet completed 2422942 wikitext text/x-wiki 4tf1l6cpjveb8c3jq9nx16q5fg3g9xj 2422943 2422942 2026-06-03T09:28:59Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2422943 wikitext text/x-wiki fdavcfwk5y8wgu84ty19op6gotzi126 2422944 2422943 2026-06-03T09:31:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T426633)', diff saved to https://phabricator.wikimedia.org/P93666 and previous config saved to /var/cache/conftool/dbconfig/20260603-093119-fceratto.json 2422944 wikitext text/x-wiki qw5ht9dzxr68wdzp7qkxjk5cxybzsxl 2422946 2422944 2026-06-03T09:31:40Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance 2422946 wikitext text/x-wiki dkq8ujm5jubz0qp89i6fe6d1lds493w 2422947 2422946 2026-06-03T09:31:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1175 (T426633)', diff saved to https://phabricator.wikimedia.org/P93667 and previous config saved to /var/cache/conftool/dbconfig/20260603-093146-fceratto.json 2422947 wikitext text/x-wiki 5fhksd2tat7og5ipn6ve97dz4s3jr79 2422948 2422947 2026-06-03T09:38:15Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet 2422948 wikitext text/x-wiki no63dmcevv9vp97no75jkgj9h8aodm6 2422949 2422948 2026-06-03T09:38:28Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2215: Migration of db2215.codfw.wmnet completed 2422949 wikitext text/x-wiki 0z16jrd8gb7nybwujiod8zvfwk0keyp 2422950 2422949 2026-06-03T09:38:30Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2422950 wikitext text/x-wiki 0107s8m9rx9h08onyhjuw5m3zvvducw 2422951 2422950 2026-06-03T09:40:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T426633)', diff saved to https://phabricator.wikimedia.org/P93669 and previous config saved to /var/cache/conftool/dbconfig/20260603-094014-fceratto.json 2422951 wikitext text/x-wiki ca1ah37l8upvrtj38xt3jwd187eqwlu 2422952 2422951 2026-06-03T09:41:50Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage 2422952 wikitext text/x-wiki dkftd0dhjt7gn4avk7lbnvsrq79lxaw 2422953 2422952 2026-06-03T09:41:54Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage 2422953 wikitext text/x-wiki gu9p710h07dlmoppnjtq3ovlrchfrdl 2422954 2422953 2026-06-03T09:43:45Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet 2422954 wikitext text/x-wiki 0isfleif0ni4qdzisx4dfxwxk5smiha 2422955 2422954 2026-06-03T09:47:45Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2422955 wikitext text/x-wiki 6f4dc50udiz1p77x65ucz2tv9fzbn6g 2422956 2422955 2026-06-03T09:48:51Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1053.eqiad.wmnet with OS trixie 2422956 wikitext text/x-wiki 4jmiwy2skjja72m9aj0bdotof3tcvm8 2422957 2422956 2026-06-03T09:49:36Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2422957 wikitext text/x-wiki ne3ssmz3gv7llzhttw2tn6i79www6f6 2422958 2422957 2026-06-03T09:49:42Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2422958 wikitext text/x-wiki pn0jdmmb92w0ryltj59aix49dccfd4n 2422959 2422958 2026-06-03T09:50:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93670 and previous config saved to /var/cache/conftool/dbconfig/20260603-095022-fceratto.json 2422959 wikitext text/x-wiki m5uaq5xj4lsa2knh38q5cmijeg0ifps 2422960 2422959 2026-06-03T09:51:06Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422960 wikitext text/x-wiki f79a3m3lo6isjs4rzrzir10cegzi3vf 2422961 2422960 2026-06-03T09:51:10Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422961 wikitext text/x-wiki t65vi1jng14gfkmq3zapdvwpt7qkkmf 2422962 2422961 2026-06-03T09:51:12Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422962 wikitext text/x-wiki tsacqx89hm9b8k8zyy8niidu611o2tu 2422963 2422962 2026-06-03T09:52:06Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422963 wikitext text/x-wiki d07zahq7ejkj9tocqv659w6gqi53bxk 2422964 2422963 2026-06-03T09:52:07Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422964 wikitext text/x-wiki c98if7wbq5bzw2bbrcawie26piv5k9p 2422965 2422964 2026-06-03T09:52:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422965 wikitext text/x-wiki 9ckdcolux3u0fi3jfdltw2zc10ukp38 2422966 2422965 2026-06-03T09:52:35Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2422966 wikitext text/x-wiki 1nju9e5adwa7mqem30m1foix5nqfmez 2422967 2422966 2026-06-03T09:52:54Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1053: repool after upgrade 2422967 wikitext text/x-wiki sdeajp1oboukysuu94mhb90hd84fa74 2422968 2422967 2026-06-03T09:53:01Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2422968 wikitext text/x-wiki o1uizti38648qnbor586802hiem18wn 2422970 2422968 2026-06-03T09:53:09Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2422970 wikitext text/x-wiki 98e6oxv69i8oul33figisrcesyf2dut 2422971 2422970 2026-06-03T09:53:22Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2422971 wikitext text/x-wiki dmi0hxd4ux9lutbkgto99426nfervai 2422972 2422971 2026-06-03T09:57:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2422972 wikitext text/x-wiki pw5p4lzh7yc4sggpmcb3urlr224tag9 2422973 2422972 2026-06-03T09:57:22Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2196: Upgrading db2196.codfw.wmnet 2422973 wikitext text/x-wiki 393rv7t2c7ibowczml79ez5evdga248 2422974 2422973 2026-06-03T09:57:43Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: Upgrading db2196.codfw.wmnet 2422974 wikitext text/x-wiki hbiduffc7di2t7r9jyv0ieo20duxfg9 2422975 2422974 2026-06-03T09:59:10Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2196.codfw.wmnet with OS trixie 2422975 wikitext text/x-wiki pgydi2m9pllnax3ruztmjw3x2x94v98 2422976 2422975 2026-06-03T10:00:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93673 and previous config saved to /var/cache/conftool/dbconfig/20260603-100029-fceratto.json 2422976 wikitext text/x-wiki kud66djckixyieseqmu9h6ih4xdur3v 2422978 2422976 2026-06-03T10:10:21Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet 2422978 wikitext text/x-wiki 6di5dopq3y26f2k4fur4u719qwp8spu 2422979 2422978 2026-06-03T10:10:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T426633)', diff saved to https://phabricator.wikimedia.org/P93675 and previous config saved to /var/cache/conftool/dbconfig/20260603-101037-fceratto.json 2422979 wikitext text/x-wiki m2k45hol749vsy5smepu8ruervc5cvh 2422980 2422979 2026-06-03T10:10:59Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance 2422980 wikitext text/x-wiki itc1brlh7k3oky6ttcwuu7uzen48532 2422981 2422980 2026-06-03T10:11:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 (T426633)', diff saved to https://phabricator.wikimedia.org/P93676 and previous config saved to /var/cache/conftool/dbconfig/20260603-101105-fceratto.json 2422981 wikitext text/x-wiki satffrpnvwelemmq9txbmq264ulmvah 2422982 2422981 2026-06-03T10:15:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2196.codfw.wmnet with reason: host reimage 2422982 wikitext text/x-wiki t9daonavtmvkoccz5gt2xpl7qef2jmo 2422983 2422982 2026-06-03T10:15:44Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet 2422983 wikitext text/x-wiki rqaq6cw7wqgsi74gahb75ehceeafc6n 2422984 2422983 2026-06-03T10:19:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T426633)', diff saved to https://phabricator.wikimedia.org/P93677 and previous config saved to /var/cache/conftool/dbconfig/20260603-101916-fceratto.json 2422984 wikitext text/x-wiki t7tln58ravptwujjqztm6oiziu9eci8 2422985 2422984 2026-06-03T10:19:56Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2196.codfw.wmnet with reason: host reimage 2422985 wikitext text/x-wiki lo8gy7gzrb54mum3zrsw3y5ojg7tivc 2422987 2422985 2026-06-03T10:21:29Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1066 2422987 wikitext text/x-wiki 6tw6n0ckxkkq275qgv0rbigddsabeka 2422988 2422987 2026-06-03T10:22:18Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1067 2422988 wikitext text/x-wiki rexc0w58t397a28363htpy2n0962ioy 2422989 2422988 2026-06-03T10:24:55Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297090|hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] 2422989 wikitext text/x-wiki 5hzf2az0a4dfgbasdmwror9vpvx0g56 2422990 2422989 2026-06-03T10:26:52Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297090|hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2422990 wikitext text/x-wiki 2qxet0y6liit5tzhpmy9o0p1833ixci 2422991 2422990 2026-06-03T10:29:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93679 and previous config saved to /var/cache/conftool/dbconfig/20260603-102924-fceratto.json 2422991 wikitext text/x-wiki iy076mkimvp3h4ylft3im0mevlrq0uv 2422992 2422991 2026-06-03T10:30:37Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2422992 wikitext text/x-wiki gn2k6mebe9h7vqtfgffemqlajfa7kfr 2422993 2422992 2026-06-03T10:31:53Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2422993 wikitext text/x-wiki lhod41aq81ok2z4qm7gt4s102mmf5a9 2422994 2422993 2026-06-03T10:32:47Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2422994 wikitext text/x-wiki 9wgsra391b6vwi87hvjh3ijvg7r0gz4 2422995 2422994 2026-06-03T10:36:58Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297090|hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] (duration: 12m 03s) 2422995 wikitext text/x-wiki n835fftw2n8zsafhcftazi7xm1dhwy4 2422996 2422995 2026-06-03T10:37:54Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2196.codfw.wmnet with OS trixie 2422996 wikitext text/x-wiki 3pzwm2hv0xjwj4zw01ziv4wku14s3c1 2422997 2422996 2026-06-03T10:38:17Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1053: repool after upgrade 2422997 wikitext text/x-wiki 9tfe2ke8ga7vcwhh9x0zzvsys0hrk45 2422998 2422997 2026-06-03T10:39:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93681 and previous config saved to /var/cache/conftool/dbconfig/20260603-103931-fceratto.json 2422998 wikitext text/x-wiki hd16epnm542dobswsw961x8xdft92n0 2422999 2422998 2026-06-03T10:40:06Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2422999 wikitext text/x-wiki e7ddxwl32esyyuifz7kb8nb5yu0wub8 2423000 2422999 2026-06-03T10:40:43Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2423000 wikitext text/x-wiki ki6mt7udzhr7czjtwn2255sclofzsv3 2423001 2423000 2026-06-03T10:40:58Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2423001 wikitext text/x-wiki s9zuce4pb3pxgahroojne3lesdk1w7a 2423002 2423001 2026-06-03T10:41:05Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2423002 wikitext text/x-wiki jbjwpiatsgh4nd2kw48h0z818cpwl7z 2423004 2423002 2026-06-03T10:44:17Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply 2423004 wikitext text/x-wiki fftmj449nfq39fkppzogto1evxmkd07 2423005 2423004 2026-06-03T10:44:38Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Migration of db2196.codfw.wmnet completed 2423005 wikitext text/x-wiki dwmesfsyt6rys64f5i8oxkpnkvp8mt9 2423006 2423005 2026-06-03T10:45:03Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2423006 wikitext text/x-wiki 90abv9fbnvv5pvvpyfe6wq4zfjf6p2b 2423007 2423006 2026-06-03T10:45:18Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2423007 wikitext text/x-wiki lwrlqc1kmabki2ktok9ahuhp1sledyw 2423008 2423007 2026-06-03T10:49:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T426633)', diff saved to https://phabricator.wikimedia.org/P93683 and previous config saved to /var/cache/conftool/dbconfig/20260603-104939-fceratto.json 2423008 wikitext text/x-wiki 3ly5wwl60umzditunla9gk3s2k62llg 2423009 2423008 2026-06-03T10:50:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance 2423009 wikitext text/x-wiki 2uyiy8k7tozxuuhc2drr0g64rqjqdna 2423010 2423009 2026-06-03T10:50:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1198 (T426633)', diff saved to https://phabricator.wikimedia.org/P93684 and previous config saved to /var/cache/conftool/dbconfig/20260603-105006-fceratto.json 2423010 wikitext text/x-wiki 6e22qlfs39bhp9uh1w0j1h6o3mlx9by 2423011 2423010 2026-06-03T10:51:58Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2423011 wikitext text/x-wiki eck3guk04qh9l7kss29ef77unhy83vn 2423012 2423011 2026-06-03T10:53:17Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1289895|Update UserInfoCard to be enabled by default for certain user groups (T426021)]] 2423012 wikitext text/x-wiki kcgff0lr2musbiem1pfmdsdrd6tdlpy 2423013 2423012 2026-06-03T10:53:48Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2423013 wikitext text/x-wiki a7iz4kh9jtep13w02j5wyef7glbh17l 2423014 2423013 2026-06-03T10:54:17Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2423014 wikitext text/x-wiki shvynmt4q3rlgnd50g3a06vd29zk560 2423015 2423014 2026-06-03T10:54:34Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply 2423015 wikitext text/x-wiki mqh7isxabt0nle6hul5xrdqujf6ikks 2423016 2423015 2026-06-03T10:55:14Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1289895|Update UserInfoCard to be enabled by default for certain user groups (T426021)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2423016 wikitext text/x-wiki 10bjgib85i3ad7zhl2mrqseuecf7jtj 2423017 2423016 2026-06-03T10:56:42Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2423017 wikitext text/x-wiki 6tuh7ygxopnkwyhi8imxglsffw2ng6w 2423018 2423017 2026-06-03T10:57:01Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2423018 wikitext text/x-wiki 1rzh7ltihjfrcx2vqocdyf8wljpvwgh 2423019 2423018 2026-06-03T10:57:26Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423019 wikitext text/x-wiki jr3inpzkzcy0hshfol9r88e01wqfroe 2423020 2423019 2026-06-03T10:58:03Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply 2423020 wikitext text/x-wiki lgekvi5efpcx1yi9uzw0tzc80fjwhlx 2423021 2423020 2026-06-03T10:58:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T426633)', diff saved to https://phabricator.wikimedia.org/P93685 and previous config saved to /var/cache/conftool/dbconfig/20260603-105815-fceratto.json 2423021 wikitext text/x-wiki 0bca9v1cj55hfo68h18x91djtg2uoqf 2423022 2423021 2026-06-03T10:58:26Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply 2423022 wikitext text/x-wiki iepn7o5na9ss2usdkr0r01r3nhbau83 2423023 2423022 2026-06-03T10:58:39Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply 2423023 wikitext text/x-wiki dcln7qbl0yqvpsn3vt4vjordvbdctrt 2423024 2423023 2026-06-03T10:59:16Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423024 wikitext text/x-wiki 08a4yigv6yzrsvsf4kejhgon9mmsgkc 2423025 2423024 2026-06-03T10:59:40Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply 2423025 wikitext text/x-wiki g8femsvhiprybclnvfm98uc39fnkzqv 2423026 2423025 2026-06-03T10:59:46Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423026 wikitext text/x-wiki pefvmdm4bp4au5w8clm7mw3clv8bb6c 2423027 2423026 2026-06-03T10:59:52Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply 2423027 wikitext text/x-wiki dz6spfllt5khd7dm2bfnk8i2vb2jkug 2423028 2423027 2026-06-03T11:00:07Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423028 wikitext text/x-wiki 9r7kff7t81z6x78907m0u9hqt8fd9n6 2423029 2423028 2026-06-03T11:00:54Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289895|Update UserInfoCard to be enabled by default for certain user groups (T426021)]] (duration: 07m 37s) 2423029 wikitext text/x-wiki 5tltiualzg78ukt9w9mfz7xok7tr2bp 2423030 2423029 2026-06-03T11:01:07Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423030 wikitext text/x-wiki d6bew6f7rmqge6zp6s6yilt1e3f8p4q 2423031 2423030 2026-06-03T11:01:14Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423031 wikitext text/x-wiki 2chzrcelcwvvtlunrlrj9knuf7qva9c 2423032 2423031 2026-06-03T11:02:01Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423032 wikitext text/x-wiki 398um36269d0820890icaeyiykw1ffv 2423033 2423032 2026-06-03T11:02:34Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2423033 wikitext text/x-wiki qkmous89zjxo0t99rwfsh6h40bgaefx 2423034 2423033 2026-06-03T11:02:36Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423034 wikitext text/x-wiki oa53leffbag2pd76vdt6a2lovfwx6jq 2423035 2423034 2026-06-03T11:02:50Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423035 wikitext text/x-wiki f486qaie12ldbor8flc9ljwnjj4m3yy 2423036 2423035 2026-06-03T11:02:59Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423036 wikitext text/x-wiki lyjdwadzgl0b59rwnv6ndg18nroe6k2 2423037 2423036 2026-06-03T11:03:18Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2423037 wikitext text/x-wiki javv1t8pxc445tl35tle1jqkyc4yapn 2423038 2423037 2026-06-03T11:05:11Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply 2423038 wikitext text/x-wiki 5cg49cw90nmw82hm6doujgpyuv06wen 2423039 2423038 2026-06-03T11:06:08Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply 2423039 wikitext text/x-wiki gl7y5x21a0cir0lez7jbly1vrgzqmiy 2423040 2423039 2026-06-03T11:07:06Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply 2423040 wikitext text/x-wiki d8i2ycns8u1txmbylfkxzsp51cz0emy 2423041 2423040 2026-06-03T11:07:09Z Stashbot 7414 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1066 2423041 wikitext text/x-wiki 8fsp2b2etqxtdvixe1k9ny0vrxdn2w2 2423042 2423041 2026-06-03T11:08:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93687 and previous config saved to /var/cache/conftool/dbconfig/20260603-110823-fceratto.json 2423042 wikitext text/x-wiki g2v7ol9nflufl8r7mgyq5wd131hzup6 2423043 2423042 2026-06-03T11:08:41Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply 2423043 wikitext text/x-wiki 3epszhaczc8v6srtvskua8mkjtg9n46 2423044 2423043 2026-06-03T11:09:04Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply 2423044 wikitext text/x-wiki kmdr712w4be6wc6ipjgte1hjrpyt9xw 2423045 2423044 2026-06-03T11:09:32Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply 2423045 wikitext text/x-wiki n7xjnkk07qsz3axm2sboltr6xpnhyep 2423047 2423045 2026-06-03T11:14:51Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2423047 wikitext text/x-wiki eoytf6nz4e7y2edu4116n2jc2i4ubw6 2423052 2423047 2026-06-03T11:18:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93689 and previous config saved to /var/cache/conftool/dbconfig/20260603-111831-fceratto.json 2423052 wikitext text/x-wiki 3fxkpuak0fgpmv0v40bstpt2ohxf8dw 2423053 2423052 2026-06-03T11:19:53Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423053 wikitext text/x-wiki g1nhez3lahyju3qw5d4uh9es4kb51w3 2423054 2423053 2026-06-03T11:20:04Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423054 wikitext text/x-wiki rh477xedzjvfeyaisb1ccspeg11xn0i 2423055 2423054 2026-06-03T11:20:53Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423055 wikitext text/x-wiki qeu9sk6xygsip28r0odsmewifa8700q 2423056 2423055 2026-06-03T11:20:59Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423056 wikitext text/x-wiki 4dpfo87sb614fd46xoy3yhnsod35fic 2423057 2423056 2026-06-03T11:24:54Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2423057 wikitext text/x-wiki f9yh36spn0d2tmw9xa56av1cutweyzv 2423059 2423057 2026-06-03T11:28:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T426633)', diff saved to https://phabricator.wikimedia.org/P93690 and previous config saved to /var/cache/conftool/dbconfig/20260603-112838-fceratto.json 2423059 wikitext text/x-wiki 3fie92ccjh3d3v70gf2hgnkgrrgy7fx 2423060 2423059 2026-06-03T11:28:51Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance 2423060 wikitext text/x-wiki dhgn9lfrik4phl4rqk74grztzwuuc6r 2423061 2423060 2026-06-03T11:29:03Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2423061 wikitext text/x-wiki t35yd63ce5aoz3up8xsr04lmh7xf3v1 2423062 2423061 2026-06-03T11:29:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1212 (T426633)', diff saved to https://phabricator.wikimedia.org/P93691 and previous config saved to /var/cache/conftool/dbconfig/20260603-112909-fceratto.json 2423062 wikitext text/x-wiki 48ce0i8hqjp0cs4m2r90244abq1vr4w 2423063 2423062 2026-06-03T11:30:07Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Migration of db2196.codfw.wmnet completed 2423063 wikitext text/x-wiki sw6ce5buqdq1ezpd3atus56u4iflwy9 2423064 2423063 2026-06-03T11:30:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2423064 wikitext text/x-wiki euwxizcrxnqevkkpi9dihmqim7lrpsb 2423065 2423064 2026-06-03T11:32:13Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423065 wikitext text/x-wiki 7pfaf52tj4vjjfmsvqlus6djfkuvmt6 2423066 2423065 2026-06-03T11:32:31Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423066 wikitext text/x-wiki 1h914daa5chq4tby0jz91efrbjv95eq 2423067 2423066 2026-06-03T11:33:20Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423067 wikitext text/x-wiki 52nzjw1u48h0s5knf16p4dsdorthr8o 2423068 2423067 2026-06-03T11:33:31Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2423068 wikitext text/x-wiki d3wq6jwpdvs35bene6vxtf99kow0y36 2423069 2423068 2026-06-03T11:36:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T426633)', diff saved to https://phabricator.wikimedia.org/P93693 and previous config saved to /var/cache/conftool/dbconfig/20260603-113611-fceratto.json 2423069 wikitext text/x-wiki lqtc2asyjekn046lwi2tr1dfxcg2aqv 2423070 2423069 2026-06-03T11:39:33Z Stashbot 7414 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1067 2423070 wikitext text/x-wiki i0bxwu7y0uw85624lqp5hivgwalzyme 2423071 2423070 2026-06-03T11:40:02Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1066.eqiad.wmnet with OS bullseye 2423071 wikitext text/x-wiki jca169kvim1f46v0ph6l6oep08doo4a 2423072 2423071 2026-06-03T11:40:24Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1066 2423072 wikitext text/x-wiki aqeo4svojxo7ccry8umwx6djpd766fj 2423073 2423072 2026-06-03T11:41:02Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2423073 wikitext text/x-wiki khgbq1ouu3ccuc1vgytf5te0xfohwfk 2423074 2423073 2026-06-03T11:42:04Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply 2423074 wikitext text/x-wiki tg85jla5c5xidu9jqmh0es7h25z90ca 2423075 2423074 2026-06-03T11:42:15Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply 2423075 wikitext text/x-wiki gouo00qsw325f4vil52vd1o9isr5v78 2423076 2423075 2026-06-03T11:42:22Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply 2423076 wikitext text/x-wiki 03ki4vi43qrod5lfarneyvfgzrfy7ed 2423077 2423076 2026-06-03T11:42:41Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply 2423077 wikitext text/x-wiki divgl7ub49w5cbc91pvl4pjnbrgn067 2423078 2423077 2026-06-03T11:42:49Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply 2423078 wikitext text/x-wiki 6g2ylpm77kzq7gg2pusxfvjmh4r96e3 2423079 2423078 2026-06-03T11:43:13Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply 2423079 wikitext text/x-wiki 34iphqlkxc207d5eb4enkyt9g77k604 2423080 2423079 2026-06-03T11:45:16Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" 2423080 wikitext text/x-wiki ieangfgfn2mmn8o9c3yo6gxh7goujsc 2423081 2423080 2026-06-03T11:45:23Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" 2423081 wikitext text/x-wiki 7v48rtpazbvrwnpfmxurahuqjswj39f 2423082 2423081 2026-06-03T11:45:25Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2423082 wikitext text/x-wiki rh9tdughcu5d9i3603mqhl384zw7lkz 2423083 2423082 2026-06-03T11:45:27Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2423083 wikitext text/x-wiki g8daljxsabgqjyg5cvyol2b3aqs0a3s 2423084 2423083 2026-06-03T11:45:28Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2423084 wikitext text/x-wiki q6frwx9xnco0m3prj5s49r1f7w78eez 2423085 2423084 2026-06-03T11:45:30Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1066 2423085 wikitext text/x-wiki c045wtbvwzrgql0kfrmajamjsis1a0g 2423086 2423085 2026-06-03T11:46:16Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1066 2423086 wikitext text/x-wiki e933ga3uequ6ri34y9yw8j7p4k7l1ft 2423087 2423086 2026-06-03T11:46:18Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1066 2423087 wikitext text/x-wiki l610qtisl118vyo1s7ue5lsdugrj98n 2423088 2423087 2026-06-03T11:46:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93695 and previous config saved to /var/cache/conftool/dbconfig/20260603-114618-fceratto.json 2423088 wikitext text/x-wiki nidqac8idxpzskau53c7xgt9lvy8vvk 2423089 2423088 2026-06-03T11:46:40Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1067.eqiad.wmnet with OS bullseye 2423089 wikitext text/x-wiki au2kbn6b8d0ce364grv1h3yf7zgvuyz 2423090 2423089 2026-06-03T11:46:59Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1067 2423090 wikitext text/x-wiki hiu2ggavl94eeafuc78l23ruvepk9mz 2423091 2423090 2026-06-03T11:47:35Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2423091 wikitext text/x-wiki 6j7g4tmz2corb2lspddog1xfc49k6ww 2423092 2423091 2026-06-03T11:48:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2423092 wikitext text/x-wiki qq37o7tccpqlqexgrabt2ge77ho48mw 2423093 2423092 2026-06-03T11:48:36Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2186: Upgrading db2186.codfw.wmnet 2423093 wikitext text/x-wiki 3kd1exvtknpdz8h556r4xmebmpfqppy 2423094 2423093 2026-06-03T11:48:54Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2186: Upgrading db2186.codfw.wmnet 2423094 wikitext text/x-wiki 1haktq78sa0c09iny70i0rl00cc2t3y 2423095 2423094 2026-06-03T11:52:08Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" 2423095 wikitext text/x-wiki l92iuy840wbrgr8f6tbcs1drlhhjzaf 2423096 2423095 2026-06-03T11:52:13Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" 2423096 wikitext text/x-wiki i45qc8qn8s56qdxqtc7fzrovoov9iwj 2423097 2423096 2026-06-03T11:52:15Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2423097 wikitext text/x-wiki 52csuu4toqb1pe0km5f743eikkh85v6 2423098 2423097 2026-06-03T11:52:17Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2423098 wikitext text/x-wiki mreol0pozub3c9h35vyujxkjx9f9uax 2423099 2423098 2026-06-03T11:52:19Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2423099 wikitext text/x-wiki bkckeejfwfj9x8xbvy8nh8q26ft8dqf 2423100 2423099 2026-06-03T11:52:21Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1067 2423100 wikitext text/x-wiki 2vqv7m89beyskcvn5kan0met70czvwv 2423101 2423100 2026-06-03T11:54:12Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1067 2423101 wikitext text/x-wiki n7v3aexr739qm2b17rqvb7l1jc6vtw1 2423102 2423101 2026-06-03T11:54:14Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1067 2423102 wikitext text/x-wiki h13x3l6iebe2d3w8mpzmypnaj20kofx 2423103 2423102 2026-06-03T11:54:30Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2186.codfw.wmnet with OS trixie 2423103 wikitext text/x-wiki 9ptkm1nhdr4m0yicaaladxh3cld3v4c 2423104 2423103 2026-06-03T11:56:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93697 and previous config saved to /var/cache/conftool/dbconfig/20260603-115626-fceratto.json 2423104 wikitext text/x-wiki 1a7ymqrdofzm8j9lqlqkpot8wl5lo2p 0 8495 2422588 2403414 2026-06-02T17:15:46Z BCornwall-WMF 31521 BCornwall-WMF moved page [[X-Analytics]] to [[CDN/Headers]]: Consolidating all CDN header articles into one page 2403414 wikitext text/x-wiki nvwhkchpwqqza2t172cqcvnm4fzyyzo 2422593 2422588 2026-06-02T17:19:45Z BCornwall-WMF 31521 Add "see" template, headings 2422593 wikitext text/x-wiki 66bu4d4qjn9qe00hp716t0dcazkbcfu 2422594 2422593 2026-06-02T17:21:13Z BCornwall-WMF 31521 Import [[X-Provenance]] 2422594 wikitext text/x-wiki lrpg8go91rh3lblaff36n18fnwndz0c 2422597 2422594 2026-06-02T17:24:26Z BCornwall-WMF 31521 Import [[CDN#Headers]] 2422597 wikitext text/x-wiki 5l385vstfb6tq07lt1yz9b1ej9rjzgc 2422598 2422597 2026-06-02T17:25:40Z BCornwall-WMF 31521 Move x-varnish-cluster to new [[#Historical headers]] section 2422598 wikitext text/x-wiki dx4sfm7kft5dk70tpopqug911cfjpow 2422599 2422598 2026-06-02T17:26:37Z BCornwall-WMF 31521 Add categories 2422599 wikitext text/x-wiki 9hkcc6rnhxbscoxb58k7jtdonvh93zg 2422603 2422599 2026-06-02T17:31:54Z BCornwall-WMF 31521 Import [[X-Image-Generator]] 2422603 wikitext text/x-wiki 9ki5bgviijbxun0sjuzwm52rlt82jez 2422604 2422603 2026-06-02T17:33:12Z BCornwall-WMF 31521 Merge [[#See also]] sections 2422604 wikitext text/x-wiki qpk8mz9nzdbhemq7tiqkei8y2dq85vt 2422621 2422604 2026-06-02T17:45:20Z BCornwall-WMF 31521 /* X-Varnish-Cluster */ past tense 2422621 wikitext text/x-wiki cks5nvntjh0jw46kwuzx6i2scdxa4qh 2422636 2422621 2026-06-02T17:53:16Z BCornwall-WMF 31521 Add intro blurb 2422636 wikitext text/x-wiki lz73mrjo62hwv25rhk66gtv0tr3bbib 2422644 2422636 2026-06-02T17:58:44Z BCornwall-WMF 31521 Add "set by"/"returned to client" table for X-image-generator and X-Provenance 2422644 wikitext text/x-wiki rt8n1wkn6aw91ob9sgdov4pcj3tblfq 2422678 2422644 2026-06-02T18:15:24Z BCornwall-WMF 31521 Adopt similar general formatting for X-Provenance and X-Image-Generator sections 2422678 wikitext text/x-wiki n05foeejptmu0eeokllkurzb6r71j20 2422680 2422678 2026-06-02T18:16:06Z BCornwall-WMF 31521 Promote inccorect X-Forwarded-Proto header to correct h2 2422680 wikitext text/x-wiki 9lxfdoxcdklu9gpul3ily7ra9vv6eaj 2422681 2422680 2026-06-02T18:16:49Z BCornwall-WMF 31521 /* Keys */ Remove old, irrelevant comment 2422681 wikitext text/x-wiki 63uq8j7beoc4hr4wuxasm121aual5qi 2422682 2422681 2026-06-02T18:18:52Z BCornwall-WMF 31521 Rename "Set by" to "Origin" for consistency with [[#X-Analytics]] column header 2422682 wikitext text/x-wiki a2wgfo9t8kmqg0etx3dljtme9obvdir 2422688 2422682 2026-06-02T18:25:26Z BCornwall-WMF 31521 /* X-Analytics */ Remove unnecessary info sections that are available in the mediawiki extension page, link that instead 2422688 wikitext text/x-wiki f8m7r7ryp3hnhss1jaduvbdq22nzr5x 2422694 2422688 2026-06-02T18:30:15Z BCornwall-WMF 31521 /* Keys */ Use table captions instead of headings for current and former keys 2422694 wikitext text/x-wiki 1lbp4j4h5otnrhm7hajhl8kf3r9koz8 2422698 2422694 2026-06-02T18:35:06Z BCornwall-WMF 31521 /* Keys */ Collapse massive-wide X-Analytics keys tables by default 2422698 wikitext text/x-wiki bg4yty5q3wyzj9jo6otsiknyhbspb2s 2422711 2422698 2026-06-02T19:01:00Z BCornwall-WMF 31521 Merge in X-JA3N and X-JA4H from [[CDN/Backend api]] 2422711 wikitext text/x-wiki hbfvfkyqugjw43sl06ga2t7xnptr8j6 2422712 2422711 2026-06-02T19:04:25Z BCornwall-WMF 31521 Merge in X-Trusted-Request from [[CDN/Backend api]] 2422712 wikitext text/x-wiki 3ex8nblremrececx8ibg8xrw70g1n20 2422714 2422712 2026-06-02T19:07:23Z BCornwall-WMF 31521 Import X-Is-Browser from [[CDN/Backend api]] 2422714 wikitext text/x-wiki fp96lf6qnkofwrfhuy54zwusoibqsay 2422719 2422714 2026-06-02T19:11:55Z BCornwall-WMF 31521 Import X-UA-Contact from [[CDN/Backend api]] 2422719 wikitext text/x-wiki fqoy9o1hdivr7m9444tottducsyaajv 2422723 2422719 2026-06-02T19:16:25Z BCornwall-WMF 31521 /* See also */ Remove backend api link 2422723 wikitext text/x-wiki ntndgf9vigbbp8uctkp981p8e3wy9mo 2422724 2422723 2026-06-02T19:18:54Z BCornwall-WMF 31521 /* X-Provenance */ Integrate the future plans section into the blurb 2422724 wikitext text/x-wiki m5ubvuah22mc2uofmy5bw1iufl4zmfn 2422726 2422724 2026-06-02T19:42:14Z BCornwall-WMF 31521 /* X-Trusted-Request */ Reword header to value 2422726 wikitext text/x-wiki 932ja6qztu99x8934udel84wq9fegk5 2422727 2422726 2026-06-02T19:43:22Z BCornwall-WMF 31521 /* X-Trusted-Request */ add examples 2422727 wikitext text/x-wiki 79dbkn2x5prh6cnthg5zxwl4ki4k3my 2422768 2422727 2026-06-03T01:48:57Z Krinkle 44 +nav 2422768 wikitext text/x-wiki mzrqnmylq400k1gubyd5ewhag0ggpw5 2422769 2422768 2026-06-03T01:52:34Z Krinkle 44 2422769 wikitext text/x-wiki h7agnpkg5iklxpaxh4s62npykczno5v 2422770 2422769 2026-06-03T01:54:47Z Krinkle 44 /* X-Image-Generator */ move relevant see also link into here 2422770 wikitext text/x-wiki riakxy7lfbn100zifzcxdq71u2dfxlp 2422800 2422770 2026-06-03T06:25:29Z SLyngshede-WMF 31501 Remove clear, which created a weird break in the page. 2422800 wikitext text/x-wiki c9rp85jyudoxhuot52yy9r7dj3l7q8t 2422801 2422800 2026-06-03T06:26:09Z SLyngshede-WMF 31501 Undid revision [[Special:Diff/2422800|2422800]] by [[Special:Contributions/SLyngshede-WMF|SLyngshede-WMF]] ([[User talk:SLyngshede-WMF|talk]]) 2422801 wikitext text/x-wiki riakxy7lfbn100zifzcxdq71u2dfxlp 110 15819 2422779 1832805 2026-06-03T02:02:00Z Krinkle 44 Krinkle moved page [[Metrics.wikimedia.org]] to [[Obsolete:Metrics.wikimedia.org]] 1832805 wikitext text/x-wiki fw7fn0kkaredv3wj78afrffrvysdvw7 2422781 2422779 2026-06-03T02:02:40Z Krinkle 44 2422781 wikitext text/x-wiki je1z35edpbs7xw9c400x2xtrqdvtbw3 0 17290 2422412 2422002 2026-06-02T14:05:51Z Stashbot 7414 jnuche: trying to reconnect Gearman 2422412 wikitext text/x-wiki njbtlwtoso1ucfyp47rz71v4p0yv30t 2422417 2422412 2026-06-02T14:12:01Z Stashbot 7414 jnuche: option to "Enable Gearman" times out. Can't re-enable from UI. Gearman plugin logs are empty. Neat 2422417 wikitext text/x-wiki kjqvghnfve11iiitlacvv9z9ctrbntd 2422430 2422417 2026-06-02T14:23:11Z Stashbot 7414 jnuche: trying to reconnect castor node, see if that helps somehow 2422430 wikitext text/x-wiki a77zxkrjqb8wg6w2wdv0knudbk72hwn 2422432 2422430 2026-06-02T14:28:28Z Stashbot 7414 jnuche: bring back castor node, that didn't help 2422432 wikitext text/x-wiki je9nkcvzfz7ll95sxkgbdr5vvrh5id8 2422442 2422432 2026-06-02T14:38:41Z Stashbot 7414 jnuche: restarting Jenkins 2422442 wikitext text/x-wiki kknf0ntvg9ljp0315lsos5b3knadvf0 2422469 2422442 2026-06-02T14:57:26Z Stashbot 7414 jnuche: Jenkins/Zuul is back 2422469 wikitext text/x-wiki rxnthgig17xhejeldaveqvlznxxx6ck 2422508 2422469 2026-06-02T15:24:50Z Stashbot 7414 hashar: Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/1295865 # T423607 2422508 wikitext text/x-wiki o013krwppniip77954z1z1a2l5wp236 2422531 2422508 2026-06-02T15:49:06Z Stashbot 7414 jnuche: Updating buildkitd to v0.30.0 in gitlab-cloud-runners (T426212) 2422531 wikitext text/x-wiki g8ko7n6e7vgvv9t1cz8mu1aiopavppx 2422549 2422531 2026-06-02T16:05:44Z Stashbot 7414 hashar: integration: upgraded pypy from 7.3.11 (py3.9) to 7.3.20 (py3.11) # T423607 2422549 wikitext text/x-wiki mwk66syikwx1wq684hjfen9x6qmcs3z 2422674 2422549 2026-06-02T18:13:33Z Stashbot 7414 brennen: devtools phab/phorge: deployed work/2026-06-01-merge-phorge to https://phabricator.wmcloud.org/ for testing (T410849) 2422674 wikitext text/x-wiki 2ysts0l6v07m7j2usmmghxh8y5bb241 2422763 2422674 2026-06-02T23:10:56Z Stashbot 7414 thcipriani: tag quibble 1.18.1 @ 4b7959553c095d811426f394165c69ecc13a44eb 2422763 wikitext text/x-wiki c8p6bmks6idrwyhxd3dlk8n52esjbvs 2422765 2422763 2026-06-02T23:23:59Z Stashbot 7414 thcipriani: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1296695 2422765 wikitext text/x-wiki t7hhhorco4kcbsqn53gq19d0qkx6ial 2422873 2422765 2026-06-03T08:26:40Z Stashbot 7414 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1296559 "inference-services: Add LLM generated editing suggestions CI/CD pipelines." # T427794 2422873 wikitext text/x-wiki c8yl04bw5lga0sjdfm78yytv1qaw8tl 0 23733 2423048 2401087 2026-06-03T11:15:00Z CWilliams-WMF 55316 Added notes regarding reimaging of the source 2423048 wikitext text/x-wiki t73gu4zt8ly76tdfdqyfe6cpxuia36w 498 30942 2422743 2419780 2026-06-02T20:31:23Z Stashbot 7414 bd808: `kubectl sudo delete cm -n tool-arb-bot maintain-kubeusers-arb-bot` to trigger regeneration of .kube/config (IRC request) 2422743 wikitext text/x-wiki hg15xzwphfg7e4cokbctauvk2dh06mf 10 37907 2422601 2405885 2026-06-02T17:28:54Z BCornwall-WMF 31521 Add [[CDN/headers]] 2422601 wikitext text/x-wiki igc8ky7yvx2kvtop438p6jec3njlllq 0 152515 2422778 2201956 2026-06-03T02:01:19Z Krinkle 44 2422778 wikitext text/x-wiki p36pwa7w3utamr72s0mplkjvrtt0ka8 0 441095 2422945 2422130 2026-06-03T09:31:39Z CMooney (WMF) 24288 /* CoreRouterInterfaceDropPercent */ 2422945 wikitext text/x-wiki sbqaswr2e7ggbp5p1mk4opr8hmk9h9t 2422969 2422945 2026-06-03T09:53:03Z CMooney (WMF) 24288 /* InterfaceOpticLowPower */ 2422969 wikitext text/x-wiki cya33exq44xpn73d9h637h9ojo658tj 2422986 2422969 2026-06-03T10:20:20Z CMooney (WMF) 24288 /* InterfaceOpticLowPower */ 2422986 wikitext text/x-wiki 07b0xvigpbsiegy0lnwkxp20eptnt7m 0 442952 2422504 2421075 2026-06-02T15:20:24Z BLiviero-WMF 53603 Removed volunteer column; as i was advised that several of these have no longer been active here. 2422504 wikitext text/x-wiki n3pna77nqpv89ggm4fbrc9360ngzheb 2422511 2422504 2026-06-02T15:29:33Z BLiviero-WMF 53603 Reworked the "bus factor" table to match the skills matrix table 2422511 wikitext text/x-wiki lan7lq740pbrcj1frfk0tz08672veit 2422705 2422511 2026-06-02T18:43:57Z AndrewBogott (WMF) 51047 2422705 wikitext text/x-wiki fevnoa764xm0fnfy59bresw1s18fobv 2422706 2422705 2026-06-02T18:44:43Z AndrewBogott (WMF) 51047 2422706 wikitext text/x-wiki dm77su48qdsgigpv1rel4pwe3c1li3g 2422707 2422706 2026-06-02T18:45:10Z AndrewBogott (WMF) 51047 2422707 wikitext text/x-wiki 9yxpfye3d51l5tadyx81ly3g8r19ags 2422871 2422707 2026-06-03T08:25:50Z FNegri-WMF 32595 Fill my column 2422871 wikitext text/x-wiki rl378ihdrych8obkq703ssf4mxixran 2423003 2422871 2026-06-03T10:42:08Z Taavi-WMF 41365 update table 2423003 wikitext text/x-wiki nwqflzn06zx3ehwlt0ki3va8rlnwu06 0 444148 2422310 2371690 2026-06-02T12:29:43Z DPogorzelski-WMF 48508 2422310 wikitext text/x-wiki tew6hta8g4e2lh9gpxi8tymegfpmsaj 0 446051 2422583 2421615 2026-06-02T17:03:54Z Ottomata 88 2422583 wikitext text/x-wiki enyw1bkrb5h8uyxee0bxkum6ofh20yy 0 447352 2423058 2375006 2026-06-03T11:27:37Z Effie Mouzeli (WMF) 12880 2423058 wikitext text/x-wiki epd6yi9bfu939jesj72rp1mz1nv1zby 0 448386 2422858 2394659 2026-06-03T08:14:15Z Effie Mouzeli (WMF) 12880 /* aux */ 2422858 wikitext text/x-wiki ktjo8uqng4foco686xlzqvybx67bf79 2422872 2422858 2026-06-03T08:25:54Z Effie Mouzeli (WMF) 12880 2422872 wikitext text/x-wiki 0oigb7rf4bhw4wvq4j8630306eq7fgg 0 448483 2422684 2342521 2026-06-02T18:20:49Z DerHexer 2993 fix 2422684 wikitext text/x-wiki 3riis7gospnhje0w95hzfp4hb1aaba1 0 449160 2422410 2422235 2026-06-02T14:03:16Z Dexbot 30554 Bot: Updating the report 2422410 wikitext text/x-wiki 4lhkn91641y3kqe7h8gs6gf2cdtxtnl 2422718 2422410 2026-06-02T19:10:01Z Dexbot 30554 Bot: Updating the report 2422718 wikitext text/x-wiki 1g2lk3j3rtsklowmccuzvwdddgixn8u 2422766 2422718 2026-06-03T00:01:36Z Dexbot 30554 Bot: Updating the report 2422766 wikitext text/x-wiki 7ffhbvsmlh5p68rwrg3zcotv38q1dyt 2422809 2422766 2026-06-03T06:47:30Z Dexbot 30554 Bot: Updating the report 2422809 wikitext text/x-wiki 68zcwkggw8iyau2vn83fzk04isva3ai 2422826 2422809 2026-06-03T07:44:38Z Dexbot 30554 Bot: Updating the report 2422826 wikitext text/x-wiki 7l8t75jq4uwmcboxg7cgwth8n8uixad 2422856 2422826 2026-06-03T08:05:41Z Dexbot 30554 Bot: Updating the report 2422856 wikitext text/x-wiki 05awi2m8azi8d61mq40kknfiobtt1cn 0 451180 2422600 2416291 2026-06-02T17:28:18Z BCornwall-WMF 31521 Remove [[#Headers]] section as that's been moved over to [[CDN/Headers]] 2422600 wikitext text/x-wiki titwlk6vv6zmuu0a67xbmh94raoz2wr 2422732 2422600 2026-06-02T19:55:45Z BCornwall-WMF 31521 Remove history section as it's been migrated to [[CDN/History]], link to it in [[#See also]] 2422732 wikitext text/x-wiki bhdawareze8jb4hnh33ypegu643knl8 2422733 2422732 2026-06-02T19:58:09Z BCornwall-WMF 31521 Remove history mention in intro blurb 2422733 wikitext text/x-wiki bi4n79kt48ihnrs44hew4kjjdufot0e 2422745 2422733 2026-06-02T20:59:53Z BCornwall-WMF 31521 Remove [[#Further reading]] as that is better suited to [[CDN/History]] 2422745 wikitext text/x-wiki 8p86yat8fd0lq0up9hbuesn7q6edfs8 0 451448 2422783 2411160 2026-06-03T02:10:17Z Jdforrester (WMF) 468 /* Write a config change to deploy a new version of the orchestrator */ Now with git tags! 2422783 wikitext text/x-wiki c3hckwitjhsb4z6fz0528s1jpuogdvm 0 451490 2422709 2421985 2026-06-02T18:54:31Z LSandergreen-WMF 46345 2422709 wikitext text/x-wiki 2m93kbtwuoxwcgmc3mglewwlucsn2rn 0 452006 2422555 2380431 2026-06-02T16:11:36Z BTullis (WMF) 25295 /* Rados Gateway Daemons */ Add reference to the codfw S3 interface 2422555 wikitext text/x-wiki p6a7sg9iibtcmcx5zs16hwsoxvrmd40 2422560 2422555 2026-06-02T16:22:30Z RKemper (WMF) 17058 /* Rados Gateway Daemons */ fix inconsequential typo (s/oject/object) 2422560 wikitext text/x-wiki qfepj8kwhfjwoolumow0o8kxy2taqic 116 453906 2422400 2422061 2026-06-02T13:57:15Z Gitlabaccountapprovalbot 37332 prasunaenumarthy was rejected. 2422400 wikitext text/x-wiki o5coxy8macbxy2bq0mb746iamt80cwr 2422747 2422400 2026-06-02T21:21:11Z Gitlabaccountapprovalbot 37332 mgagat was rejected. 2422747 wikitext text/x-wiki df4uui2ca6i8uhl63se0qnce42qadmp 0 456082 2422818 2374634 2026-06-03T07:18:49Z ISarantopoulos-WMF 34784 2422818 wikitext text/x-wiki 3y6pcloip1fq8o9b86oly0azhes29u0 0 458580 2422721 2380419 2026-06-02T19:13:17Z BCornwall-WMF 31521 Update x-trusted-request link 2422721 wikitext text/x-wiki fp2gir0qlkx7rllax83y2n40glouah0 0 459055 2422596 2365247 2026-06-02T17:22:07Z BCornwall-WMF 31521 Redirect to [[CDN/Headers#X-Provenance]] 2422596 wikitext text/x-wiki jeh9o7hrf1glh8bdwerpgjj9dkg3x0t 0 459115 2422519 2421314 2026-06-02T15:34:30Z AndrewBogott (WMF) 51047 /* Openstack Components */ 2422519 wikitext text/x-wiki 7y84zmgkpmghvj60mmvadj3cuief1s6 2422557 2422519 2026-06-02T16:13:38Z AndrewBogott (WMF) 51047 /* image repo */ 2422557 wikitext text/x-wiki nwstl4nnjw34cxgc5svbg1i3gw4yotv 0 459138 2422756 2419339 2026-06-02T22:16:35Z WBrown (WMF) 37696 update for bad login 2422756 wikitext text/x-wiki o9fpbwhkz09pxfimfepob6y2d9x4px0 0 459194 2422977 2346934 2026-06-03T10:04:51Z SSapaty (WMF) 28955 Add brief note to point to initial assessment 2422977 wikitext text/x-wiki 0kpy7dtkajdax4o7ugzms8klkw5aqxy 2423051 2422977 2026-06-03T11:17:06Z SSapaty (WMF) 28955 2423051 wikitext text/x-wiki ealfb7rvmyb218hxz25updmmj9znoc2 0 459366 2422471 2380679 2026-06-02T14:59:09Z Krinkle 44 +[[X-Image-Generator]] 2422471 wikitext text/x-wiki 1cp6xzedx0kim7twknuhmjjd76nw8p0 2422474 2422471 2026-06-02T15:00:10Z Krinkle 44 2422474 wikitext text/x-wiki kqse9ny24s8smgwg7jwt91xj8xqpfpo 2422477 2422474 2026-06-02T15:00:49Z Krinkle 44 +[[Category:Caching]] [[Category:SRE Traffic]] 2422477 wikitext text/x-wiki 3pt01lr9b7stcg685t6tom0mtcj5456 2422605 2422477 2026-06-02T17:37:24Z BCornwall-WMF 31521 Update X-Image-Generator and X-Provenance locations 2422605 wikitext text/x-wiki a6tnc1ys8qoqosfgdekxnoqzrc0eqts 2422720 2422605 2026-06-02T19:12:25Z BCornwall-WMF 31521 Redirect to [[CDN/Headers]] 2422720 wikitext text/x-wiki clcsosun4uicd78o3aq7qm90wug84ea 0 459560 2422324 2416721 2026-06-02T12:45:25Z DAdedoyin (WMF) 39150 Test 2422324 wikitext text/x-wiki 87ji5pgvr882s4uyiojbar1xvqcjt6b 0 459699 2422762 2421964 2026-06-02T22:55:51Z Stashbot 7414 eileen: config revision changed from 46874905 to 67231927 2422762 wikitext text/x-wiki ozq1704mu86vrlvgaf66lcswf6pdvf1 2422764 2422762 2026-06-02T23:14:32Z Stashbot 7414 eileen: config revision changed from 67231927 to c011a0d6 2422764 wikitext text/x-wiki k01lyb536hpvu2vfoh3licy96gy64i3 2422782 2422764 2026-06-03T02:07:55Z Stashbot 7414 eileen: civicrm upgraded from 663c0c30 to f6839b65 2422782 wikitext text/x-wiki t2d2y2xcx58xu438s93bxr09moob3mi 2422784 2422782 2026-06-03T03:32:45Z Stashbot 7414 eileen: civicrm upgraded from f6839b65 to 219cf085 2422784 wikitext text/x-wiki nh6pde5s44pw9nrj9lzu9efcoirspdj 2422785 2422784 2026-06-03T04:28:09Z Stashbot 7414 eileen: civicrm upgraded from 219cf085 to 55adc0bb 2422785 wikitext text/x-wiki 7borwu8y5rtahxdjtepj3jexi1bwc80 2422792 2422785 2026-06-03T05:46:44Z Stashbot 7414 eileen: civicrm upgraded from 55adc0bb to 6b61d3a5 2422792 wikitext text/x-wiki 3iwfmhjbtl31sj6ricdtoxuwhmarnxy 0 459705 2422771 2400712 2026-06-03T01:55:48Z Krinkle 44 Krinkle moved page [[CDN/Backend api/Sitemap access]] to [[Sitemap access]] over redirect 2400712 wikitext text/x-wiki ihikvudqp2qzh9ns5i5fkftprfuckxe 2422773 2422771 2026-06-03T01:56:09Z Krinkle 44 2422773 wikitext text/x-wiki 03lp0pqk1r2ympkh5nidugmozdalfx1 0 459783 2422722 2420363 2026-06-02T19:15:27Z BCornwall-WMF 31521 Update CDN headers link 2422722 wikitext text/x-wiki 0l0u3etsj1u5dsybv2ujx5kfc1mas28 2423050 2422722 2026-06-03T11:16:28Z CGoubert-WMF 32775 Add small redioscope blur 2423050 wikitext text/x-wiki 3hxcy2x321w8bnea4qf4v6qkouq0da6 0 459923 2423046 2389744 2026-06-03T11:14:14Z CGoubert-WMF 32775 Clarify redioscope's current use 2423046 wikitext text/x-wiki snkfkbzt9d9xkn23bw7bmk4817km4q8 2 460106 2422776 2407061 2026-06-03T02:00:30Z Krinkle 44 Krinkle moved page [[User:Effie Mouzeli (WMF)/SCROLL/Service page&action=edit]] to [[User:Effie Mouzeli (WMF)/SCROLL/Service page]] without leaving a redirect 2407061 wikitext text/x-wiki 3rz07mxjqezis93gdbjqmaz1l6zb0mp 2422777 2422776 2026-06-03T02:00:56Z Krinkle 44 comment out category 2422777 wikitext text/x-wiki i89vaj7zps7ar94mpwe8bnvi4cdz3hs 1 460190 2422590 2416300 2026-06-02T17:15:46Z BCornwall-WMF 31521 BCornwall-WMF moved page [[Talk:X-Analytics]] to [[Talk:CDN/Headers]]: Consolidating all CDN header articles into one page 2416300 wikitext text/x-wiki ik432vwnp3jzhhlka76739w4chvqwwa 2422592 2422590 2026-06-02T17:16:14Z BCornwall-WMF 31521 /* Proposal to merge into CDN/Headers */ Reply 2422592 wikitext text/x-wiki 97bzis7agf3cvqfzqjdaci1kyjwvks6 0 460232 2422748 2421966 2026-06-02T22:01:53Z MPopov (WMF) 5891 /* View and understand experiment results */ 2422748 wikitext text/x-wiki kx75jfjmffngl33hv6qlvxbnsy4wrvn 0 460273 2422296 2026-06-02T12:14:59Z SLyngshede-WMF 31501 Created initial page for X-Image-Generator 2422296 wikitext text/x-wiki 33kh0n9h3dhgo5xqfj96zxl354hni2g 2422298 2422296 2026-06-02T12:16:19Z SLyngshede-WMF 31501 Added link to MediaWiki implementation 2422298 wikitext text/x-wiki niwrk1ola8w4sux921c479g1u08w0c8 2422470 2422298 2026-06-02T14:57:50Z Krinkle 44 wikify 2422470 wikitext text/x-wiki 6339e9mzlvasui13okk0c222ol8x98b 2422473 2422470 2026-06-02T14:59:49Z Krinkle 44 2422473 wikitext text/x-wiki grkvgvj2539ndwmhhyzn8rcecx515bm 2422475 2422473 2026-06-02T15:00:16Z Krinkle 44 +{{Navigation Wikimedia infrastructure|expand=caching}} 2422475 wikitext text/x-wiki 51w9nuum8s6qxj4tk2afeohd9cjiakt 2422476 2422475 2026-06-02T15:00:41Z Krinkle 44 +[[Category:Caching]] [[Category:SRE Traffic]] 2422476 wikitext text/x-wiki kzucwo3b77j2hf4ze4p43ygyl7qzhyi 2422650 2422476 2026-06-02T18:02:12Z BCornwall-WMF 31521 Redirect to [[CDN/Headers#X-Image-Generator]] 2422650 wikitext text/x-wiki 76v0hjet6vcmsry34f8z7cqucd72bm9 0 460274 2422338 2026-06-02T12:57:01Z Effie Mouzeli (WMF) 12880 Redirected page to [[MariaDB]] 2422338 wikitext text/x-wiki tqfxq5mbewfxmcb2t1x8daq4iitk88n 0 460275 2422589 2026-06-02T17:15:46Z BCornwall-WMF 31521 BCornwall-WMF moved page [[X-Analytics]] to [[CDN/Headers]]: Consolidating all CDN header articles into one page 2422589 wikitext text/x-wiki clcsosun4uicd78o3aq7qm90wug84ea 1 460276 2422591 2026-06-02T17:15:46Z BCornwall-WMF 31521 BCornwall-WMF moved page [[Talk:X-Analytics]] to [[Talk:CDN/Headers]]: Consolidating all CDN header articles into one page 2422591 wikitext text/x-wiki 88bsx5gsk6t6wet3h6tucuolh3lv590 0 460277 2422731 2026-06-02T19:54:33Z BCornwall-WMF 31521 Import History section from [[CDN]] 2422731 wikitext text/x-wiki mmjhgiu1cp60yoag8ls26gtwdvz8555 2422746 2422731 2026-06-02T21:00:16Z BCornwall-WMF 31521 Include [[#Further reading]] from [[CDN]] 2422746 wikitext text/x-wiki bsaqryzhe6cy9ic9orzh327iiywq4jc 0 460278 2422772 2026-06-03T01:55:48Z Krinkle 44 Krinkle moved page [[CDN/Backend api/Sitemap access]] to [[Sitemap access]] over redirect 2422772 wikitext text/x-wiki a44nfcjn5gk4zscbgosdh16ydf5t1au 0 460279 2422780 2026-06-03T02:02:00Z Krinkle 44 Krinkle moved page [[Metrics.wikimedia.org]] to [[Obsolete:Metrics.wikimedia.org]] 2422780 wikitext text/x-wiki iuxz1fnzjmptrjz6xx6ywn9j26vdous 0 460280 2423049 2026-06-03T11:15:55Z SSapaty (WMF) 28955 Draft of initial assessment. 2423049 wikitext text/x-wiki roniythw6nlw6nwvzil7w3ciylmzn6b